Finding the optimal K in kmean: a incremental kmeans in python

I was looking for an good implementation of an incremental k-means where I don't have to set the optimal K. There are interesting papers (x-means, gmeans etc.) but couldn't find any python implementation.

I have decided to write a incremental version on top of sklearn.
The idea is simple:

1. Start at K=x
2. identify worst cluster based on an unsupervised measure (ex: silhouette)
3. Split the worst cluster into 2 clusters
4. measure the global improvement with the new clusters
5. if you get an improvement continue adding clusters

You can find the source code in mlboost/clustering/ikmeans.py
A special thanks to scikit-learn lib to let me prototype this version so fast. 

kmeans with configurable distance function: How to hack sklearn.kmeans to use a different distance function?

Like others, I was looking for a good k-means implementation where I can set the distance function.

Unfortunately, sklearn.kmeans doesn't allow to change the cost function. Euclidean is hard coded. 

As pointed out by the sklearn team, it is quite complex to generalize due to:

•  mean is not always the average and is most of the time not trivial (http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf  see table 8.2 page 501)
• generalization isn't always compatible with optimized version

So if your distance function is cosine which has the same mean as euclidean, you can monkey patch  sklearn.cluster.k_means_.eucledian_distances this way: (put this code before calling kmean.fit(X).

from sklearn.metrics.pairwise import cosine_similarity
def new_euclidean_distances(X, Y=None, Y_norm_squared=None, squared=False) 
    return cosine_similarity(X,Y)

# monkey patch (ensure cosine dist function is used)
from sklearn.cluster import k_means_k_means_.euclidean_distances 

k_means_.euclidean_distances = new_euclidean_distances 

Warning: you need to normalize your input vectors.