7

The algorithm in question is Kohonen's SOM. But the question could also apply to PCA and some others.

When the umatrix (or the codebook?) is examined, is there a way to tell how successful clustering was?

And would it be a good idea to apply GA's to optimize size, lattice structure, learning rate, and the learning degradation functions as well as the epoch count for clustering, or is there a danger of overfitting in this instance?

Assume that the SOM data is coming from a demonstrably weak PRNG and that the very first attempt shows a distinct structure. Is there some statistical property or algorithm that can evaluate the presence and degree of structure to be used for a GA fitness function?

AkKoh
  • 71
  • 1
  • 2

3 Answers3

10

First, PCA is not a clustering method. It is a dimensionality reduction scheme. You can assess the performance of PCA through analysis of the percent of variance in the dataset that is retained as you decrease the number of dimensions. Retaining 99%,95%, or 90% is usually ideal, depending on your problem.

With regards to clustering, you probably want to start with the Silhouette Coefficient. This combines assessments of both the cohesion (how tight a cluster is) and separation (how well separated each cluster is from other clusters).

  • Calculate a = average distance of i to the points in its cluster
  • Calculate b = min (average distance of i to points in another cluster)
  • The silhouette coefficient for a point is then given by: s =1 – a/b, if a < b

It is typically between 0 and 1 with larger numbers being "better". You can average the coefficients over a cluster or the entire region to get an assessment of the cluster or the entire clustering procedure for your data.

More generally, try googling "assessing clustering" or "cluster validity" to read about all of the other ways that you can score your clustering algorithm. Here is a very complete treatment of the topic.

Hope this helps!

AN6U5
  • 6,808
  • 1
  • 24
  • 42
4

It is very hard to evaluate the correctness of the results produced by an unsupervised algorithm. In many cases, such evaluation will be totally subjective, and it will require some knowledge about the domain of the problem.

If we focus on clustering algorithms (as stated in a previous answer, PCA is not a clustering algorithm), many cluster validation measures may be applied, like the ones enumerated in the "Evaluation and assessment" section in the "Cluster analysis" Wikipedia page. These measures return a number that you can use to compare different clustering solutions, in terms of cluster compactness (how close to each other are the elements within each cluster) and separation (how far to each other are the elements from different clusters). You can, of course, use these measures in order to perform hyperparameter selection (lattice size, structure, learning rate, etc.) by means of cross validation.

However, it must be noted that different cluster validation measures may produce different validation results, and as a consequence, your best clustering solution may vary depending on the chosen measure. Therefore, even the choice of a validation measure is subjective. Once again, knowledge about your data is very important to take this decision.

Pablo Suau
  • 1,787
  • 1
  • 13
  • 20
3

Silhouette Score, it was mentioned in a previous answer but I don't have the reputation to comment. Here and here are useful links if you are using python to implement clustering. Check section 2.3.9.4.

Silhouette Score takes overfitting into consideration I think. For example if I have a dataset with 24 points to cluster, if I put them in 23 clusters the score is 0.0263. If I put them in 6 cluster using K-Means then I get a score of 0.2705.

So in your GA, you can use this quite nicely as a fitness function.

Leela Prabhu
  • 183
  • 1
  • 7