Introduction

Gaussian Mixture Models (GMMs) are a probabilistic approach to cluster analysis.

Unlike methods like K-means, which assign each data point to a single cluster, GMMs assume that data points are generated from a mixture of multiple Gaussian distributions. This probabilistic framework allows for more flexibility in capturing complex cluster shapes and assigning a probability of belonging to each cluster for every data point.

Detail

Gaussian Mixture Models (GMM) assume that our data comes from several groups (or clusters), each shaped like a 3D bell curve, called a Gaussian distribution.

The components of a Gaussian Mixture

Every cluster in GMM is defined by three things:

  1. Mean ($μk$) (’mew’): This is the centre of the cluster. Think of it as the average position of the data points in that group.
  2. Covariance ($Σk$): This describes the shape and size of the cluster. It tells us how spread out the points are and in which direction they vary.
  3. Mixing Coefficient ($πk$): This is how much weight each cluster contributes to the overall dataset. For example, if 60% of the data comes from one group and 40% from another, the weights would be 0.6 and 0.4. All the weights add up to 1.

The probability of a data point

For each cluster, GMM uses a mathematical formula to calculate the probability that a point belongs to that cluster. This formula is called the Gaussian (or Normal) Distribution.

In simple terms, it describes how data is spread out around the mean.

For a data point x in cluster k, the probability is:

$$ \mathcal{N}(x | \mu_k, \Sigma_k) = \frac{1}{\sqrt{(2\pi)^d |\Sigma_k|}} \exp\left(-\frac{1}{2}(x - \mu_k)^T \Sigma_k^{-1} (x - \mu_k)\right) $$

Here’s what each part means: