The Akaike Information Criterion (AIC) is widely used metric for model selection in statistical and ML.
It was created by Hirotogu Akaike.
It gives us a way to compare different models based on how well they fit the data while penalising model complexity.
The AIC is calculated as:
$$ AIC=2k−2ln(L) $$
where $k$ represents the number of model parameters, and $L$ is the likelihood of the model given the data.
Lower AIC values indicate a better balance between goodness of fit and model simplicity.
When comparing multiple models, the one with the lowest AIC is generally preferred, as it suggests the best trade-off between explanatory power and overfitting. However, AIC doesn’t directly test the quality of a model, but rather ranks competing models relative to each other.
Unlike hypothesis testing methods (e.g., likelihood ratio test) AIC does not provide a probability or significance level; instead, it helps identify the model that minimises information loss.
Next reading: