Cross-validation and other model validation tools. Part 2 – Cross-validation: pitfalls and possibilities

Cross-validation (CV) is used by SIMCA to estimate the number of components in a multivariate model (PCA/PLS/OPLS/O2PLS). The primary output from the cross-validation procedure is a parameter denoted Q2, the predicted variance or the goodness-of-prediction. Briefly, cross-validation is performed by dividing the training set data into a number of groups and then developing a number of parallel models from the reduced data with one of the groups deleted. After developing a model, the deleted data are used as a test set, and differences between actual and predicted values are calculated for the test set, eventually leading forward to the Q2-value.  

In most cases, cross-validation provides a very reasonable first approximation of the predictivity of a multivariate model. However, there are some cases where cross-validation is known to face problems:

  • Design of experiment (DOE)-data: When parts of the data are removed in CV, the underlying design often collapses and there is no longer any preferred projection direction. This may lead to unrealistically low estimates of the predictive power.
  • Grouped/Clustered data: Although many samples are omitted in each CV-cycle, strong groupings may still persist and thereby cause the estimate of the predictive power to be higher than it should be.
  • Auto-correlated process data: Predictive power may not be reliably estimated, because time points (observations) adjacent to those eliminated in a CV-round, carry information very similar to the eliminated data points.
  • Batch evolution models (BEM): Removal of parts of developing batches may lead to an over-optimistic view on predictive power.

The implementation of cross-validation in SIMCA contains a number of adjustable settings and options that can help to mitigate some of the risks noted above. Attend this webinar and learn more about cross-validation and understand what options are available to fine-tune the cross-validation behaviour.