How to validate a model before using it for a control decision

Direct answer

It remains to be evaluated whether the model retains usefulness on data that did not participate in fitting and whether it represents the intended decision. A good retrospective fit may depend on future information, a limited regime or operator actions the model does not distinguish.

Validation mechanism

Separate fitting and evaluation periods while preserving time order when observations are dependent. Data transformations must be estimated on the fitting set. Information reserved for evaluation loses its value if it is used repeatedly to select the model.

The comparison should include a simple baseline and the conditions under which the error changes. Predictive performance alone does not demonstrate causality or the safety of a control action.

Information that must be retained

• the objective, variable and horizon to be estimated;
• actual availability of each input at decision time;
• fitting, selection and evaluation periods;
• changes in material, setpoints and operating regime;
• exclusions, signal quality and assumptions;
• error, bias and performance against a simple baseline.

Sources and review

The scikit-learn documentation on model evaluation explains data separation, information leakage and time-based splits. The discussion of the operating decision is an editorial proposal and requires engineering review. Using one split technique is not claimed to validate a complete industrial application.