Validating Vision AI: False Rejects, Missed Defects and Counting Accuracy

Match metrics to the task

For inspection, distinguish defective items accepted from good items rejected and state each denominator. For counting, record missed, duplicated and net-count errors. For OCR, include whole-field correctness. Report unreadable or unknown cases explicitly instead of removing them from the test set to improve the headline result.

Build independent reference evidence

Have qualified reviewers label representative samples, with a process for disagreement. Separate evaluation data from training and tuning data, including related batches where leakage is possible. Include difficult but legitimate operating conditions. Record sample size and defect prevalence so readers can judge what the test actually demonstrates.

Validate the complete application

Measure camera availability, processing delay, event identity and downstream handling alongside model decisions. A correct classification arriving too late may still reject the wrong part. Test restart and connection-loss behaviour. Keep results by product, line speed and shift rather than relying only on a pooled average.

Maintain a revalidation trigger

Record the accepted model, camera settings, lighting and product scope. Define changes that require retesting, such as new packaging or relocated cameras. Sample production results after release and investigate trends before silently retuning thresholds. Preserve the prior evaluation so an apparent improvement in one metric can be checked for regressions elsewhere.

FAQ