Abstract
High-volume manufacturing at Los Alamos National Laboratory requires rigorous quality assurance; however, sustained manual inspection can be inconsistent, costly, and fatiguing. This paper presents the model-evaluation component of a larger human-in-the-loop artificial intelligence project for manufacturing anomaly detection. MicroSentryAI uses pretrained models to identify potential anomaly regions, while AnnoMate provides accessible tools for annotating, validating, and comparing those regions. The study emphasizes the distinction between classification, which determines whether a sample is anomalous, and segmentation, which identifies the location of an anomaly. Using Intersection over Union (IoU), precision, recall, and Euclidean distance between centroids, experiments with simulated mask errors demonstrated that IoU provided the most reliable overall measure of spatial agreement. Precision and recall helped distinguish between overannotation and underannotation, whereas centroid distance was highly dependent on anomaly size and shape and was therefore less effective without normalization. These findings support the development of more nuanced evaluation methods that combine multiple metrics rather than relying on individual thresholds. They also inform the continued development of the open-source AnnoMate with MicroSentryAI workflow for human-in-the-loop manufacturing inspection.
Recommended Citation
Smithson, Gavin; Fulp, Dakota; Jones, William M.; and Chakrabarti, Sharmistha
(2026)
"From Classification to Segmentation: Evaluating Machine Learning Defect Localization and Inspector Performance in Manufacturing,"
Journal of the South Carolina Academy of Science: Vol. 24:
Iss.
2, Article 8.
Available at:
https://scholarcommons.sc.edu/jscas/vol24/iss2/8