Acquiring Train Images

Part Variance & The Deep Learning Model

Training the deep learning model to process a inspection point requires a suitable set of train images. These images should represent the variance seen between parts, perspectives and lighting. The amount of images required depends on the amount of variance in the part and the reliability of the deep learning models ability to identify the parts features. Typically, this will require 50-400 images to be trained in before the system produces predicable results.

Images acquired for the training phase will typically be split 70/30 for training and testing. This means that around 70% of the images will be trained and the remaining 30% will be used to test the accuracy of the model. It is crucial that the images represent the variance the system will see in production. Ideally this means that each train image will be of a different part with a suitable number of images (50%+) of the part being in a failed state. This approach is not practical in situations where the MVS system is being installed as part of a new part program, as access to parts may be limited. In this situation the variance may need to be simulated on a limited number of parts.

The closer the variance is in the train set to the actual variance the system will see in production the more reliable it will be. This means that simply adding more images in the train set does not equate to a more accurate model. You could take hundreds of images of the same part with little to no variance and create a model that is inferior to a model created with only a dozen images of different parts. Acquiring train images with suitable variance is much more important than acquiring large amounts of images with little or no variance.

Its also worth noting that variance is not always apparent to the human eye. Small scratches and varing reflections on surfaces will be picked up by the deep learning model. These features will be trained out of the system with a suitable training set, but models that have been thoroughly trained on only a single part will typically be unreliable when a second part is introduced. Once several parts are trained into the model these small, part-unique features will typically be trained out of the model.

Simulating Variance

In order to simulate variance it is important to understand the part and the feature(s) being inspected. The deep learning can only process and analyze the pixels in the image sent to it, so its important to have a understanding of which features on the part will be used to train the model and how those features might change. For example, the part may have variants where certain features change color, presence, position or size. The part might have highly visible hex bolts that will be used by the deep learning to fixture the image, and those bolts could be tightened to any rotation. The lighting/reflections of the part may also change if the part is moved or if the inspection takes place in different cells.

Any variance that is expected in production should be simulated in the train set. Some common forms of variance and ways to simulate them are listed below:

  • Bolts: these can come in any orientation and should be rotated when possible.
  • Lighting: reflections off surfaces should be changed when possible by rotating the part or moving it around in the inspection area (if possible).
  • Perspective: try to take images of the part from slightly different perspectives and distances. Try to keep this within reason, as attempting to support too much perspective variance can lower the accuracy of the inspection.
  • Flexible/Movable parts: Should be adjusted between images (ie, wires, tubes, anything with inelastic deformation).

Simulating Failure Modes

The deep learning model will only be as intelligent as you can train it. For example, if you train a model how to classify oranges and apples then show it a banana, it will not know what to do and generate a unexpected result. This translates to manufacturing where if you train a good part and one type of failure mode, it might not understand a new type of failure mode (note there are tools that can be used for these unsupervised problems, but typically only excel at surface inspections). For example if you want to ensure the correct bolt is installed you could train the correct bolt and a incorrect bolt into the model. If you only train with 2 bolt types (1 good and 1 bad) but there are multiple bad bolt types that can be installed on the part, the model will not understand the untrained type and can generate incorrect results. This is why its important to simulate all possible failure modes in the train set.

This also carries over into presence checks on good parts. If there are variants of the bolt that will be installed and the goal of the inspection is to simply check presence/absence of a bolt, it is important to train the model on all valid types of the bolt that can be installed (assuming the bolt has some visual difference).

Simulating all failure modes is only realistic for presence checks, and doesn't necessarily apply to other types of checks (location, orientation, etc). In these situations its best to attempt to capture the minimum and maximum variance in both the good and bad cases, along with a decent number of images between the minimum and maximum. The goal here should be to acquire images with a even distribution of variance.

Cheat Sheet

The following is a rough guide for acquiring images with simulated variance and failure modes. In general, as many images as possible with some new form of variance introduced in each image.

  • Aim to acquire about at least 50 images total for the initial training/feasibility of the inspection's perspective (phase 1). If the inspection results are ok, increase the number of images from 50 to a few hundred, ideally introducing more variance/parts (phase 2)
    • 60% bad images
    • 40% good images
  • Introduce part variance and perspective variance at the same time.
    • Take 3-4 images from slightly different perspectives after each adjustment of the part.
  • Iterate through all possible forms of part variance.
    • Swap components, adjust moveable parts, move the part / adjust lighting etc. They don't all need to be adjusted at the same time, but also don't need to be adjusted individually. Choose a process that will work best for you.