Contemporary vision systems are often located on production lines where products conveyed through them are automatically inspected for defects that are known to occur. One inspection system typically consists of multiple acquisition stations inspecting different surfaces of the object.
At a typical acquisition station, a product is illuminated by a light source and an image of the product captured by a camera equipped with the appropriate optics. Having captured an image, the system software performs an analysis on it to determine whether any defects that are known to appear in such a product are present. Having done so, the system either rejects or accepts the product.
Because there is no one-fits-all solution, to automatically inspect all parts of the object such systems need to be custom built for each product, or series, of similar products. The development of such systems is performed by systems integrators who work closely with a manufacturer to determine the most appropriate inspection solution to meet the demands of the application.
During a typical consultation process with a systems integrator, a manufacturer defines the number and type of defects that the system is required to identify on a product line. Having done so, the systems integrator tests different acquisition system prototypes that might yield images that are suitable for subsequent defect detection. The decision on the use of hardware components in the prototypes is based mostly on the experience of the engineer building the system and involves the iterative empirical process of result evaluation.
When a prototype is developed, the efficiency of the system in determining the number and type of faults in the product or product line can be evaluated and the results discussed with the manufacturer. The system may then need to be refined until all the quality control requirements of the manufacturer are met. Once they are, the prototype is further optimized before the final system is built and integrated into the production line.
Let’s make it agile
Such systems have proved highly effective in many modern production environments. However, they are typically built to detect defects on only a limited number of product variants, and unable to inspect a variety of product types. In the future, however, the production of highly customized products means that manufacturers are likely to demand that many smaller batches of different product types should be inspected on a single line. Hence the existing inflexible concept of inspecting such products through a single dedicated inspection station will need to be seriously rethought.
According to Petra Gospodnetic, a PhD student at the Fraunhofer Institute for Industrial Mathematics in Kaiserslautern, Germany, the solution is to automate the initial inspection planning process. In a presentation entitled “Inspection 4.0 – Let’s make it agile”, Ms. Gospodnetic outlined her ideas on how that might be achieved at the EMVA Business meeting held in June this year in Dubrovnik, Croatia.
Ms. Gospodnetic acknowledged that many industry veterans in the audience might be somewhat skeptical of whether such an all-encompassing vision system could be developed at all. Indeed, she acknowledged that many might consider that creating such a flexible system might be thought of as impossible, especially considering the variety of products, shapes and materials – not to mention the variety of defect types -- that such a system would need to be versatile enough to accommodate. Nevertheless, she argued that the development of such a vision inspection solution might indeed be possible if one took advantage of the many recent advances in computer vision, computer graphics, machine learning and robotics. If the expertise developed in all these perhaps seemingly disparate areas might be brought to bear on the issue of machine vision inspection, then it may be possible to create highly flexible inspection stations that could accommodate numerous product variants.
Virtual image processing
To create such a flexible system, Ms. Gospodnetic proposed that it would be necessary to develop a computer simulation framework capable of virtual image processing. The framework would realistically model the inspection environment and use that model to assess different image processing algorithms.
Modeling the complete inspection environment implies it would not only model the properties of the product to be inspected in virtual space, but also properties of all of the hardware components in the system as well, such as the illumination system, camera and optics. Based on a set of predefined requirements, such as speed, coverage and precision, the framework would further yield a minimum set of points of interest, marking where and how the product should be inspected.
To create a flexible vision inspection system, Fraunhofer researchers believe it is necessary to develop a computer simulation framework capable of virtual image processing. The framework would realistically model the inspection environment and use that model to assess different image processing algorithms. Modeling the complete inspection environment implies it would not only model the properties of the product to be inspected in virtual space, but also properties of all of the hardware components in the system as well. Based on a set of predefined requirements, such as speed, coverage and precision, the framework would yield a minimum set of points of interest, marking where and how the product should be inspected.
In the modeling phase, the virtual inspection environment offers a possibility to take into account the properties and evaluate the visibility of potential defect types for each product a system should inspect before a system integrator committed to assembling the actual hardware and software products required to build an inspection station. Once created, a resulting inspection plan would then be passed to a robotic arm simulator -- already available on the market -- in order to create the traversal path for the robot during the inspection procedure.
Ms. Gospodnetic argued that a 3D model (or digital twin) of a product -- rather than the product itself -- could be used as the basis from which to create an inspection routine that could be deployed to inspect the product. Today, physical-based rendering is already used in the film and advertising industry to create computer generated images of various scenes, which are visually difficult to discern from actual images of such scenes. Following that approach, it is possible to do the same for products to be checked by an inspection system and use the results for inspection planning and optimization purposes.
Today, systems integrators faced with inspecting objects of complex shapes need to determine from personal experience how to set up the lighting and the cameras in an inspection station to inspect a specific object so that there are no occlusions and the whole surface is thoroughly inspected. However, through using a physically based simulation of the part, Ms. Gospodnetic proposed that it is possible to calculate the surface regions that are visible for the camera and find a minimal set of camera positions needed to cover all of them.
To make such a system a reality, a 3D model of a part (which is usually a triangulated mesh) could be used to make a parametric surface approximation, thus creating a mathematical description enabling the surface characteristics such as curvature and orientation at each point to be evaluated. From the parametric surface approximation, viewpoint candidates that cover the surface of the geometric model could be generated and optimized to ensure that a vision inspection system would be able to cover the target surfaces.
Of course, to adequately inspect a product, the effect of lighting on the product would also need to be simulated in order to take into account the number and type of defects on an ideal part that would likely be encountered in a real life scenario. Today systems integrators build the systems based on many different samples of defect products. However, the Fraunhofer team working on the development of the new framework aims to change the paradigm and use machine learning on many samples of the product in order to model the correct and expected response of the material and use it to discern defects as a variation from the ideal.
Modeling the process
Using a 3D model of a single part to define areas of interest based on the geometry would, however, only be one part of the process of developing such a simulator. To model an entire inspection system and then model its effectiveness would be another thing entirely. As noted before, this would also require taking into consideration modeling all the other component parts of the system used to inspect a product, such as the movement of the mechanical structure of a robotic system and the cameras and lighting mounted on it.
To determine how lighting would affect the image of a product captured by a camera, such a simulator would require mathematical modelling of the illumination employed in the system. Here, computer graphic techniques such as ray tracing could be used to simulate images that would be captured by a camera in the system by tracing the path of light from the light source and simulating the effects of its encounters with virtual objects.
Naturally enough, while modeling the parts to be inspected may bring us one step closer to being able to build a flexible virtual vision inspection system that could be extensively trialed prior to deployment in the real world, more work is required before a fully fledged virtual vision system could be used to model a system prior to construction. Such a system would require collaboration between manufacturers of vision components and require the establishment of numerous standards to ensure that the hardware and software models required to build such a virtual environment all interoperated effectively.
While many executives in the EMVA audience in Dubrovnik were very impressed by the futuristic vision presented by Ms. Gospodnetic, others remained somewhat cynical that such a grand idea could ever be realized. Ms. Gospodnetic recognized these issues, however, and called upon those members present to collaborate with her research group to make her vision of flexible inspection a reality.
Written by Dave Wilson, Senior Editor, Novus Light Technologies Today
Back to Features
