Xesol Innovation is a globally active company specializing in artificial vision and deep learning. Its offerings include solutions for safer driving and road safety. One such solution is Drivox Security: A safety system for industrial vehicles. It detects pedestrians and objects to the front and rear of the vehicle to prevent collisions. FRAMOS is supplying industrial GigE depth cameras with IP66 protection to Xesol and providing integration support.
The challenge
Intralogistics has high demands for efficiency. Warehouses often contain numerous industrial trucks and forklifts moving rapidly back and forth between workers and pallets. Highly dynamic processes of this kind bring risks and dangers. Drivers are often inexperienced and haveto work under intense time pressure. According to Xesol Innovation, a Spanish company for solutions based on neural networks and artificial intelligence, every year there are 110,000 accidents involving forklifts. Apart from interrupting operations, these accidents also result in personal injury, and sometimes fatalities.
Most are collision accidents, where the persons involved are pushed, rushed, or struck by forklifts. Although not required by law, audible reverse warning signals could prevent many accidents. A risk assessment can be performed to determine the extent to which it is necessary and makes sense to retrofit these signals to forklifts. There are various ways of detecting people and objects that could trigger an audible warning signal, e.g., ultrasound systems, infrared technology, and safety laser scanners, or camera-based solutions.
Having studied the problem in depth, Xesol Innovation decided on a 3D vision system, since ultrasound systems offer poorer resolution and have little ability to distinguish between humans and machines. Laser-based systems, in turn, are susceptible to interference from reflections and daylight. Systems based on 3D stereoscopic, on the other hand, are regarded as the most robust technology for both indoor and outdoor applications. In addition to the actual depth image, they can also output RGB images for better recognition performance.
New solution
The current solution Drivox Security by Xesol Innovation consists of three components (see figures 1 and 2):
Figure 1: Components (l. to r.): TFT LCD monitor, FRAMOS depth camera (behind the metal panel) and the Intel processor unit for AI-based image analysis.
Figure 2: Arrangement of components in the forklift: (1) display, (2) processor unit, (3) cameras (front and rear).
FRAMOS D435e cameras, fitted at the front and rear of the vehicle for three-dimensional perception, a microprocessor unit for AI-based image analysis, and a display serving as a monitor for the driver, which also shows the calculation results. If a potential collision risk occurs, the system warns the driver with a visible and audible signal. One of the project´s main goals was to limit the number of accidents in working environments to prevent interruptions to operations, and above all to protect workers´ lives.
In this project, FRAMOS supplies the rivox Security’s “eyes”. The global provider of image processing products, embedded vision technologies, customized solutions, and OEM services has developed an industrial GigE depth camera based on Intel RealSense technology, which gives drivers a view of areas that they would not always be able to see 100% without the camera. FRAMOS also provided support for integrating the cameras into the complete safety solution. The depth technology and 3D imaging are crucial in this application, where people and objects must be reliably detected at various distances, in any posture, position or surroundings, and the respective hazards assessed.
Figure 3: Enclosed FRAMOS D435e industrial camera with IP66 protection and mounted on the driver´s cab (title page).
Xesol Innovation was one of the first customers touse the new camera from FRAMOS, which is based on Intel RealSense technology, in the beta stage. The most important criteria for Xesol Innovation were the gigabit Ethernet interface and an industrial camera housing: these are the two main features that differentiate the FRAMOS from the standard version of the Intel RealSense camera. Intel suggested that Xesol Innovation should contact FRAMOS to implement the RealSense camera, and this was the first step towards the successful development of a solution.
FRAMOS depth camera
FRAMOS has developed an industrial version of the 3D camera that allows the advantages of Intel technology to be used for industrial vision systems in harsh environments. The FRAMOS camera works with the Intel sdk and thus allows easily to migrate from regular RS cameras without/ with little need to rewrite code. With advanced image analysis or AI functions, the background in the depth image can be masked out, distant objects can be hidden, or objects can be shown with a distance indicator, and so on. The depth accuracy depends on the distance between the camera and the object, the specific site, and the lighting conditions.
The Intel RealSense depth modules are a ready-to-use stereoscopic 3D system. It determines how far away an object is by computing disparities in the stereo image. This technology requires considerable processing power, as the disparity has to be calculated for each pixel in the two images. For this application, it is essential that the two cameras are stably positioned and focused. The Intel RealSense cameras are therefore pre-calibrated and precisely aligned. The internal Intel D4 processor in the camera performs the computations, i.e., it takes care of “merging” the two images. To do this, it calculates the disparity and hence the associated depth in real-time for every pixel in the image, to reliably detect objects and people. An IR projector in the camera ensures good image results even if the scene contains very little structure for calculating the disparity, for example a flat wall. Additional texture from the IR projector (texture generator) improves the computation results and means a depth map can be generated even with uniform structures.
The housing of the FRAMOS camera offers IP66 protection against water, dust, and has four M3 holes for flexible mounting. It features lockable gigabit Ethernet (M12), power supply and GPIO (M8) connectors. FRAMOS has developed its own processor board for the camera to meet industrial interface requirements. The camera also supports PoE operation (power over Ethernet via Ethernet/PoE interface) to minimize cabling. Its operating temperature range is between 0 °C and +55°C.
The main benefits of the FRAMOS camera briefly: Simple and fast installation Plug & play with Intel RS SDK 2.0 IP66 protection against dust and strong jets of water. The housing with M12 connector is designed for Ethernet and PoE operation. Long cable length up to 100 m is possible. Network technology allows multiple camera operation.
Results
In Drivox Security by Xesol Innovation, the camera enables a complete view of pedestrians and objects at distances between 1.0 and 8.0 m from the forklift. It transmits images of the surroundings, in a live stream at 30 fps with a resolution of 1280 x 720 pixels, to the display in the forklift (see fig. 4).
Figure 4: Camera´s field of vision to the rear; the driver can see everything on the screen.
The screen in the driver´s cab continuously displays the images from the cameras fitted at the rear and front of the vehicle (see fig. 5).
Figure 5: Person detection on the display, with alarm status.
AI-based analysis of the video stream is performed in the separate microprocessor unit. Drivox Security can detect pedestrians and obstacles in three different detection zones and indicates the individual collision risks in each case. These are computed by neural networks for element detection and identification, with corresponding decision algorithms. The software is configured individually for the specific vehicle.
When pedestrians or obstacles are identified in the various detection zones, the system gives a visible and/or audible alarm (see table in fig. 6).
Figure 6: Alarm signals when the vehicle approaches pedestrians or objects.
The colors red, orange, and yellow represent the different collision risk levels. The system reliably detects pedestrians in all conceivable positions and postures, so it cannot be outsmarted even by a “disguised” approach (see fig. 7).
Figure 7: The software reliably detects people in any posture or position.
Advantages
- Fewer accidents at work and lower associated costs
- Detection of risk situations and early accident warning
- Reduction of downtime costs
- Greater route planning efficiency with heat maps
- Determine key factors for increases in incidents/accidents
Increased safety while driving
Investigations are currently being carried out to determine whether the camera´s field of view can be enlarged. It would be desirable to have a camera with a greater range of vision, both in terms of depth and with larger horizontal and/or vertical parameters. Experts at FRAMOS are assisting Xesol Innovation with the integration of new industrial 3D vision cameras,especially in applications that combine 3D technology with artificial intelligence (AI functions).
Written by Ferdinand Reitze, 3D Product Manager, Framos
Back to Features
