図1：本取り組みが目指す工場設備点検の省力化イメージDecember 20, 2024
NTT Data Group Inc.
NTT Comware Corporation
Itochu Techno-Solutions Corporation
Mitsubishi Chemical Group Corporation
NTT Data Group, hereinafter referred to as NTT Data Group, NTT Comware Corporation, hereinafter referred to as NTT Comware, Itochu Techno-Solutions Corporation, hereinafter referred to as Itochu Techno-Solutions, and Mitsubishi Chemical Group Corporation, hereinafter referred to as Mitsubishi Chemical Group, aim to reduce the burden on workers inspecting factory equipment by realizing smart maintenance through the utilization of IOWN APN and AI. A joint verification of remote inspection of factory equipment using APN-based robot remote operation and AI-based image analysis was conducted. As a result of the verification, real-time detection of cracks in the pipes on factory walls and precise analysis of vibrations of the pipes, which are signs of deterioration, were achieved, obtaining a high standard of numerical results that indicate practical applicability in terms of image delay time and quality. In the future, by simultaneously acquiring environmental information such as images and sounds using multiple robots and devices, and implementing multimodal AI analysis, the aim is to realize a world where the state of factories in remote locations can be understood with higher accuracy and in real-time.
Figure 1: Image of labor-saving factory equipment inspection that this initiative aims for.
Background
The NTT Data Group, NTT Comware, Itochu Techno-Solutions Corporation, and Mitsubishi Chemical Group are participating in the activities of the IOWN Global Forum (hereinafter, IOWN GF) towards the realization of a sustainable society. These companies have developed a reference implementation model for the Remote Controlled Robotic Inspection use case, which includes functionalities and requirements for remotely operating robots to perform equipment inspections, in collaboration with IOWN GF partner companies.
Overview of Verification
In manufacturing sites such as factories, regular inspections for equipment maintenance are essential, and when the facility is large, a lot of effort is required for inspections. There are also risks associated with inspections at heights, such as falls. To reduce the burden on field workers, we verified a system that allows robots to patrol from remote locations utilizing the strengths of the IOWN APN, which enables high-speed, ultra-low-latency, and wide-bandwidth communications, as well as a system to detect pipe abnormalities using real-time video. Specifically, we constructed an APN environment 120 km apart between Odaiba and Gotanda and verified remote transmission of high-quality images with low latency from multiple devices and anomaly detection of equipment using AI analysis.
Roles of Each Company in Verification
| NTT Data Group | Provision of robots and a vibration analysis system for pipes |
| NTT Comware | Providing APN environment, providing image recognition AI, providing applications that visualize the analysis results of cracks in pipes on walls within a Digital Twin environment, providing robots. |
| Itochu Techno-Solutions Corporation. | Providing RDMA-compatible FA camera environment, validation of uncompressed real-time video transmission via APN. |
| Mitsubishi Chemical Group. | Definition of functional and non-functional requirements necessary for factory equipment inspections. |
Figure 2: Image of this verification.
Figure 3: Robots used in the verification.
(Top left) HBA SMART ROBOT manufactured by HBA, used by NTT Commware.
(Bottom right) Unitree dog robot utilized by NTT Data Group.
The verification details and results for each company are as follows.
NTT Data Group
By combining the computer vision technology accumulated by NTT Data Group with the latest robots, there is a goal to realize remote equipment automatic inspection and monitoring. In this verification, the Unitree Go2, a dog robot manufactured by Unitree, was operated remotely, and the vibrations of the inspected pipes were analyzed using AI based on the footage captured by the camera equipped on the robot. Abnormal vibrations in pipes indicate important data that shows signs of deterioration or damage. Inspection tasks that have previously relied on workers' visual or auditory judgments depend on the skilled level of the workers, necessitating a reduction in the risks of omitting or misjudging. This verification verified whether artificially generated vibrations in the pipes could be accurately analyzed from the footage taken by the robot, achieving the target values set for both remote operation and video analysis.
Figure 4: Verification image of NTT Data Group.
Remote control.
The robot was able to connect to the APN between locations without any issues. Furthermore, the video streamed on demand from the robot's camera enabled the operator at a remote PC to perform operations without experiencing latency, confirming that the operator could remotely control the robot using the PC's keyboard or remote while viewing the footage.
Video analysis.
Vibrations were analyzed from the video of the pipes recognized and captured by the robot's camera. The robot properly identified the target pipes from the markers without any misidentification. Additionally, vibrations' amplitude and frequency from the pipes were also extracted from the video analysis. Although the robot itself was vibrating due to its motors, it successfully analyzed the vibrations of the pipes, identifying the duration of the vibrations. Following this result, confirmation is underway with the Equipment Technology Department of Mitsubishi Chemical Corporation's Okayama plant to analyze if the specified practical reference values of amplitude 0.1mm and frequency 60Hz can be achieved.
Figure 5: Results of pipe vibration analysis.
NTT Comware
The aim is to achieve smart maintenance by AI analyzing footage captured by the inspection robot's camera of factory equipment at remote locations and creating a digital twin. In this verification, overall management of the project was undertaken, including the construction and implementation of the verification environment and progress management. The verification included analyzing streaming footage via APN, detecting cracks in the pipes on the wall, and reflecting them in real-time within the digital twin environment for reference from remote locations.
・Construction of verification environments such as APN
Designed and built an APN that enables verification over a maximum distance of 120 km between Odaiba and Gotanda, the verification location. Furthermore, the digital space used for this verification was implemented as a 3D Digital Twin space utilizing NTT Comware's 4DVIZ.
・Crack detection and analysis using image recognition AI
Based on the images captured by a camera attached to the arm of a remote-operated inspection robot (HBA SMART ROBOT), analysis was conducted using the image recognition AI Deeptector to detect cracks on pipes on walls in the real space and immediately reflect them in the Digital Twin environment. Additionally, in the Digital Twin environment, it was confirmed that clicking on an icon representing cracks plotted at the same position as in the reality allows access to images of the cracked areas in the real space. In this verification, high-resolution video at 4K 60fps was transmitted via wireless and APN, confirming that crack detection and analysis could be performed without any delay even at remote locations, achieving a level suitable for practical application in equipment inspections.
Figure 6: Image recognition AI crack detection image
Itochu Techno-Solutions Corporation.
Conducted a verification to transmit large capacity video live on APN using Remote Direct Memory Access (RDMA) cameras, which enable high-speed data communication to memory without going through the CPU. By varying the distance between the camera and PC over APN from 0 km to 120 km during video transmission, it was observed that as the distance between APN increased, throughput (data transfer speed) and fps (frames per second) decreased. Simulations were conducted based on the measured values of 2K and 4K videos to clarify the relationship between APN distance and throughput. Additionally, regarding video delay, the impact of processing delay from the equipment was found to be greater than the transmission delay of APN, highlighting the importance of device selection. Furthermore, it was confirmed that the transmission using RDMA reduced CPU load by 6% compared to general TCP transmission, demonstrating an effect on power consumption reduction.
Figure 7: Itochu Techno-Solutions verification configuration image.
Mitsubishi Chemical Group.
Before this verification, through preliminary verification at the manufacturing site, assistance was provided in identifying the needs and conditions that should be understood during remote inspections. Additionally, looking towards future practical applications, an in-depth investigation was conducted into the causes of the burdens faced by field workers, such as equipment maintenance engineers, during factory equipment inspections. These insights from the field were reflected in the verification conditions (such as vibration conditions and data capacity and speed).
Future outlook.
The four companies will continue to promote discussions aimed at achieving remote operation of robots, advanced analysis, and seamless video distribution in long-distance environments. Moreover, by simultaneously acquiring environmental information such as images and sounds using multiple robots and devices, and conducting multimodal AI analysis, the goal is to realize a world where the status of remote sites can be grasped with high precision and in real-time. This will significantly reduce the burden on workers during inspection tasks and lead to solutions to the challenges faced by manufacturers, such as improved safety in hazardous high-altitude work. Moving forward, discussions will be held at IOWN GF while advancing the creation of solutions using the APN and developing new functions. In the next step, a verification will be conducted regarding abnormal detection using robot utilization and AI analysis in the manufacturing sites of Mitsubishi Chemical Group.
Notes.
- Note 1: APN (All-Photonic Network) is one of the major technological fields that make up IOWN, introducing photonics (light) based technology from terminals to networks, aiming to realize end-to-end wavelength networks that provide a wavelength path with overwhelming low power consumption, high-speed large capacity, and low-latency transmission.
The APN environment is a collective term for the infrastructure that supports APN. - Note 2: Multimodal AI is an AI that can make integrated determinations using various types of input information according to the desired content. Modal refers to the types of input information for AI (such as video, sound, text, etc.).
- Note 3: The IOWN Global Forum is an international organization established to realize the IOWN concept. Currently, over 150 companies and organizations are participating.
- Note 4: The straight-line distance is approximately 5 km, but in this verification, communication is conducted over a total distance of 120 km via multiple locations. Gotanda is in Shinagawa Ward, Tokyo, and Odaiba is in Minato Ward, Tokyo.
- "4DVIZ" is a registered trademark of NTT Comware Corporation in Japan.
- "Deeptector" is a registered trademark of NTT Comware Corporation in Japan.
- Other commodity ETF, company names, and group names are trademarks or registered trademarks of their respective companies.
Contact information regarding this matter
Contact information for media inquiries
NTT Data Corporation
Public Relations Department
Umehara
TEL: 03-5546-8051
E-mail: nttdata-pr-inquiries@am.nttdata.co.jp
NTT Comware Corporation
Public Relations Office
TEL: 03-5796-4139
E-mail: kouhou@nttcom.co.jp
Itochu Techno-Solutions Corporation
Public Relations Department
TEL: 03-6433-4570
E-mail: press@ctc-g.co.jp
Mitsubishi Chemical Group Corporation
Public Relations Headquarters Broadcasting Relations Department
TEL: 03-6748-7140
Contact for inquiries about products and services
NTT Data Group Inc.
Technology Innovation Headquarters
Innovation Technology Department
Tabata, Yamaji
TEL: 050-5546-9741
