The VEDLIoT project demonstrates efficient AI for a wide range of IoT use cases.
The VEDLIoT project is dedicated to creating energy-efficient Deep Learning (DL) methods for distributed Artificial Intelligence of Things (AIoT) applications. Our comprehensive strategy emphasises algorithm optimisation while also tackling the safety and security issues inherent in AIoT systems. At its core, this strategy utilises a modular and scalable cognitive IoT hardware platform that incorporates microserver technology, allowing for the customisation of hardware to suit a wide range of applications. We employ heterogeneous computing to enhance performance and energy savings, and integrate a complete range of hardware accelerators, including specialised ASICs and FPGAs for adaptable computing. The project contributes to trusted computing, remote attestation, and secure execution environments, aiming to aid in the creation and implementation of robust, efficient AIoT systems. Further details on the VEDLIoT approach are outlined in a previous article titled ‘Unlocking the potential of IoT systems: The role of Deep Learning and AI’ in Issue 15 of The Innovation Platform.
We validate our architecture through use cases ranging from Smart Homes to automotive and industrial IoT devices. The following sections offer a detailed analysis of these applications, covering their general architecture, how VEDLIoT technology is incorporated, and the outcomes for each use case in various domains.
Automotive domain
The VEDLIoT project focused on the pedestrian automatic emergency braking (P-AEB) system, a critical safety feature in the automotive industry. This system addresses the challenge of detecting unpredictable pedestrians, who might suddenly enter the roadway from various points like sidewalks or behind obstacles. The P-AEB system relies on sensor data, predominantly from an RGB camera, and utilises Artificial Intelligence and Deep Learning (AI/DL) for rapid decision-making in emergency situations.
The system’s architecture comprises several components: Local sensor processing units, a central computer unit (CCU) in the vehicle, and edge processing units. These components are interconnected through a standard 5G connection. The primary challenge in this use case is balancing the limited processing power available on the vehicle with the need for quick, low-latency processing. To address this, the project explores the use of additional computational power from edge processing, considering the variable communication channel conditions that can affect data throughput.
For practical implementation, the project employed various AI models, with EfficientNet being a key model for evaluation. A significant aspect of this implementation is the dynamic partitioning of the DL model to distribute computational tasks between the onboard systems and edge resources effectively. This distribution considers several factors, including end-to-end latency, cellular communication capacity, and the processing speed of the edge units.
Data collection for training and testing the models was conducted using a camera system designed to meet specific requirements. Fig. 1 shows an example of an urban environment. The vehicle’s dynamic environment necessitates the design of multiple traffic scenarios to optimise the DL model effectively. The project also places high importance on functional safety, robustness, and security, given the critical nature of automotive applications and the use of open communication links.
Comparative analyses of computational loads across various hardware setups were conducted. These setups ranged from using only the sensor’s processing unit to incorporating the CCU, expanding to include the edge unit, and finally performing computations entirely on the edge. The results indicated that different combinations of processing resources could reduce latency while maintaining AI/DL model accuracy.
The project aims to focus on the total energy consumption of different configurations, including the energy required for sustaining communication links. This future work seeks to find an optimal balance between processing efficiency and the stringent safety requirements inherent to the P-AEB system.
Industrial IoT domain
The VEDLIoT project is at the forefront of integrating Deep Learning into IoT for industrial applications, specifically focusing on two key areas: Motor condition monitoring and series arc fault detection (AFD) in low-voltage direct current (LVDC) systems. These areas represent crucial aspects of industrial IoT, involving predictive maintenance and anomaly detection, each with distinct AIoT solution requirements.
In motor condition monitoring, the project emphasises the importance of smart field devices (SFDs) for data collection. These devices monitor various aspects of motor operation, such as operational, thermal, and mechanical conditions. Data collected by SFDs, which include operational hours, cooling system status, and the state of mechanical components like bearings, is vital for ensuring motor reliability and longevity. However, network congestion and limited bandwidth in large-scale IoT systems pose a significant hurdle. To address this, on-site data processing within SFDs using Deep Learning is explored to reduce the volume of data transmitted wirelessly, necessitating the integration of DL algorithms into resource constrained SFDs while maintaining hardware power efficiency. Fig. 2 shows the solution designed within the project VEDLIoT, where the customised SFD integrates essential sensors and an AI-accelerator.
The other major focus area, series arc fault detection in LVDC systems, deals with the complexities of detecting series arc faults, which are notoriously difficult to identify due to their low current draw. AI’s adaptability and learning capabilities present a significant advantage over traditional statistical methods. The VEDLIoT project’s efforts in developing AI-powered solutions for LVDC systems include a specially designed test-bench, capable of generating electrical arcs and collecting relevant data for AI analysis. This is depicted in Fig. 3, showing the test-bench built for AI system development and evaluation, including the AI accelerator and microcontroller setup.
Both use cases face the overarching challenge of insufficient data, especially for anomalies. The VEDLIoT project addresses this through test-benches for data collection and validation and is developing a comprehensive framework aimed at enhancing data collection efficiency and AI algorithm explainability.
The system structure of real-time AFD is depicted in Fig. 4, the coloured block marks technology researched by VEDLIoT that can help accelerate AI-based solution development. In the arc fault detection use case, the development procedure is expedited with the provided streamline. A test bench designed according to the standard U1699B is built to generate arc faults and collect current data. Based on the detailed requirements derived from problem analysis and benchmarking results from the VEDLIoT project, the hardware Nvidia Jetson Xavier NX is chosen to process current data in real-time. Deep learning model pruning has been employed to ensure model inference within the target detection time. The pruning software, developed under the VEDLIoT project, has yielded impressive results, resulting in up to a 70% reduction in model runtime while maintaining an arc fault detection accuracy of 98%.
In motor monitoring, the energy efficiency of SFDs, expected to last two years on battery power, is a key focus. The project addresses this by employing the MAX78000 microcontroller with a built-in CNN accelerator, optimising power consumption and processing efficiency.
Overall, the VEDLIoT project is making significant strides in advancing AI and DL in industrial IoT systems, addressing specific challenges in motor condition monitoring and arc fault detection, and developing frameworks and tools for efficient data collection, hardware selection, and software optimisation.
Smart Home domain
In the VEDLIoT project’s smart home use case, a smart mirror was developed, prioritising data protection through local processing. This mirror, designed to replace traditional ones in areas like dressing rooms or bathrooms, integrates a PC monitor display within a custom acrylic casing, complemented by an Intel Realsense 3D stereo camera. It functions to display information such as weather forecasts, train times, and personal appointments, while also controlling other smart home devices.
Interaction with the mirror is contactless and intuitive, using the stereo camera to capture depth and RGB images. Machine Learning (ML) algorithms recognise objects, faces, and gestures for user interaction. Facial recognition and tracking are used for user identification, and hand gestures control the mirror. Additionally, a voice assistant utilising natural language processing (NLP) enables voice command control.
To ensure privacy, all AI processing occurs locally on the device, using open-source software. This approach avoids sending sensitive user data to external servers. Object detection is based on the Yolo neural network (version 7), gesture detection uses YoloV7’s tiny model, and face recognition employs a Siamese network with RetinaFace detector output. The Robot Operating System 2 (ROS2) middleware facilitates modular software integration, while an open-source MagicMirror framework provides an interactive HTML frontend (see Fig. 5).
Balancing local AI model computation with power consumption is crucial. The device uses a Nvidia Jetson Orin AGX system-on-module and a Hailo8 AI accelerator, achieving significant power efficiency with a consumption of 45W and maintaining a 30 FPS framerate.
Voice control is enabled through a secure microphone based on an Arduino Nano RP2040 Connect. This microphone activates upon a spoken hotword, sending encrypted audio data to the mirror via TLS WiFi. The mirror then transcribes and processes voice commands internally, ensuring security and privacy in sensitive smart home locations (see Fig. 6). This combination of encrypted communication and local processing of image and audio data upholds the security and privacy of user data.
Additional use cases
Besides the primary use cases of VEDLIoT discussed earlier, the project has expanded its scope to include ten additional use cases from various application domains such as agriculture, industrial, automotive, and medical fields. As shown in Fig. 7, these projects cover a diverse range of applications, including AI for driving school purposes, laser welding, and pollen analysis, among others. Each of these projects has incorporated various elements of VEDLIoT technology into their specific applications. This integration includes the use of the hardware platform, certain components of the middleware, security features, or the requirements engineering methodology developed within VEDLIoT. More information regarding the different open call projects is available on the VEDLIoT website.
Outlook
VEDLIoT addresses the challenge of incorporating Deep Learning into IoT devices that have limited processing power and require low energy use, emphasising the need for power-efficient computing. The VEDLIoT AIoT hardware platform provides custom hardware and additional accelerators suited for AIoT applications, spanning from embedded systems to edge and cloud computing. The project integrates innovative methods in requirements engineering with safety and security concepts to overcome the difficulties of applying Deep Learning across its entire framework. These ideas are rigorously tested in essential industrial areas like automotive, automation, and smart homes. Each case study demonstrates the benefits of integrating AI, ML, and specifically DL techniques, developed by VEDLIoT, in modern AIoT applications distributed throughout the computing spectrum. This integration aims to enhance functionality and performance, while also boosting key indicators like efficiency, security, and safety.
The VEDLIoT project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957197.
Please note, this article will also appear in the sixteenth edition of our quarterly publication.
