June 05, 2022 by Timothy Malche
Developing a TinyML solution for the Shipping Industry that allows users to track the status of a shipment/package in real time.
Artificial Intelligence (AI) and Machine Learning (ML) enable us to solve most of our everyday problems efficiently, giving us the power to create intelligent devices that can make decisions by themselves in real time. In this project, I'd love to demonstrate how TinyML (Tiny Machine Learning) works for a global industry such as Smart Shipment.
With the help of my project, users will be able to track the status of their package/shipment whether the package/shipment is:
There may be more classes for STATUS of a package and not just limited to those mentioned above. First, let's watch a video demo to see how it works:
Consumers always want to keep track of their packages and their proper handling during shipment. As such, tracking the status of the packages enables users to make sure that everything will arrive safely.
The project "Smart Shipment" is based on the Internet of Things (IoT) and Artificial Intelligence (AI). With the help of AI, the status of a package can be identified. IoT is used to send the status in real time to a cloud server which can be accessed via a Web App over the Internet.
A TinyML model is embedded in a microcontroller (MCU). Each package/shipment consists of an MCU. The inferencing occurs on the device itself and the result is transmitted to the server as a simple text stream.
This approach, when the machine learning model is run on the device (and not in the cloud), allows you to significantly reduce the amount of information sent to the cloud. As such, we send only triggers of the changed state to the cloud instead of constantly sending data to determine the state (which usually increases the cost in such projects.).
The required hardware and software to build this project include:
Hardware Requirements
Software Requirements
Any MCU can be used for this project as the TinyML model built with Neuton.ai can be embedded and executed on all types of MCUs. For this project, I will use M5Stack Core 2 AWS IoT Kit which is an ESP32-based MCU. It has the following features:
The accelerometer and gyroscope sensing units are required for this project. Accelerometers measure linear acceleration along with one or several axes. A gyroscope measures angular velocity. To track the status of this package, these sensors are utilized to measure linear acceleration and orientation.
M5Stack Core 2 AWS Kit is built in 6-Axis IMU (Inertial Measurement Unit) Unit MPU6886. The MPU6886 is a 6-axis attitude sensor with a 3-axis gravity accelerometer and a 3-axis gyroscope, which can calculate tilt angle and acceleration in real-time.
The data transmission in this project is done using the MQTT protocol. MQTT stands for Message Queuing Telemetry Transport. MQTT is a machine-to-machine Internet of Things connectivity protocol. It is an extremely lightweight and publish-subscribe messaging transport protocol.
MQTT Publish / Subscribe Architecture
The Smart Shipment Project requires the following steps:
Here's the walkthrough of the steps specified above:
Prerequisite: Please set up Arduino environment for M5Stack Core 2 as specified in this article.
Open the Arduino IDE and use the following code shipment_data_collection.ino to collect the data.
When you execute the code you will see information in the serial monitor as below.
Copy the accelerometer and gyroscope data from the serial terminal to a text file and save it as CSV.
Using the above code collect the data for the following classes:
For a single class, the firmware collects 10 seconds of data with a delay of 20 ms. The 10-12 iteration of data needs to be collected for each class separately. In this way, you will have to create 12.csv files for each class.
Each class indicates the status of the shipment/package as shown here:
Collect data for all classes and combine the data into a single CSV file by adding one more "target" column at the end. Where the target column contains value 0 for stationary, 1 for moving, and 2 for picked. For example, combine 12 CSV files for stationary class with target column as 0, combine 12 CSV files for moving class with target column as 1, and so on. The data is collected it has to be prepared in the following structure as training_data.csv
Sign in to the free-to-use tool, Neuton.ai, and click on "Add New Solution".
Give the name to your solution as "Smart Shipment" and description, after that click Next.
You will have to upload a training dataset that you have prepared.
After the dataset is uploaded, click OK.
After that, select a target variable and click Next.
In the next screen, select task type "Multi Classification", metric "Accuracy", and enable "TinyML".
Scroll down to TinyML settings and select input data type as "FLOAT32", enable "Digital Signal Processing", select the checkbox "Window size auto selection", and then click the "Start Training" button.
The training will be complete and you will see the Target Metric Accuracy. Now you can download the C Library that will be used to program the Smart Shipment Firmware.
The embedding size of TinyML model developed using the neuton.ai is only 0, 445 kB. When I was trying to build the model for this project with Tensor Flow Lite I got model of 8, 95 kB. So with the same accuracy the model developed using the neuton.ai is 20 times smaller in size, that is critical when you embed the model to small MCU:
Create a root project directory for your Arduino "smart_shipment" project with the same name as your Arduino source file, create a subdirectory "src" and your smart_shipment.ino source file inside the root directory and extract and copy all the files that you have downloaded as "C source library" from neuton.ai inside the "src" subdirectory. Your directory structure must look like the following:
code for smart_shipment.ino
The Smart Shipment Web App allows users to view the status of the shipment in a user-friendly way. The following is the code for Web App.
Web App displays the result as shown below.
The project creation and execution are summarized in the following diagram.
It consists of the following two stages:
The Model Building Stage has four steps
In the Model Execution stage, both the Smart Shipment Firmware (MQTT Client Publisher) and Web App (MQTT Client Subscriber) communicate with each other via the MQTT broker. The following is a detailed diagram that shows this communication process.
The Live Shipment Data is fed using the accelerometer and gyroscope to the MCU running the TinyML Model. After the inferencing is done, the data is published to the MQTT Broker on the topic "STATUS" by the MQTT Client Publisher running on MCU. The Publisher is connected to the Broker over a TCP port. The Web Application running MQTT Client Subscriber connects/subscribes to MQTT Broker over Websocket port at the same "STATUS" topic. As soon as data is published by the MQTT Publisher (MCU), it is received by the MQTT Subscriber (Web App).
Maker, Educator, Researcher
