This repo contains a prototype that demonstrates how to stream a vehicle's OBD-II data to HoloLens using Azure Sphere over MQTT, without the latency involved with sending the data to the cloud and back to the HoloLens.
We want to set the Digital Field Worker of the future free of serial cables, diagnostic ports and telnet!
The Digital Field Worker of the future will be required to diagnose and repair complex, expensive, heavily instrumented equipment. Existing solutions such as HoloLens Guides and Remote Assist are extremely helpful. But what if we could also infuse real-time equipment data and the ability to interact with that equipment into a Holographic application?
By adding real-time data, we empower the Digital Field Worker to read equipment trouble codes, see real-time performance data, power off equipment, reset equipment, and interact with the equipment all through the HoloLens, even when disconnected from Azure!
This prototype demonstrates a scaled-down version of the Digital Field Worker scenario; specifically, an automotive mechanic diagnosing vehicle issues using a HoloLens. Real time engine data from the OBD-II port is streamed to a HoloLens application using Azure Sphere over the MQTT protocol to an MQTT broker and displayed holographically while the mechanic views the engine. The mechanic can potentially clear the check engine light and perform other commands on the vehicle holographically. Dynamics 365 guides can be added to provide repair instructions for mechanic.
This specific scenario can be expanded to many other types of equipment and industry verticals. For example, an aircraft mechanic can holographically view real-time data streaming from an idling aircraft engine while standing at a safe distance. A repair technician for a office tower backup generator can diagnose, start, stop, and interact with the generator holographically.
The MQTT broker software can run on any device installed at the facility and receive telemetry from numerous pieces of equipment at the facility. As the worker, wearing a HoloLens, walks up to a piece of equipment, the HoloLens identifies the equipment (e.g. QR Code, etc.) and connects to the MQTT broker and MQTT topic the equipment is connected to.
The architecture for this project is depicted below and described in the following sections:
- Unity 2019.4.X (This project used 2019.4.1f1)
- Visual Studio 2019
- Windows SDK 18362+
- Universal Windows Build Support for Unity
After cloning or downloading the repository, open the “HoloLens” project folder in Unity. More information on how to deploy to HoloLens can be found here https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/BuildAndDeploy.html
Ensure that the target platform is Universal Windows Platform:
- Open menu : File > Build Settings
- Select Universal Windows Platform in the Platform list
- Click the Switch Platform button
Ensure that Windows Mixed Reality SDK has been added
- Navigate to Edit > Project Settings, Player
- Under XR Settings in the UWP tab, make sure Virtual Reality Supported is enabled, - and the Windows Mixed Reality SDK has been added to SDKs.
Build a Visual Studio solution
- Open menu : File > Build Settings
- Press build
Build to HoloLens
- Plug the HoloLens into the computer using the USB cable
- Open Visual Studio solution
- Update Solution settings
- Solution Configuration: Release
- Solution Platform: X86 for HoloLens 1 or ARM64 for HoloLens 2
- Target: Device
- Deploy to Device
Update MQTT Settings
- Update MQTT settings by navigating to the settings menu in the HoloLens app, updating the values, and pressing the connect and subscribe buttons.
- MRTK v2.3.0 - https://github.com/microsoft/MixedRealityToolkit-Unity/releases
- Cross-platform toolkit for building Mixed Reality experiences.
- JSON .NET For Unity - https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347
- HoloLens compatible library for serialization.
- M2Mqtt - https://www.nuget.org/packages/M2Mqtt/
- Library for the MQTT client. We used .net 4.5 for this project.
- Engine model - https://sketchfab.com/3d-models/disassembled-v8-engine-block-3026bd87ca3945d6829d24c86ce695f0
The OBD-II reader is comprised of:
- Azure Sphere MT3620 running the
Sphere_HighLevelApp\. - SparkFun Car Diagnostics Kit an OBD to UART interpreter
- OBD-II to DB9 Cable
- Breadboard (optional)
The following is a schematic view of the wiring.
Here are some pictures of the assembled OBD-II reader.
If you are new to Azure Sphere, check out this page for Quick Start
You may refer here for Sphere Wifi configuration
-
Even if you've performed this setup previously, ensure you have Azure Sphere SDK version 20.04 or above. At the command prompt, run azsphere show-version to check. Install the Azure Sphere SDK as needed.
-
Connect your Azure Sphere device to your computer by USB.
-
Enable application development, if you have not already done so, by entering the following line at the command prompt:
azsphere device enable-development
See the following Azure Sphere Quickstarts to learn how to build and deploy this sample:
Add MQTT broker IP address in app_manifest.json (allowed_connection) and main.c (mqttConf_brokerIp)
The code publishes coolant temperature, RPM and fuel to the vehicle/telemetry MQTT topic every 500ms. You can change the polling interval by changing MQTT_PUBLISH_PERIOD. Note: polling too frequently or for too much information will make the OBD-II reader return erratic results or hang.
- The MQTT client library used in the Sphere comes from https://github.com/LiamBindle/MQTT-C/
- Eclipse Mosquitto MQTT broker
In reality, the MQTT broker can be any MQTT 3.1.1 compliant broker running on virtually any type of device. MQTT 5 was not tested.
- HoloLens - Contains the Unity project for the HoloLens app that includes the UI and the code to receive and display data from an MQTT broker.
- Sphere_HighLevelApp - Contains the Azure Sphere source code that polls a vehicle's OBD-II port for Coolant Temperature, Current RPM and Fuel and sends the data to an MQTT broker over Wi-Fi.
- mkr1000 - Contains Arduino source code for an mrk1000 that polls a vehicle's OBD-II port for Coolant Temperature, Current RPM and Fuel and sends the data to an MQTT broker over Wi-Fi. A block diagram with wiring instructions is here.
- Update Azure Sphere firmware to make it an MQTT subscriber so HoloLens can send additional OBD-II codes and commands back to the OBD-II port (e.g. clear check engine light).
- Incorporate QR code support to allow HoloLens to lookup vehicle information automatically, and to spatially anchor the menu (assuming QR code is a sticker, stuck in/around the engine compartment).
- Incorporate Dynamics 365 Guides into the HoloLens app to provide repair instructions for mechanic.
For more background information, visit the starthere repo.