A European consumer electronics brand that manufactures a wide range of devices for in-home use. As part of its Digital Transformation activities, the company is looking for a technology solution to present user manuals in a simple and engaging manner.
T&M (Time and Materials)
Kanban
iOS Developers
Project Manager
Deep Learning Developer
3D Designer
UI/UX Designer
QA Engineer
Augmented Reality renders graphical information on a three-dimensional scene. The technical challenge here is “attach” superimposed imagery to a physical object in the real world. This is made more difficult by the fact that AR development tools may fail to detect physical objects with a low-texture, glossy, or transparent surface. Environmental factors, such as light intensity, can also affect object detection accuracy.
Softeq suggested building a mobile app prototype with ARKit to test the feasibility of the concept and devise a technical vision for future implementations. The prototype would identify several models of dishwashing machines from our client’s catalog and support the following use cases:
Technology-wise, the application was to meet the following performance requirements:
When choosing the appropriate technology stack for the iOS app, our team had to address three challenges:
We tested several ARKit configurations, including Object Detection and Image Recognition, to determine whether we could accomplish the tasks solely with the native iOS Augmented Reality tools. Eventually, the team decided to only use the ARKit World Tracking configuration to position AR objects in the physical space.
Additionally, we needed a 3rd-party library capable of calculating the distance and rotation of an object in the real world relative to the camera, and for this we chose OpenCV. To improve the framework’s pose estimation results, our iOS development team had to deactivate ARKit autofocus, perform extra calculations, and apply Kalman filtering to reduce the movement and distortion of virtual objects. We also implemented the iOS CoreML framework for real-time object detection, which would work in sync with a custom-trained neural network.
The ultimate goal was to teach Deep Learning algorithms to detect a dishwashing machine in live video data, identify its type, and deliver relevant troubleshooting tips. For that purpose, we set up a Google Cloud server and trained a MobileNetV2-SSD model using TensorFlow.
To train the neural network model, we photographed the dishwashing machines from different angles and in different proximity zones and applied various augmentations to expand the dataset from 300 to 4500 images. The team also annotated the photographs using the Supervise.ly platform to improve the detection of certain parts of a dishwasher front panel where AR objects would be displayed.
Softeq completed the project on time and in line with the customer’s requirements. The solution features a neural network that detects several types of dishwashing machines manufactured by our client, and an iOS mobile app prototype.
Here’s how the application works:
Our client will study the demand for Augmented Reality manuals within the consumer electronics market. If the customer decides to proceed, Softeq will design a market-ready solution with a distinct monetization model. We can also scale the application so that it supports more home appliances and use cases. Expansion to the Android mobile platform is also possible with a different technology stack.
