SoundSee – insight with Audio AI

Modern sensors may hear better than humans, but they lack the ability to understand sound the way humans do. Bosch Research set out to develop robust, scalable audio signal processing and machine learning algorithms that can understand sound patterns.

Along with human speech, two key classes of sound are environmental sounds and machine sounds. Environmental sounds are all around us – a car passing by, a door closing or glass breaking. Machine sounds originate from machines – engines, tools or motors, to name a few. These sound patterns encode various physical events of interest to be interpreted for practical use. Did the door latch properly when it was closed? Does the fan on a cooling unit sound like it is about to malfunction?

But SoundSee goes beyond applications in space. It applies widely across a range of industries and settings thanks to the inexpensive and versatile nature of audio sensing, which fits pretty much anywhere while staying out of sight.

Predictive maintenance in industrial settings is an example where SoundSee can be impactful by augmenting existing sensing capabilities. For instance, in a Hybrid Automatic Voltage Control system, SoundSee could analyze a motor’s sound and predict a malfunction before it occurs – learning from subtle deviations in noise signatures. So it could be an additional layer of monitoring solution for early warning systems that offer incredible value for industry by reducing downtime and saving extensive repair costs.

There are audio AI use cases in healthcare as well. That is why healthcare is another exciting application domain for SoundSee. The human body generates sounds such as the heartbeat or breathing which can contain clinically relevant information. SoundSee could contribute to wholesome, data-driven healthcare decisions by listening to the body. It could also register physiological sound indicators of distress and give audible alarms when a person requires assistance. The ability to detect panic in a person’s voice or a cry for help could make the difference in an emergency.

As an all-round talent, SoundSee’s audio AI could also be integrated as a value-add feature in building technology solutions, particularly in the context of physical safety and security, e.g. by detecting and localizing threat events like alarms or glassbreak – an all-round talent.

Adapting the SoundSee audio AI to a given environment and ensuring robust performance is a challenging task. To address that, high-quality data is needed to train the audio AI. Bosch Research scientists use a combination of classical signal processing and data-driven AI (machine learning) to solve unique challenges associated with each application scenario under consideration. They need a well-defined use case and performance metric to determine what sounds the audio AI will need to be trained on together with appropriate data labels/annotations (human assigned labels and/or other sensory signals as supervisory cues). Both are critical factors that support machine learning.

Recently, the SoundSee team had promising outcomes in a collaborative project involving predictive maintenance with Bosch Powertrain Solutions. The audio AI was trained using a multi-modal dataset that included audio and torque signals before and after artificially-induced damage to six fuel pumps. SoundSee demonstrated that audio AI could be used for inferring other signal modalities (e.g. torque, fuel pressure) with surprisingly high fidelity, which, in turn, strongly correlates with the operating state or malfunctions of the pumps. Such “virtual sensing” capability with Audio AI is a promising direction towards augmenting current sensor data analytics solutions for improving predictive maintenance.