The number of new user interface technologies entering the consumer market is currently growing faster than ever. Computer Game Consoles and Mobile Devices are being equipped with new sensors and display technologies. These technologies aim to provide more “natural” user interaction and are therefore called “Natural User Interfaces” (NUI).
In parallel, the use of computer technology is currently changing due to the rise of cloud based services. This allows us to create decentralized interaction services where sensing (capture), processing and displaying real-time data can be done using internet technologies.
In this paper we present a system for sound source localization and directional audio based on two consumer NUIs, namely the Kinect extension module for the Microsoft XBOX 360 game console which provides a video camera, a depth camera and a microphone array together with a Google Android mobile phone equipped with a touch sensitive display and an accelerometer. Audio is captured by the microphone array from a Microsoft Kinect which is connected to a personal computer. A beamforming algorithm enables the microphone array to capture audio from a certain direction and suppresses audio from other directions. The beamforming can be computed for several directions at the same time which makes it possible to locate sound sources.
The directed audio signal is then streamed to the Google Android mobile phone wirelessly and played back on the device. By moving a finger over the touchscreen the user can control the direction of the perceived audio signal. To help the user in determining the desired beam angle a video stream from the Kinect video camera is also displayed on the mobile device. This enables the user to directly point to an object captured by the Kinect and listen to it.
AutorInnen
Bernhard Rettenbacher, JOANNEUM RESEARCH
Christoph Hechenblaikner, Technische Universität Graz
Moritz Fišer, JOANNEUM RESEARCH
Maria Fellner, maria.fellner@joanneum.at Austria Joanneum Research Forschungsgesellschaft mbH