How do humans listen?
Modelling auditory perception in the Two!Ears project
The goal of the Two!Ears project is to improve computer models of human hearing, and thereby advance our understanding of human auditory information processing. The project tackles this problem with a new approach: the human listener is seen as a multimodal agent who develops a world model through interactive listening and viewing. Coordinated by the TU Berlin, the international consortium has set the goal to develop an intelligent, active model of auditory perception and experience in the context of listening and viewing. The project received 3 million Euro funding from the European Commission.
Up to now, computer models of human hearing have tended to focus on the evaluation of the signals received at the two ears – in other words, they are signal-driven. In the new Two!Ears approach, the prediction of human understanding and action will be improved by including hypothesis-driven processing. In such a scheme, world knowledge steers and improves the signal-driven evaluation. The system will be enabled to receive auditory events and integrate them with visual and proprioceptive information, for instance indications about the head orientation or the listener’s position in a room. It will thus attempt to describe an acoustic scene in the same way as a human listener does, in terms of primary perceptual constructs such as loudness, timbre and spatial extent. Additionally, the Two!Ears system will determine the meaning of the scene; for example, whether the perceived sounds come from a known or unknown speaker. The Two!Ears system will be realized on a robotic system which is capable of actively exploring its physical environment to orient itself and move around.
The system is based on an open architecture which allows easy modification and extension. This is considered a crucial approach to enable widespread use within the scientific community of the auditory models and cognitive functions developed within the project. Two!Ears will have considerable impact on future developments in information and communication technologies where understanding and responding to sound is relevant. In addition, research in adjacent areas such as biology, medicine, perception and cognitive psychology will benefit from the outcomes of the project. As an example, hearing aid algorithms for listening in situations with many competing speakers might benefit from the insights of the project.
The Two!Ears project started in December 2013 and has a funding period of three years. The international consortium comprises the following nine research institutes from the EU and the US: TU Berlin, Ruhr-University Bochum, TU Denmark, TU Eindhoven, Université Pierre et Marie Curie Paris, University Rostock, University of Sheffield, CNRS Toulouse, Rensselaer Polytechnic Institute Troy.