Technologies that can robustly and accurately analyse human facial, vocal and verbal behaviour and interactions in the wild, as observed by omnipresent webcams in digital devices, would have profound impact on both basic sciences and the industrial sector. They could open up tremendous potential to measure behaviour indicators that heretofore resisted measurement because they were too subtle or fleeting to be measured by the human eye and ear, would effectively lead to development of the next generation of efficient, seamless and user-centric human-computer interaction (affective multimodal interfaces, interactive multi-party games, and online services), would have profound impact on business (automatic market research analysis would become possible, recruitment would become green as travels would be reduced drastically), would enable next generation healthcare technologies (remote monitoring of conditions like pain, anxiety and depression), to mention but a few examples.
The overall aim of the SEWA project is to enable such technology, i.e., to capitalise on existing state-of- the-art methodologies, models and algorithms for machine analysis of facial, vocal and verbal behaviour, and then adjust and combine them to realise naturalistic human-centric human-computer interaction (HCI) and computer-mediated face-to-face interaction (FF-HCI) for data recorded by a device as cheap as a web-cam and in almost arbitrary recording conditions including semi-dark, dark and noisy rooms with dynamic change of room impulse response and distance to sensors. This extends and contrasts considerably the current state of the art in existing technological solutions to machine analysis of the facial, vocal and verbal behaviour that are used in (commercially and otherwise) available human-centric HCI and FF-HCI applications.
To wit, shortcomings of existing technologies for automatic analysis of human behaviour are numerous.
The main aim of the SEWA project is to address these shortcomings of the current HCI and FF-HCI technology and develop novel, robust technology for machine analysis of facial, vocal and verbal behaviour in the wild as shown by a single person or by two (or more) interactants.
As a proof of concept, and with the focus on novel HCI and FF-HCI applications, SEWA technology will be applied to:
Funding from the European Commision Horizon 2020 Programme.