Human-Machine Interfaces, intelligent systems and cultural heritage
Maria Giulia Losi, PhD, Project Leader and Interaction Designer at RE:Lab; Silvia Chiesa, PhD, Project Manager in the R&D group of RE:Lab; Annalisa Mombelli, R&D group of RE:Lab, Reggio Emilia
Human-Machine Interfaces, intelligent systems and cultural heritage
Perception might be reality
In today’s context, related to health emergency, some aspects of technological design assume a peculiar role. The first is the theme of distance; the second is the theme of the extension of sensory capacity through digital technologies; and the third is the recognition of the state of the subjects. These three sets of factors, the need to manage distance, the need to extend technological sensoriality into the distance, and the need to recognise in the distance not only the sensoriality but also the state of the user—whether a patient, driver or viewer—thus constitute a new axis of technological design.
Picture left: Interfaccia suono, Copyright: RE Lab
A new anthropocentric setup for technologies
This new axis prefigures the attempt to fill the gaps, the attempt to relaunch an idea of technology that can bring people together, even in this current social framework of difficulty. Therefore, starting from this idea of rapprochement, a new anthropocentric approach to technological performance can be founded, which is undoubtedly one of the central axes of a New Renaissance, based on the integration between the anthropocentric culture of the Renaissance tradition and the potential 4.0 of the technologies mentioned above: those intended to extend the sensory framework; those intended to increase the recognition of the subject and respect of his emotional and cognitive state,; and finally those oriented to extend his ability to live experiences not necessarily in presence.Starting from this premise, RE:Lab, in the framework of the activities of the Emilia Romagna Region, wanted to compose this complex puzzle with diversified initiatives within several projects. First of all, in the midst of the pandemic, the experience of restitution of a phenomenon of intense bodily relations such as dance was sought. Subsequently, the idea of extending the sensory capacity in the relationship, born even before the health emergency took shape, was resumed and developed, and then relaunched during the pandemic experience through the concept of an Atelier, that is, a creative space that takes into account the management of sound manipulation. Finally, we added the idea of including in the processes of technological relations the recognition of the state of the user, not only cognitive but also emotional, so that his emotions allow to recognise the state of use and on the basis of this to orient and design systems of safer and more effective interactions.
A Living Lab for Novel Human-Machine Interactions – A New Grammar for Dramaturgy and Choreography in Digital Space
How can user experience design foster innovation processes in the dance and live performance sector?In a historical moment like the current one, where distance has become one of the hallmarks of experience, the performing arts and dance sector has been put to the test. At the beginning of 2020, the Aterballetto Dance Foundation asked itself some questions:
– Can the emotion of the performance exist without the performance?
– Is it possible for a dance performance to engage the viewer while the dancer is not physically present?
– Is there a safe-conduct to ensure that art can express its expressive power in the absence of contact and in the presence of „distancing“?
In the most uncertain months of 2020, marked by the closure of theatres, the Aterballetto Foundation has managed, through the project Virtual Dance for Real People1, in collaboration with RE:Lab, to imagine a different user experience, through the techniques of Cinematic Virtual Reality and the use of virtual reality visors (such as Oculus Quest).
The design of the user experience has led not only to the creation of a new format for viewers but also to the formulation of a new grammar involving choreographers, interaction designers, and video experts, in identifying a common language and a new way of thinking about choreography. The spectator in fact has been imagined in a central position, in which the viewer can dance together with the dancers.
The search for a relationship of closeness with the spectator has led us to investigate the immersive aspects of technology, also through the use of spatialised sounds, without making the dance abstract: the choreography, conceived for the final technological instrument, remains at the centre of the experience, able to release new and unexpected emotions.
Picture right: MicroDanza Meridiana, Fondazione Nazionale della Danza / Aterballetto, Coreografia: Diego Tortelli, Danzatrici: Casia Vengoechea, Annemieke Mooji, Foto Celeste Lombardi, Technological development and user experience design RE:LAB
Dynamic Mapping—Sound Atelier in distance environments
Many studies have analysed how much music and the sound environment that surrounds us can influence cognitive and linguistic development.How can we investigate, through new immersive audio technologies, the relationship between children and sound, considered as a compositional element of musical language but also of language itself?
From the encounter between engineering research in the area of the spatialisation of sound and the mission of Reggio Children (International Center for the Defense and Promotion of the Rights and Potentialities of All Children), the Sound Atelier2 was born, a laboratory in which children can experiment with a new and complete concept of sound fruition, for which the new child-friendly application EUPHONIA was designed.
The design of the user interface was developed for and with the children, who were involved as design partners to verify their understanding of a 3D interface. In fact, through the support of the figures of the atelierista in collaboration with the designer, it was possible to highlight the difficulties that children had to orient themselves in a virtual environment and a new UX design was redefined for the tablet application called EUPHONIA from the union of the ancient Greek words EU (true) and PHONIA (sound). The app’s interface reproduces the room that houses the atelier in order to give the user an idea of the position of the sound in the space. From a dynamic acoustic library it is possible to choose sounds, associated with representative images, and position them in the space, thus composing 3D soundscapes.
Technological-pedagogical research on sound in space has thus taken its most natural form as the „Atelier of Sound”, an immersive audio environment/laboratory in which children can experiment and dynamically explore the entity “sound“ in all its dimensions.
On the one hand, therefore, it is an expression of the Reggio Emilia Approach, a socio-constructivist educational philosophy based on research and the values of lifelong learning that strongly believes in the dialogue between theory and practice, and sees the child in his relationships with others, as an active part of the learning process. On the other hand, the interest of the multidisciplinary research group in the rendering of sound spatialisation has also approached the concepts of Acoustic Ecology described by the teacher/composer Murray Schafer in his great work The tuning of the world (R.Murray Schaefer, 1985): The soundscape is our acoustic environment, the ever-present noises we all live with. The author suggests that today we suffer from acoustic overload and are less able to hear the nuances and subtleties of sound. Our task, he argues, is to listen, analyse, and make distinctions despite the noise pollution.
The result of this Audio Interaction Lab is a dynamic 3D sound scene whose content and spatial organization is controlled in real time by the children, enhancing their awareness and sensitivity of sound in space.
Picture above: Euphonia, (particolare dell’App, schermata dell’ Area ambiente sonoro), progetto Atelier del suono/Reggio children, Technological development and user experience design RE:LAB
Spillover effects of Sound Design: Increasing driving safety and comfort through the recognition of one's emotional state: from the HU-DRIVE experience to the NextPerception project
Sound Design and its value-added in Human Machine Interaction are not limited to the Cultural and Creative Sectors and Industries—for illustration we review one project in the automotive sector: Several studies in the literature show a strong relationship between emotions and traffic accidents. In fact, negative emotions, such as fear, anger, disgust, or sadness, increase the likelihood of exhibiting dangerous behaviours while driving (Magaña et al., 20203).
In recent years, affective technologies have been developed that offer new opportunities to improve road safety by using the recognition of the driver’s emotions and state, and their regulation through appropriate interaction strategies on board the vehicle. Affective user interfaces are those interfaces that are capable of understanding through psychophysiological measures, conversational analysis, and facial expression recognition human emotions while interpreting, adapting, and potentially responding appropriately (Braun, Weber, & Alt, 2020).4
Within the HU-Drive5 project, the HUman Driver Assistance System presented at CES 2021 in collaboration with the startup Emoj, RE:Lab presents an innovative solution to increase safety, quality and comfort of driving. The HU Drive system, in fact, automatically recognises the driver’s emotional and cognitive state and adapts the vehicle interface in real time.
Novel Human-Machine Interactions based on trust and transparency
Recently, new trends are emerging in the topic of human-computer interaction. In order to achieve better levels of trust in automation, the „negotiation-based“ interaction approach has been studied (Koo et al., 20146; Castellano et al., 2018)7 which consists in providing the user with explanations instead of warnings, in order to increase the transparency of the system.
Taking this line of research as a basis, RE:Lab is carrying out the activity of developing an interface able to adapt to the state of the driver within the NextPerception project. The main objective of the project is to develop a Driver Monitoring System, able to classify both the driver’s cognitive states (distraction, fatigue, workload, drowsiness) and his emotional state (anxiety, panic, anger) as well as the driver’s position inside the vehicle. What can we learn here about the role of culture in digital societies and take spillover back to Cultural and Creative Industries?
In conclusion, RE:Lab’s research activity focused on the design of innovative strategies of interaction with emerging technologies, including artificial intelligence systems that do not want to replace people but put users, at best humanism, at the centre stage to amplify human capabilities.New conceptual models, new strategies, new metaphors will be born in future research about man and machine interaction, making the user more aware of what happens when interacting with these systems, and which are the expression of an interweaving between science and art, technology and cognitive and emotional languages: all elements at the basis of the redefinition of a New Contemporary Renaissance.
References
1 Trailer VIRTUAL DANCE FOR REAL PEOPLE (2021): https://youtu.be/h9u1Cs2GMf8
2 Federica Protti, Simone Fontana, Elena Maccaferri, Claudia Giudici, and Roberto Montanari. 2013. Audio-interaction lab: designing an immersive environment to explore the acoustic ecosystem with a tablet interface. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI (CHItaly ’13). Association for Computing Machinery, New York, NY, USA, Article 19, 1–8. DOI:https://doi.org/10.1145/2499149.2499160
3 Magaña, Víctor Corcoba, et al. „The effects of the driver’s mental state and passenger compartment conditions on driving performance and driving stress.“ Sensors 20.18 (2020): 5274.
4 Braun, M., Weber, F., & Alt, F. (2020). Affective Automotive User Interfaces–Reviewing the State of Emotion Regulation in the Car. arXiv preprint arXiv:2003.13731.
5 EMOJ & RELAB present HU-Drive @CES2021, the innovative solution presented at CES 2021 by EMOJ and RE:Lab: https://www.youtube.com/watch?v=SQ_EWRAM23o
6 Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., & Nass, C. (2015). Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM), 9(4), 269-275.
7 Castellano, A., Landini, E., & Montanari, R. Un nuovo paradigma di interazione per la guida cooperativa: il progetto AutoMate.
Maria Giulia Losi
Maria Giulia Losi, PhD in „Humanities and Technologies: a research integrated path“. She is a project leader in RE:Lab with many years of experience in the design and development of Human-Machine Interaction systems and EU research projects. At RE:Lab since the beginning of 2011, she has been involved in several projects as Project Manager and Interaction Designer in different domains, from Automotive and Off-HIghway vehicles HMI and IoT Apps to Cultural Heritage interactive exhibitions design. Key publications are reported in the following:
1. Leandro Guidotti, Maria Giulia Losi, Roberto Montanari, Francesco Tesauri, “Motorsport Driver Workload Estimation in Dual Task Scenario”. IARIA Cognitive 2011
2. Daniele Pinotti, Fabio Tango, Maria Giulia Losi, Marco Beltrami. “A model for an innovative lane change assistant HMI”. HFES 2013 Conference Proceedings
3. Corazza F., Snijders D., Arpone M., Stritoni V., Martinolli F., Daverio M., Losi M.G., Soldi L., Tesauri F., Da Dalt L., Bressan S. „Development of a tablet app to optimize the management of pediatric cardiac arrest: a pilot study“. IPSSW2020 – 12th International Pediatric Simulation Symposia and Workshops
Annalisa Mombelli
Annalisa Mombelli, Graduated with a Master’s Degree in Conservation of Cultural Heritage at University of Parma, defending a thesis about the innovation in publishing by Bodoni with his illustrated masterpiece “Epithalamia”. She also graduated in Advanced European Studies at European College of Parma. She worked as a freelance coordinator for private and public cultural projects. Now, she works in the R&D group of RE:lab, following activities in the field of education and culture.
Silvia Chiesa
Silvia Chiesa, PhD in Psychological, Anthropological and Educational Science at the University of Turin, defending a thesis concerning the spatial cognitive abilities in disabled persons. She expanded her activities outside the academic field working in the context of Human-Machine Interaction and more in general in the User Experience Research. She is Project Manager in the R&D group of RE:Lab. Here, she led international projects and activities concerning usability and acceptance assessment, in the domains of transports, household appliances and healthcare.
