Human-Machine Interfaces, intelligent systems and cultural heritage
Maria Giulia Losi, PhD, Project Leader and Interaction Designer at RE:Lab; Silvia Chiesa, PhD, Project Manager in the R&D group of RE:Lab; Annalisa Mombelli, R&D group of RE:Lab, Reggio Emilia
Human-Machine Interfaces, intelligent systems and cultural heritage
Perception might be reality
In today’s context, related to health emergency, some aspects of technological design assume a peculiar role. The first is the theme of distance; the second is the theme of the extension of sensory capacity through digital technologies; and the third is the recognition of the state of the subjects. These three sets of factors, the need to manage distance, the need to extend technological sensoriality into the distance, and the need to recognise in the distance not only the sensoriality but also the state of the user—whether a patient, driver or viewer—thus constitute a new axis of technological design.
Picture left: Interfaccia suono, Copyright: RE Lab
A new anthropocentric setup for technologies
This new axis prefigures the attempt to fill the gaps, the attempt to relaunch an idea of technology that can bring people together, even in this current social framework of difficulty. Therefore, starting from this idea of rapprochement, a new anthropocentric approach to technological performance can be founded, which is undoubtedly one of the central axes of a New Renaissance, based on the integration between the anthropocentric culture of the Renaissance tradition and the potential 4.0 of the technologies mentioned above: those intended to extend the sensory framework; those intended to increase the recognition of the subject and respect of his emotional and cognitive state,; and finally those oriented to extend his ability to live experiences not necessarily in presence.Starting from this premise, RE:Lab, in the framework of the activities of the Emilia Romagna Region, wanted to compose this complex puzzle with diversified initiatives within several projects. First of all, in the midst of the pandemic, the experience of restitution of a phenomenon of intense bodily relations such as dance was sought. Subsequently, the idea of extending the sensory capacity in the relationship, born even before the health emergency took shape, was resumed and developed, and then relaunched during the pandemic experience through the concept of an Atelier, that is, a creative space that takes into account the management of sound manipulation. Finally, we added the idea of including in the processes of technological relations the recognition of the state of the user, not only cognitive but also emotional, so that his emotions allow to recognise the state of use and on the basis of this to orient and design systems of safer and more effective interactions.
A Living Lab for Novel Human-Machine Interactions – A New Grammar for Dramaturgy and Choreography in Digital Space
How can user experience design foster innovation processes in the dance and live performance sector?In a historical moment like the current one, where distance has become one of the hallmarks of experience, the performing arts and dance sector has been put to the test. At the beginning of 2020, the Aterballetto Dance Foundation asked itself some questions:
– Can the emotion of the performance exist without the performance?
– Is it possible for a dance performance to engage the viewer while the dancer is not physically present?
– Is there a safe-conduct to ensure that art can express its expressive power in the absence of contact and in the presence of „distancing“?
In the most uncertain months of 2020, marked by the closure of theatres, the Aterballetto Foundation has managed, through the project Virtual Dance for Real People1, in collaboration with RE:Lab, to imagine a different user experience, through the techniques of Cinematic Virtual Reality and the use of virtual reality visors (such as Oculus Quest).
The design of the user experience has led not only to the creation of a new format for viewers but also to the formulation of a new grammar involving choreographers, interaction designers, and video experts, in identifying a common language and a new way of thinking about choreography. The spectator in fact has been imagined in a central position, in which the viewer can dance together with the dancers.
The search for a relationship of closeness with the spectator has led us to investigate the immersive aspects of technology, also through the use of spatialised sounds, without making the dance abstract: the choreography, conceived for the final technological instrument, remains at the centre of the experience, able to release new and unexpected emotions.
Picture right: MicroDanza Meridiana, Fondazione Nazionale della Danza / Aterballetto, Coreografia: Diego Tortelli, Danzatrici: Casia Vengoechea, Annemieke Mooji, Foto Celeste Lombardi, Technological development and user experience design RE:LAB
Dynamic Mapping—Sound Atelier in distance environments
Many studies have analysed how much music and the sound environment that surrounds us can influence cognitive and linguistic development.How can we investigate, through new immersive audio technologies, the relationship between children and sound, considered as a compositional element of musical language but also of language itself?
From the encounter between engineering research in the area of the spatialisation of sound and the mission of Reggio Children (International Center for the Defense and Promotion of the Rights and Potentialities of All Children), the Sound Atelier2 was born, a laboratory in which children can experiment with a new and complete concept of sound fruition, for which the new child-friendly application EUPHONIA was designed.
The design of the user interface was developed for and with the children, who were involved as design partners to verify their understanding of a 3D interface. In fact, through the support of the figures of the atelierista in collaboration with the designer, it was possible to highlight the difficulties that children had to orient themselves in a virtual environment and a new UX design was redefined for the tablet application called EUPHONIA from the union of the ancient Greek words EU (true) and PHONIA (sound). The app’s interface reproduces the room that houses the atelier in order to give the user an idea of the position of the sound in the space. From a dynamic acoustic library it is possible to choose sounds, associated with representative images, and position them in the space, thus composing 3D soundscapes.
Technological-pedagogical research on sound in space has thus taken its most natural form as the „Atelier of Sound”, an immersive audio environment/laboratory in which children can experiment and dynamically explore the entity “sound“ in all its dimensions.
On the one hand, therefore, it is an expression of the Reggio Emilia Approach, a socio-constructivist educational philosophy based on research and the values of lifelong learning that strongly believes in the dialogue between theory and practice, and sees the child in his relationships with others, as an active part of the learning process. On the other hand, the interest of the multidisciplinary research group in the rendering of sound spatialisation has also approached the concepts of Acoustic Ecology described by the teacher/composer Murray Schafer in his great work The tuning of the world (R.Murray Schaefer, 1985): The soundscape is our acoustic environment, the ever-present noises we all live with. The author suggests that today we suffer from acoustic overload and are less able to hear the nuances and subtleties of sound. Our task, he argues, is to listen, analyse, and make distinctions despite the noise pollution.
The result of this Audio Interaction Lab is a dynamic 3D sound scene whose content and spatial organization is controlled in real time by the children, enhancing their awareness and sensitivity of sound in space.
Picture above: Euphonia, (particolare dell’App, schermata dell’ Area ambiente sonoro), progetto Atelier del suono/Reggio children, Technological development and user experience design RE:LAB
Spillover effects of Sound Design: Increasing driving safety and comfort through the recognition of one's emotional state: from the HU-DRIVE experience to the NextPerception project
Sound Design and its value-added in Human Machine Interaction are not limited to the Cultural and Creative Sectors and Industries—for illustration we review one project in the automotive sector: Several studies in the literature show a strong relationship between emotions and traffic accidents. In fact, negative emotions, such as fear, anger, disgust, or sadness, increase the likelihood of exhibiting dangerous behaviours while driving (Magaña et al., 20203).
In recent years, affective technologies have been developed that offer new opportunities to improve road safety by using the recognition of the driver’s emotions and state, and their regulation through appropriate interaction strategies on board the vehicle. Affective user interfaces are those interfaces that are capable of understanding through psychophysiological measures, conversational analysis, and facial expression recognition human emotions while interpreting, adapting, and potentially responding appropriately (Braun, Weber, & Alt, 2020).4
Within the HU-Drive5 project, the HUman Driver Assistance System presented at CES 2021 in collaboration with the startup Emoj, RE:Lab presents an innovative solution to increase safety, quality and comfort of driving. The HU Drive system, in fact, automatically recognises the driver’s emotional and cognitive state and adapts the vehicle interface in real time.
Novel Human-Machine Interactions based on trust and transparency
Recently, new trends are emerging in the topic of human-computer interaction. In order to achieve better levels of trust in automation, the „negotiation-based“ interaction approach has been studied (Koo et al., 20146; Castellano et al., 2018)7 which consists in providing the user with explanations instead of warnings, in order to increase the transparency of the system.
Taking this line of research as a basis, RE:Lab is carrying out the activity of developing an interface able to adapt to the state of the driver within the NextPerception project. The main objective of the project is to develop a Driver Monitoring System, able to classify both the driver’s cognitive states (distraction, fatigue, workload, drowsiness) and his emotional state (anxiety, panic, anger) as well as the driver’s position inside the vehicle. What can we learn here about the role of culture in digital societies and take spillover back to Cultural and Creative Industries?
In conclusion, RE:Lab’s research activity focused on the design of innovative strategies of interaction with emerging technologies, including artificial intelligence systems that do not want to replace people but put users, at best humanism, at the centre stage to amplify human capabilities.New conceptual models, new strategies, new metaphors will be born in future research about man and machine interaction, making the user more aware of what happens when interacting with these systems, and which are the expression of an interweaving between science and art, technology and cognitive and emotional languages: all elements at the basis of the redefinition of a New Contemporary Renaissance.
1 Trailer VIRTUAL DANCE FOR REAL PEOPLE (2021): https://youtu.be/h9u1Cs2GMf8
2 Federica Protti, Simone Fontana, Elena Maccaferri, Claudia Giudici, and Roberto Montanari. 2013. Audio-interaction lab: designing an immersive environment to explore the acoustic ecosystem with a tablet interface. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI (CHItaly ’13). Association for Computing Machinery, New York, NY, USA, Article 19, 1–8. DOI:https://doi.org/10.1145/2499149.2499160
3 Magaña, Víctor Corcoba, et al. „The effects of the driver’s mental state and passenger compartment conditions on driving performance and driving stress.“ Sensors 20.18 (2020): 5274.
4 Braun, M., Weber, F., & Alt, F. (2020). Affective Automotive User Interfaces–Reviewing the State of Emotion Regulation in the Car. arXiv preprint arXiv:2003.13731.
5 EMOJ & RELAB present HU-Drive @CES2021, the innovative solution presented at CES 2021 by EMOJ and RE:Lab: https://www.youtube.com/watch?v=SQ_EWRAM23o
6 Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., & Nass, C. (2015). Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM), 9(4), 269-275.
7 Castellano, A., Landini, E., & Montanari, R. Un nuovo paradigma di interazione per la guida cooperativa: il progetto AutoMate.
Maria Giulia Losi
Maria Giulia Losi, PhD in „Humanities and Technologies: a research integrated path“. She is a project leader in RE:Lab with many years of experience in the design and development of Human-Machine Interaction systems and EU research projects. At RE:Lab since the beginning of 2011, she has been involved in several projects as Project Manager and Interaction Designer in different domains, from Automotive and Off-HIghway vehicles HMI and IoT Apps to Cultural Heritage interactive exhibitions design. Key publications are reported in the following:
1. Leandro Guidotti, Maria Giulia Losi, Roberto Montanari, Francesco Tesauri, “Motorsport Driver Workload Estimation in Dual Task Scenario”. IARIA Cognitive 2011
2. Daniele Pinotti, Fabio Tango, Maria Giulia Losi, Marco Beltrami. “A model for an innovative lane change assistant HMI”. HFES 2013 Conference Proceedings
3. Corazza F., Snijders D., Arpone M., Stritoni V., Martinolli F., Daverio M., Losi M.G., Soldi L., Tesauri F., Da Dalt L., Bressan S. „Development of a tablet app to optimize the management of pediatric cardiac arrest: a pilot study“. IPSSW2020 – 12th International Pediatric Simulation Symposia and Workshops
Annalisa Mombelli, Graduated with a Master’s Degree in Conservation of Cultural Heritage at University of Parma, defending a thesis about the innovation in publishing by Bodoni with his illustrated masterpiece “Epithalamia”. She also graduated in Advanced European Studies at European College of Parma. She worked as a freelance coordinator for private and public cultural projects. Now, she works in the R&D group of RE:lab, following activities in the field of education and culture.
Silvia Chiesa, PhD in Psychological, Anthropological and Educational Science at the University of Turin, defending a thesis concerning the spatial cognitive abilities in disabled persons. She expanded her activities outside the academic field working in the context of Human-Machine Interaction and more in general in the User Experience Research. She is Project Manager in the R&D group of RE:Lab. Here, she led international projects and activities concerning usability and acceptance assessment, in the domains of transports, household appliances and healthcare.
The New Relevance of Heritage
Professor of Sociology of Law since 1986, is Rector of the Suor Orsola Benincasa University of Naples, Vice President of the CNR (National Research Council)
Case study: The Pignatelli Chapel, Naples: How bringing past Renaissance alive with digital tech opens new civic and economic spaces
The chapel or small family church of Santa Maria dei Pignatelli, located in the heart of Naples, in Piazzetta Nilo, has 14th-century origins and was first renovated from as early as the last decades of the 15th century at the behest of Ettore Pignatelli, Duke of Monteleone and Borrello and future viceroy of Sicily. The works lasted until 1515 and included the construction of the two important funerary complexes that make the chapel one of the jewels of Neapolitan art of the mature Renaissance: the tomb of Carlo Pignatelli, on the left wall, the work of Tommaso Malvito’s workshop around 1506-07; and the small chapel of Caterina Pignatelli, the work of the great Spanish sculptor Diego de Silóe around 1513-14, rich in decorations taken from Antiquity and close to the culture of the Papal Rome of Raphael, Michelangelo and Sansovino. From 1736 the chapel was restored and assumed baroque forms. It was provided with a new altar in polychrome marble designed by Gaetano Buonocore and realised by Gennaro Di Martino, with rich marble decorations on the walls; a floor in inlaid marble was also added, realised by the same Di Martino and by Antonio Di Lucca on a drawing by Ferdinando Fuga (1761). A bowl-shaped dome was frescoed in 1772—as well as the drum, pendentives and sails of the adjacent vault—by Fedele Fischetti.
This phase was soon followed by a long period in which this extraordinary Renaissance masterpiece was neglected and fell into a state of abandonment, neglect and ruin. In the 1990s, the Pignatelli family decided to donate the chapel to the Suor Orsola Benincasa University which immediately carried out a vast program of works that, between 1999 and 2007, provided for the restoration of the structure and the arrangement of the annexed rooms. Then, from 2013 to 2015, the works of art preserved in the chapel were restored through the Great Project for the Historic Center of Naples, enhancement of the UNESCO World Heritage site. Finally, between 2016 and 2017, important sculptures by Diego de Silóe pertaining to the chapel of Caterina Pignatelli were brought back and relocated on new supports in the original site which had been devastated by thefts in the 1970s.
The Pignatelli Chapel thus became once again a „splendid“ Renaissance window on the historic centre of Naples.
But the above is only the first stage of an important process of recovery and enhancement of this masterpiece. In recent years, with a project that was completed in 2020, the University Suor Orsola has designed and developed a project to enhance the space through the integration of digital technologies organised in five interactive and multimodal visit experiences.
In the first, available to the visitor as soon as they enter the Chapel, spatialised sound tracks have been created, with a story about the place and its protagonists, which can be enjoyed through wireless headphones while moving freely.
A second experience allows the visitor to immerse themselves in a video that reproduces the space in its entirety, explorable in 360 degrees with the technique of cinematic virtual reality. Through immersive visors, the user can watch a story unfold within the walls of this building. That story, furthermore, is connected to another story, set instead at the Suor Orsola University a few kilometres away. where another space equipped with an immersive visor allows for a complementary experience. Thus, in either space the visitor is engaged in a virtual and remote dialogue, united by virtual reality and
by the crossing of glances, given that the two spaces are visible to each other.
Continuing on towards the balustrade in front of the chapel, a third experience provides the visitor with augmented reality applications that, by means of a special device, allow them to observe the details of the dome that otherwise would not be visible. By directing the devices in certain areas, moreover, the user can have precise and contextual information of the details. With this technological solution, physical boundaries can be overcome and the observation of space can be enriched with informative details.In the next experience, in a room on the upper floor, the visitor can interact with gestures and explore three-dimensional models of some of the works present in the Chapel.
The last experience projects the visitor towards the city, in a space that allows the visitor to follow cultural itineraries created for the project, referring to a wide range of themes, eras and works that show the many stories that make Naples extraordinary. The routes are depicted on a large interactive touch table and along the walls, on which appear images and maps linked to the chosen routes to be explored.
Our video recounts not only the recovery of an asset of such extraordinary value, returned to the city, but also how this reunification with the city context is enhanced by the crucial role of digital technologies. It is therefore an exemplary journey: from the Renaissance as a material and cultural heritage to be preserved, to the New Renaissance with the potentialities opened up by new technologies for cultural and creative industries.
Sources pictures: All pictures are screenshots from the Video below, Curtesy of Università Degli Studi Suor Orsola Benincasa.
Case study: The Pignatelli Chapel, Naples
Lucio d’Alessandro, Full Professor of Sociology of Law since 1986, is Rector of the Suor Orsola Benincasa University of Naples, Vice President of the CNR (National Research Council) and President of the TICHE Foundation, the management body of the National Cluster of Technologies for Cultural Heritage. He is also President of the Alumni Association of the Italian Institute of Historical Studies founded by Benedetto Croce. His research focuses mainly on Moral Utilitarianism between the eighteenth and nineteenth centuries (Utilitarismo morale e scienza della legislazione, 1993), on the genealogy of the social in the thought of Michel Foucault (Pouvoir, savoir, su Michel Foucault, 1981), on the relationship between society and law (Decisione del legislatore e interpretazione del giudice, 2009; Diritto e società: per un immaginario della cultura giuridica, 2018), on the modern concept of the University, starting from the thought of Humboldt and Schleiermacher (Universitas, exodus, communitas, 2011; Università quarta dimensione, 2016). With Il dono di nozze. Romanzo epistolare involontario sui Reali d’Italia scritto nel 1896 da Gabriele D’Annunzio e altri personaggi d’alto affare (Mondadori, 2015) he won the 2016 President’s Viareggio Prize.
Picture © Suor Orsola Benincasa University of Naples