The installations of interactive sound art in the public spaces poses important challenges for artists and engineers. This aspect is important given the participation of common people, having poor or none musical or technical skills, is essential for the piece of art to be successful. The focus idea is to generate and modify, using information captured by sensors such as microphones and video cameras to analyze types of sound and gestures, the aesthetics of a base music generated by a sound synthesizer, changing some aspects of the sound in realtime, such as, the pitch, reverb, speed, randomness. The resulting audio streaming is sent to a virtual game platform to create immersive sound through an array of loudspeakers or virtual reality headsets, i.e. oculus VR.
The information captured by the sensors is handled by Pyhton language using the OpenCV and PyAudio libraries. A state machine is used to manage all the possible combination of sound and gestures using a dictionary of events. The communication with SuperCollider sound synthetizer is made by OSC messages. The JackRouter, using the Jack engine, is used to connect the SuperCollider to the Unity3D game platform, where the acoustic spaciousness and reverberation of the room selected beforehand is modified by the Resonance Audio SDK. The user can choose diferent environment/scenarios and/or base sound tracks, thus, this information is sent to Python, by OSC messages, to indicate the state machine to change accordingly. The work is supported by Audio_and_Acoustics_Laboratory_of_ISEL.
Joel was born in Almada, Portugal. He received the Ph.D. degree in electrical engineering from the University of Lisbon, Instituto Superior Técnico, IST, Portugal, in 2012, in the fields of experimental room acoustics and audio signal processing. In 1996, he joined the Department of Radio Communications of the Nautical School Infante D. Henrique as an Assistant Professor, and in 2004 he joins the Department of Engineering in Electronics, Telecommunications and Computers of the Higher Institute of Lisbon, ISEL, Institute Polytechnic of Lisbon, IPL, where he is currently a Full Professor. He is the head of the Laboratory of Audio and Acoustics of ISEL and a researcher at the Group of Acoustics and Noise Control of the CAPS-IST. His main areas of R&D interest are: room acoustics measurements, multichannel sound, beamforming transducers, audio signal processing, psychoacoustics, 3D microphones, machine learning methods applied to music and noise, immersive sound environments for VR and performing arts. 3D sound for cinema, television and arts is one of his interest nowadays. Smart Cities, IoT, entrepreneurship and commercialization the results of academic projects are in his scope. He is a member of the Audio Engineering Society, AES, and the Acoustical Society of Portugal, SPA.