Joel Paulo
The installations of interactive sound art in the public spaces poses important challenges for
artists and engineers. This aspect is important given the participation of common people,
having poor or none musical or technical skills, is essential for the piece of art to be successful.
The focus idea is to generate and modify, using information captured by sensors such as
microphones and video cameras to analyze types of sound and gestures, the aesthetics of a
base music generated by a sound synthesizer, changing some aspects of the sound in realtime,
such as, the pitch, reverb, speed, randomness. The resulting audio streaming is sent to a
virtual game platform to create immersive sound through an array of loudspeakers or virtual
reality headsets, i.e. oculus VR.
The information captured by the sensors is handled by Pyhton language using the OpenCV and
PyAudio libraries. A state machine is used to manage all the possible combination of sound and
gestures using a dictionary of events. The communication with SuperCollider sound synthetizer
is made by OSC messages. The JackRouter, using the Jack engine, is used to connect the
SuperCollider to the Unity3D game platform, where the acoustic spaciousness and
reverberation of the room selected beforehand is modified by the Resonance Audio SDK. The
user can choose diferent environment/scenarios and/or base sound tracks, thus, this
information is sent to Python, by OSC messages, to indicate the state machine to change
accordingly. The work is supported by Audio_and_Acoustics_Laboratory_of_ISEL.
Joel was born in Almada, Portugal. He received the Ph.D. degree in electrical engineering from the University of Lisbon, Instituto Superior Técnico, IST, Portugal, in 2012, in the fields of experimental room acoustics and audio signal processing. In 1996, he joined the Department of Radio Communications of the Nautical School Infante D. Henrique as an Assistant Professor, and in 2004 he joins the Department of Engineering in Electronics, Telecommunications and Computers of the Higher Institute of Lisbon, ISEL, Institute Polytechnic of Lisbon, IPL, where he is currently a Full Professor. He is the head of the Laboratory of Audio and Acoustics of ISEL and a researcher at the Group of Acoustics and Noise Control of the CAPS-IST. His main areas of R&D interest are: room acoustics measurements, multichannel sound, beamforming transducers, audio signal processing, psychoacoustics, 3D microphones, machine learning methods applied to music and noise, immersive sound environments for VR and performing arts. 3D sound for cinema, television and arts is one of his interest nowadays. Smart Cities, IoT, entrepreneurship and commercialization the results of academic projects are in his scope. He is a member of the Audio Engineering Society, AES, and the Acoustical Society of Portugal, SPA.
