Preprint C121/2010
The Guitar as a Human-Computer Interface
Marcelo Cicconet
Keywords: Human-Computer Interface | Guitar Interface | Computer Music | Music Performance | Machine Learning
In this work we explore the visual interface of the guitar. From the analysis point of view, the use of a video camera for human-computer interaction in the context of a user playing guitar is studied. From the point of view of synthesis, visual properties of the guitar fretboard are taken into account in the development of bi-dimensional interfaces for music performance, improvisation, and automatic composition. The text is divided in two parts. In the first part, we discuss the use of visual information for the tasks of recognizing notes and chords. We developed a video-based method for chord recognition which is analogous to the state- of-the-art audio-based counterpart, relying on a Supervised Machine Learning algorithm applied to a visual chord descriptor. The visual descriptor consists of the rough position of the fingertips in the guitar fretboard, found by using special markers attached to the middle phalanges and fiducials attached to the guitar body. Experiments were conducted regarding classification accuracy comparisons among methods using audio, video and the combination of the two signals. Four different Data Fusion techniques were evaluated: feature fusion, sum rule, product rule and an approach in which the visual information is used as prior distribution, which resembles the way humans recognize chords being played by a guitarist. Results favor the use of visual information to improve the accuracy of audio-based methods, as well as for being applied without audio-signal help. In the second part, we present a method for arranging the notes of certain musical scales (pentatonic, heptatonic, Blues Minor and Blues Major) on bi- dimensional interfaces by using a plane tessellation with especially designed musical scale tiles. These representations are motivated by the arrangement of notes in the guitar fretboard, preserving some musical effects possible on the real instrument, but simplifying the performance, improvisation and composition, due to consistence of the placement of notes along the plane. We also describe many applications of the idea, ranging from blues-improvisation on multi-touch screen interfaces to automatic composition on the bi-dimensional grid of notes.