An audiovisual interface-based drumming system for multimodal human-robot interaction

Creative Commons License

İnce G., Yorganci R., Ozkul A., Duman T. B., Köse H.

JOURNAL ON MULTIMODAL USER INTERFACES, cilt.15, sa.4, ss.413-428, 2021 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 15 Sayı: 4
  • Basım Tarihi: 2021
  • Doi Numarası: 10.1007/s12193-020-00352-w
  • Dergi Adı: JOURNAL ON MULTIMODAL USER INTERFACES
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex
  • Sayfa Sayıları: ss.413-428
  • Anahtar Kelimeler: Audio processing, Audiovisual interaction, Drum game, Human–computer-interaction (HCI), Human–robot-interaction (HRI), Multimodality, Non-verbal communication
  • İstanbul Teknik Üniversitesi Adresli: Evet

Özet

This paper presents a study of an audiovisual interface-based drumming system for multimodal human-robot interaction. The interactive multimodal drumming game is used in conjunction with humanoid robots to establish an audiovisual interactive interface. This study is part of a project to design robot and avatar assistants for education and therapy, especially for children with special needs. It specifically focuses on evaluating robot/virtual avatar tutors, tangible interaction devices, and mobile multimedia devices within a simple drumming-based interactive music tutoring game scenario. Several parameters, including the effect of the embodiment of the tutor/interface and the presence of feedback and training mechanisms, were the focus of interest. For that purpose, we created an interactive drumming game relying on turn-taking and imitation principles, in which a human user is able to play drums with a humanoid robot (Robotic Drum Mate). Three interactive scenarios with different experimental setups for humanoid robots and mobile devices were developed and tested. As a part of those scenarios, a system that enables drum strokes to be automatically detected and recognized using auditory cues was implemented and incorporated into the experimental framework. We verified the applicability and effectiveness of the proposed system in a drum-playing game with adult human test subjects by evaluating it both objectively and subjectively. The results showed that the physical robot tutor, the feedback and training mechanisms had a positive effect on the subjects' performance and, as expected, although the physical medium is preferred, the virtual medium for drumming caused less failure.