Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

Stiefelhagen R., Ekenel H. K., FUGEN C., GIESELMANN P., HOLZAPFEL H., KRAFT F., ...More

IEEE TRANSACTIONS ON ROBOTICS, vol.23, no.5, pp.840-851, 2007 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 23 Issue: 5
  • Publication Date: 2007
  • Doi Number: 10.1109/tro.2007.907484
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Page Numbers: pp.840-851
  • Istanbul Technical University Affiliated: No


In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforscbungsbereich) on humanoid cooperative robots.