Non-verbal Communication with a Social Robot Peer: Towards Robot Assisted Interactive Sign Language Tutoring

Akalın N., Uluer P., Köse H.

14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), Madrid, Spain, 18 - 20 November 2014, pp.1122-1127 identifier

  • Publication Type: Conference Paper / Full Text
  • City: Madrid
  • Country: Spain
  • Page Numbers: pp.1122-1127
  • Istanbul Technical University Affiliated: Yes


This paper presents a humanoid robot assisted imitation based interactive game for Sign Language (SL) tutoring. The game is specially designed for children with communication impairments. The work presented is a part of Robot Sign Language Tutor* project. In this study, a Robovie R3 humanoid robot is used to express a set of chosen words in SL using hand movements, body and face gestures. The robot platform is specially modified with LEDs in face (for non-manual facial gestures), additional DoFs in wrist and 5 independent fingers at hands for robust SL generation. The robot is able to communicate with the participant by recognizing signs and colored flashcards through a RGB-D camera based system and generating a selected subset of signs, including motivating facial gestures, in return. The game also aims to improve children's imitation and turn-taking skills and to teach the words semantically. Current paper presents results from the preliminary study with a group of hearing impaired children, where children had almost 100% score in recognizing the robot's signs from a subset of 12 words.