3D Face Animation Generation from Audio Using Convolutional Networks Evrişimsel Aǧlar ile Sesten 3B Yüz Animasyonu Üretilmesi


Ünlü T., İnceoğlu A., Yilmaz E. O., Sariel S.

30th Signal Processing and Communications Applications Conference, SIU 2022, Safranbolu, Turkey, 15 - 18 May 2022 identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.1109/siu55565.2022.9864734
  • City: Safranbolu
  • Country: Turkey
  • Keywords: 3D Face Animation from Audio, Deep Learning, Speech Processing
  • Istanbul Technical University Affiliated: Yes

Abstract

© 2022 IEEE.3D facial animation generation from audio problem is drawing attention as it is demanded for generating artificial characters in games and movies. In the literature, several studies address this problem. However, the generated facial animations are far away from being realistic. In this work, we represent faces with Facial Action Coding System (FACS) and collect a 37-minute-long dataset. We develop convolutional and transformer based models. It is observed that the trained model is able to generate animations that can be used in video games and virtual reality applications, even with novel speaker audio data of speakers it has never seen in the training data.