Towards Turkish Word Embeddings: An Intrinsic Evaluation

Arslan O. A., Duman B., Erdem H., Gunyel C., Sonmez B., Arslan D.

8th International Conference on Computer Science and Engineering, UBMK 2023, Burdur, Turkey, 13 - 15 September 2023, pp.564-568 identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.1109/ubmk59864.2023.10286768
  • City: Burdur
  • Country: Turkey
  • Page Numbers: pp.564-568
  • Keywords: intrinsic evaluation, Turkish word embeddings, word embedding evaluation
  • Istanbul Technical University Affiliated: Yes


Effective representation of textual data is a prereq-uisite for most of the downstream tasks, which increases the importance of word embedding evaluation methods. The intrinsic approach assesses the similarity between word representations and human judgements. In this paper, we present a compre-hensive intrinsic evaluation of Turkish word embedding models with different tasks using task-specific datasets such as SemEval-2017, MC-30, SimVerb-3500 for word similarity, MSR for word analogy and methods that have not been tested for Turkish before such as oncept categorization with BLESS and ESSLLI and outlier detection with 8-8-8 Dataset. While each of these datasets were originally in English, we translated them into Turkish and trained Wor2Vec, FastText and Glove language models with these datasets from scratch. The results suggest that while Word2Vec is generally more successful in word similarity and outlier detection tasks, fastText outperforms other models in word analogy and concept categorization.