In this study, we evaluate the performance of various semantic textual similarity methods on question similarity detection task in Turkish. Various handcrafted features and neural models, specifically siamese recurrent networks, are studied to detect questions which have a similar meaning to given question in a dataset. Several experiments have been performed to compare the performance of features and neural methods. Our Experiments demonstrate that siamese recurrent networks significantly outperforms traditional methods which are based on handcrafted features such as word and stem matching counts, TF-IDF vectors and similarity of word embeddings. We also observed that the performance of siamese recurrent networks could be further improved by incorporating handcrafted features to the process.