Communication security has become an indispensable demand of smartphone users. End-to-end encryption is the key factor for providing communication security, which mainly relies on public key cryptography. The main and unresolved issue for public key cryptography is to correctly match a public key with its owner. Failing to do so could lead to man-in-the-middle attacks. Different public key verification methods have been proposed in the literature. The methods which are based on verification by the users themselves are preferable with respect to cost and deployability than the methods such as digital certificates that involve the use of trusted third parties. One of these methods, fingerprinting was recently replaced by a method called safety number in the open source messaging application, SIGNAL. The developers of SIGNAL claimed this change would bring usability and security advantages however no formal user study was conducted supporting this claim. In this study, we compare the usability and security aspects of these two methods with a user study on 42 participants. The results indicate with significance that the safety number method leads to more successful results in less time for public key verification as compared to the fingerprint method.