This paper proposes a multi-classified and self-directed learning method used for sign language recognition, which adopts statistical template matching methods to recognize sign language. As sign language expressions consist of many frames, SIFT algorithm is used to position key frames and eigenvectors of sign language vocabulary. According to these key frames, the hierarchical discriminate regression (HDR) method is adopted to narrow the searching scope. Then, these obtained features are compared and matched with every words of sign language in the dynamic time warping (DTW) scope. The recognition rate of this method is 85%, which is higher than HMM under the same condition. This could greatly speed up the construction process of a sign language database.