Railway transportation has gained importance with the development of high-speed trains in recent years. Problems that occur especially on the rail surface and fasteners during railway operation affect the operating safety of the train. For this reason, it has gained importance to examine railway lines at certain intervals. In this study, a two-stage approach is proposed to detect defects on rail surfaces. In the first stage of the approximation, rail extraction is performed and the histogram of the rail surface image is modeled as a Gaussian function. In addition, the region that may be defective is modeled with a Gaussian membership function and the membership values of the pixels are calculated. According to the dependencies of the pixels, whether there is a rail surface defect is determined, and if there is a defect in the next step, the defect type is determined with the convolutional neural network model. The proposed method has been tested for different defect types and successful results have been obtained.