The self modulation of surface shear horizontal (SH) waves (Love waves) propagating on a nonlinear half space covered by a nonlinear thin layer is examined. First a nonlinear thin layer approximation is derived by assuming that constituent materials are nonlinear,homogeneous, isotropic and compressible hyper-elastic. Then employing this approximation, a two medium problem is reduced to one for a nonlinear half space with a modified nonlinear boundary condition on the top surface. This new problem is analyzed by the method of multiple scales, and a nonlinear Schrodinger (NLS) equation is derived which describes the self modulation of the waves asymptotically. Then the results of the nonlinear thin layer approximation are compared with the results of the linear thin layer approximation and with the finite layer results obtained in a previous work. It has been observed that even in moderately low wave numbers, the propagation is affected considerably by the nonlinear material parameter of the layer. (C) 2014 Elsevier Ltd. All rights reserved.