This paper presents the analyses of nonlinear roll responses of a fishing boat in waves. In addition to roll damping nonlinearity, the nonlinear roll restoring which has seventh order equation has been taken into consideration to accurate control application. To overcome nonlinearity and the effects of uncertainties, LMI (Linear Matrix Inequality) - based H-2 and H-infinity State - Feedback Control are applied for the fin roll stabilizer of a fishing boat. The fin characteristics are calculated by Star CCM+ package software. Finally, utilising the studies presented to illustrate the feasibility and efficiency of the H-2 and H-infinity control methods, the results of the simulations are demonstrated the performance of fin roll stabilizer.