In this paper we establish a rigorous spectral stability analysis for solitary waves associated to a generalized fractional Benjamin-Bona-Mahony type equation. Besides the well known smooth and positive solitary wave with large wave speed, we present the existence of smooth negative solitary waves having small wave speed. The spectral stability is then determined by analysing the behaviour of the associated linearized operator around the wave restricted to the orthogonal of the tangent space related to the momentum at the solitary wave. Since the analytical solution is not known, we generate the negative solitary waves numerically by using Petviashvili method. We also present some numerical experiments to observe the stability properties of solitary waves for various values of the order of nonlinearity and fractional derivative. Some remarks concerning the orbital stability are also celebrated.