To enhance the mechanical behavior of fiber-reinforced polymers (FRPs), the epoxy and vinyl ester resin modified by silica nanoparticles at 1, 2, 3, and 4 weight percent (wt%) were investigated. Tests of fracture toughness were conducted, and scanning electron microscopy (SEM) of the fracture surface was proposed to identify the toughening mechanism. In addition, impregnated fiber roving made of modified resin was studied for improvements in tensile strength and stiffness. The addition of silica nanoparticles to epoxy and vinyl ester resin resulted in various promotions in their mechanical properties. The fracture toughness and impact strength of the epoxy resin in the presence of 3 wt% nanoparticles were increased by 12% and 49%, respectively, compared with the virgin epoxy resin. Furthermore, the corresponding of the vinyl resin with 3 wt% nanoparticles were 20% and 94%. SEM of the fracture surface displayed a relatively rough surface with tortuous cracks, thereby leading to higher fracture toughness of the modified resin system. The tensile strength of the impregnated basalt fiber roving made of the modified vinyl resin was obviously improved, and results showed that 18% promotion was obtained at a particle content of 3 wt%. In addition, the modified epoxy resin had less tensile strength enhancement to basalt fiber roving. Damage mechanisms of resin with nanoparticles were analyzed and differences due to the varying nanoparticle content identified.