A nascent neutron star may be exposed to fallback accretion soon after the proto-neutron star stage. This high-accretion episode can submerge the magnetic field deep in the crust. The diffusion of the magnetic field back to the surface will take hundreds to millions of years depending on the amount of mass accreted and the consequent depth the field is buried. Neutron stars with large kick velocities will accrete less amount of fallback material leading to shallower submergence of their fields and shorter time-scales for the growth of their fields. We obtain the relation tau(Ohm) proportional to nu(-1) between the space velocity of the neutron star and Ohmic time-scale for the growth of the magnetic field. We compare this with the relation between the measured transverse velocities, nu(perpendicular to), and the field growth time-scales, mu/(mu)over dot, inferred from the measured braking indices. We find that the observational data are consistent with the theoretical prediction though the small number of data precludes a strong conclusion. Measurement of the transverse velocities of pulsars B1509-58, J1846-0258, J1119-6127 and J1734-3333 would increase the number of the data and strongly contribute to understanding whether pulsar fields grow following fallback accretion.