We investigate the dependence of the evolution of neutron stars with fallback disks on the strength of the magnetic dipole field of the star. Using the same model as employed by Ertan et al. (2009), we obtain model curves for different dipole fields showing that the neutron stars with magnetic dipole fields greater than similar to 10(13) G on the surface of the star are not likely to become anomalous X-ray pulsars (AXPs) and soft gamma-ray repeaters (SGRs). Other sources with conventional dipole fields evolve into the AXP phase if their disk can penetrate the light cylinder. The upper limits to the observed periods of AXP and SGRs could be understood if the disk becomes inactive below a low temperature around 100 K. We summarize our present and earlier results indicated by the evolutionary model curves of these sources with an emphasis on the importance of the minimum disk temperature and the X-ray irradiation in the long-term evolution of AXPs and SGRs with fallback disks.