Finite element method (FEM) is commonly used with topology optimization algorithms to determine optimum topology of load-bearing structures. However, it may possess various difficulties and limitations for handling the problems with moving boundaries, large deformations, and cracks/damages. To remove limitations of the mesh-based topology optimization, this study presents a robust and accurate approach based on the innovative coupling of peridynamics (PD) (a meshless method) and topology optimization (TO), abbreviated as PD-TO. The minimization of compliance, i.e. strain energy, is chosen as the objective function subjected to the volume constraint. The design variable is the relative density defined at each particle employing bidirectional evolutionary optimization approach. A filtering scheme is also adopted to avoid the checkerboard issue and maintain the optimization stability. To present the capability, efficiency, and accuracy of the PD-TO approach, various challenging optimization problems with and without defects (cracks) are solved under different boundary conditions. The results are extensively compared and validated with those obtained by element-free Galerkin method and FEM. The main advantage of the PD-TO methodology is its ability to handle TO problems of cracked structures without requiring complex treatments for mesh connectivity. Hence, it can be an alternative and powerful tool in finding optimal topologies that can circumvent crack propagation and growth in two- and three-dimensional structures.