In the next-generation wireless networks, cooperative communication is one of the auspicious techniques through which spatial diversity could be achieved by permitting the single antenna to act as virtual multiple input multiple output (VMIMO). The fundamental principle of cooperative communication was established on different types of relaying protocols and implementation of different relaying protocols according to the requirements of communication scenarios. The challenging task for achieving a high performance in cooperative communication is to find out the optimal relay node (RN) among different prevailing RNs. The basic purpose of cooperative communication is to maximize the performance of the network and minimize the overhead triggered by RNs by considering different communication metrics, i.e., signal to noise ratio (SNR), channel state information (CSI), and bit error rate (BER). This study presents a review of different cooperative relaying protocols with best relay selection techniques in the next-generation wireless network. Moreover, the different challenges faced by millimeter wave in 5G wireless networks and the role of cooperative communication to overcome those challenges are discussed. (c) 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.