The increasing demand on the internet trafficking exerts too much pressure to meet the demands of video streaming and mobile communication. Accordingly, this demand results in providing high bandwidth that increases the use of cells on the network. However, the current networks do not meet the requirements of the necessary data rate. Therefore, the Dense Wavelength Division Multiplexing (DWDM) network and the Radio over Fiber (RoF) technology are the ideal solution for providing the necessary data rate needed in the current networks. The DWDM will increase both of the transmission distance and the data rate. Nonetheless, the DWDM network will be compromised especially by the Non-linear effects. This study is intended to propose a system to find solutions for the issues of increasing the data rate and for reducing the nonlinear effects. There are number of technologies that could be adopted to fix such issues, which include; Optical Phase Conjugation (OPC), Semiconductor Optical Amplifier (SOA), and Digital Signal Processing (DSP). Notwithstanding, DSP-128 QAM-SCRO (Squared Cosine Roll off) filter has been used in this study. Then its performance was compared with the performance of the Semiconductor Optical Amplifier (SOA) to establish which one of these two is optimal to increase the data rate and to eliminate the nonlinear effects. The performance of the proposed system has been evaluated in terms of the Error Vector Magnitude (EVM), Bit Error Rate (BER), Q Factor and Eye-Diagram. The EVM has been measured in the proposed system prior to employing the DSP-128 QAM-SCRO technology, and it has reached the value 22.5%. However, when the EVM has been measured after employing the DSP-128 QAM-SCRO technology, it has reached the value 8.5%. Similarly, the values of BER and Q factor have been found to be acceptable and in accordance with ITU-T. Finally, the proposed system has been simulated via the software Optisystem 17.