In this study, we focus on automated optimization design methodologies to concurrently trade off between power gain, output power, efficiency, and linearity specifications in radio frequency (RF) high-power amplifiers (HPAs) through deep neural networks (DNNs). The RF HPAs are highly nonlinear circuits where characterizing an accurate and desired amplitude and phase responses to improve the overall performance is not a straightforward process. For this case, we propose a coarse and fine modeling approach based on firstly modeling the involved transistor and then selecting the best configuration of HAP along with optimizing the involved input and output termination networks through DNNs. In the fine phase, we firstly construct the equivalent modeling of the GaN HEMT transistor by using X-parameters. Then in the coarse phase, we utilize hidden layers of the modeled transistor and replace the HPA's DNN to model the behavior of the selected HPA by using S-parameters. If the suitable accuracy of HPA modeling is not achieved, the hyperparameters of the fine model are improved and re-evaluated in the HPA model. We call the optimization process coarse and fine modeling since the evaluation process is performed from S-parameters to X-parameters. This stage of optimization can ensure modeling the nonlinear HPA design that includes a high number of parameters in an effective way. Furthermore, for accelerating the optimization process, we use the classification DNN for selecting the best topology of HPA for modeling the most suitable configuration at the coarse phase. The proposed modeling strategy results in relatively highly accurate HPA designs that generate post-layouts automatically, where multi-tone harmonic balance specifications are optimized once together without any human interruptions. To validate the modeling approach and optimization process, a 10 W HPA is simulated and measured in the operational frequency band of 1.8 GHz to 2.2 GHz, i.e., the L-band. The measurement results demonstrate a drain efficiency higher than 54% and linear gain performance more than 12.5 dB, with better than 50 dBc adjacent channel power ratio (ACPR) after DPD.