Electromagnetic Bandgap (EBG) structures that exhibit various performances, such as preventing the electromagnetic wave propagation and reflecting an incident wave within the stopband, have unique electromagnetic characterization. Due to this reason, accurate analysis and design of the EBG structures are crucial to enhance the integrated system performance. This paper concentrates on the characterization of some planar EBGs using the Auxiliary Functions of Generalized Scattering Matrix (AFGSM) methods, with particular importance on an in- depth consideration of its bandgaps. The AFGSM method is applied to the planar EBG structures in the literature for the first time. The well-known kinds (symmetric and asymmetric cases) of mushroom type and multilayer EBG structures are considered to verify the presented method. Analysis results are compared with the Conventional Eigenvalue Equation (C-EIV) and the Generalized Scattering Matrix based Eigenvalue Equation (GSM-EIV) methods. Low computation load and accurate results are obtained to analyze the planar EBG structures with the AFGSM method due to using transmission line model. In addition, a design methodology is proposed for a chosen planar EBG structure using the AFGSM methods. Geometrical parameters of interested EBG problems are determined for acquiring the stopband frequency region of interest using the scattering parameters of unit cell configuration. The mushroom EBG model along one and two-dimensional axes is used in an antenna application to decrease mutual coupling between antenna elements. Three different scenarios are simulated in the HFSS electromagnetic simulation design environment to understand the effect of mutual coupling reduction in the antenna problem of the designed EBG structure via the AFGSM method. All designed antennas are manufactured, and the measurement results are in good agreement with the simulation results. The measurement results of the fabricated antenna application example including designed EBG using the proposed AFGSM method are compared with the existing similar problems with the same and different EBG models. It has been demonstrated that bandgap analysis, design of the planar EBG structures and integration of considered EBG model to a design application can be accurately and quickly achieved with the given methodology using the AFGSM method.