In this article, we propose an approach to create a high-quality quantum tent map by utilizing the generalized quantum dot system. Our objective is to determine if its chaos surpasses that of the traditional classic tent. To achieve this, we first introduce a quantum tent map, showcasing its chaotic behavior in relation to control parameters and initial conditions. We validate the presence of chaos by calculating the Lyapunov exponent and analyzing the time series. Next, we design a chaotic S-box based on this new map. Subsequently, we explore the possibility of obtaining strong S-boxes by employing two fractional stochastic models and three components of the proposed quantum tent map. Our primary question is whether the combination of fractional stochastic models and quantum tent maps can result in superior S-boxes. The answer to this question is a resounding “Yes.” The performance of these S-boxes exceeds that of previously proposed models. Finally, we evaluate a mixed S-box as the foundation for achieving highly secure image encryption. Among the models presented in this article, the best-performing S-box is produced through the combination of the fractional gamma distribution (FGD) with parameters α= 1.06 and β= 2.8 , along with the X-quantum tent map dimension. This model achieves an SAC (Strict Avalanche Criterion) value of 0.5, surpassing even AES (Advanced Encryption Standard). Its non-linearity value is 106.625, indicating excellent performance. Additionally, the model has an LP (Linear Property) value of 0.128906 and a DP (Differential Properties) score of 12. Furthermore, we obtain BIC-SAC 0.503209 and BIC-Non-linearity 103.679. Lastly, we present an image encryption algorithm to demonstrate the effectiveness of the mixed S-box, evaluating its performance against various attacks. The results affirm the suitability of this encryption method, with the generated image encryption offering a key space of 2 16960 , ensuring high security. The example image exhibits an entropy value of 7.9311. Moreover, it demonstrates correlations in different orientations: horizontal (0.0076), vertical (- 0.00048659), and diagonal (0.00020717). The image also achieves NPCR (Number of Pixel Change Rate) at %99.63 and UACI (Unified Average Changing Intensity) at %33.96. Notably, the algorithm executes in an impressive time of 0.810653, particularly considering the utilization of new 64 S-boxes instead of a fixed one. Even the correlation value for some standard photos approaches 0.0000667, demonstrating smallest correlation. This article underscores the effectiveness of combining new methods to generate an S-box.