Correctly modelling the nonlinear saturation characteristics of the transformer iron core is of great importance in steady-state calculations (harmonics and power quality) and calculations of the electromagnetic transients (inrush current, ferroresonance, temporary overvoltages). Thus, this paper presents an original estimation algorithm for determining the transformer nonlinear saturation characteristic from the measured inrush and steady-state current waveforms in entire windows of occurrences. The proposed algorithm is based on the minimization of a specifically defined cost function by applying the Nelder-Mead method, which was used to create a balance between the transformer steady-state current and the inrush current at zero residual flux. The algorithm uses the BDF numerical method in the background to solve the stiff differential equation system. By applying the appropriate indexes of quality estimation, it was demonstrated that, out of a total of twelve proposed analytical functions, the best optimization results were achieved with three proposed functions (the Frolich extended, three-term polynomial and special hyperbolic function). Detailed performances and accuracy suggested that the estimation algorithm has been verified by a very close agreement between the simulation results and measured transformer current waveforms in the steady-state and different inrush current scenarios.