As the retransmission mechanism of the Session Initiation Protocol (SIP) significantly increases the amount of incoming traffic and makes an overloaded SIP server unstable, the models created for a stable SIP system cannot be effectively used to evaluate the performance of the overloaded server. To overcome this difficulty, in this paper, we present a fluid-flow model to analyze the priority-based request scheduling mechanism (PRSM) behavior in the overloaded SIP server equipped with an infinite buffer. The PRSM has been proposed in our earlier studies to prevent the overloading of SIP servers, where the original SIP request messages have strict priority over the retransmissions. The performance evaluation of the proposed model for the PRSM and the reference model of the conventional SIP has been reported under various test scenarios. The experiment results of the proposed model verify the effectiveness of the PRSM fluid-flow model as we already demonstrated in our previous studies using our simulation tool. Without loss of generality, the numerical results closely match with the simulation results, indicating that the proposed models can accurately capture the dynamic behavior of a SIP server with the PRSM. (C) 2017 Published by Elsevier B.V.