With the widespread deployments of voice-over-internet protocol services, the existing session initiation protocol (SIP) design cannot scale up for large network sizes. Events triggering a demand burst or a server slowdown can cause SIP server overload, overload propagation, and crash, thus bringing down the whole SIP network. Since the SIP retransmission mechanism exacerbates the overload condition, existing models created for a stable SIP system cannot be effectively used to analyze an overloaded server. In this paper, we propose a fluid-flow model to characterize the behavior of the finite buffer SIP server equipped with priority-based request scheduling mechanism (PRSM). The model for the PRSM uses primary and secondary queues for the original request messages and the retransmitted requests, respectively. The performance metrics, namely, the failed call attempts and the response delay from sending INVITE request until receiving a 100-Trying response, are derived using the arrival time-slot tracking and the removal processes of the proposed fluid-flow model. We conducted test cases under the heavy traffic conditions, where the overload is caused by bulk and bursty arrivals or server slowdown. The numerical results closely match with the simulation results for all experiments, indicating that the proposed model can accurately capture the dynamic behavior of an SIP server with the PRSM. The experiments demonstrate that the number of failed call attempts is close to 0 and the mean response delay is kept constant around 175 ms for the PRSM when the buffer size is higher than 1K while both metrics are significantly higher for the conventional SIP.