Construction sector productivity is of great importance for the government and policymakers because it determines a nation's future living standards and creates a competitive business environment. Addressing the factors influencing labor productivity is crucial for improving the productivity performance of the sector. A theoretical structural model is developed to understand the influence of six key factors on construction labor productivity and their links to labor productivity performance on a project. A structural equation modeling technique is used to analyze data collected via a questionnaire survey of 185 respondents consisting of government authorities and construction actors. The final model adapts 29 attributes across six labor productivity factors, namely, management and control, workforce, financial, external, project, and material and equipment. The outcomes of the final structural equation model confirm the significance of management team competency level and workforce quality in enhancing labor productivity. Moreover, client support and efforts by government authorities are found to be significant for labor productivity improvements. Despite the limitations of the study, it is hoped that the research outcomes will significantly contribute to improve the labor productivity performance in the Malaysian construction industry for faster delivery of construction projects with lower cost and higher quality.