Inattentive and impaired drivers are a major cause of road accidents. Especially when impaired by drugs, fatigue or physical handicaps, the skill levels, driving habits, capabilities and decisions of a human driver are adversely affected. In such cases, the ability of a driver to safely operate a vehicle may be augmented considerably by well tuned and driver adaptive warning and assistance systems. Data from 105 drivers were collected in the Drive Safe project on an approximately 30 km route containing both city and highway traffic. The data was used to develop methods for determining inattentive/impaired drivers. This paper is on a lane keeping driver assistant system that is activated once such an inattentive/impaired driver who cannot perform a good task of lane keeping by himself is determined. Robust, parameter space based and velocity scheduled control design techniques carried out in the COMES toolbox are used for designing the lane keeping controller. A camera based image processing algorithm for lane detection and tracking is used. The image processing algorithm and the lane keeping assistant control system are evaluated first using offline simulations and then using more realistic, real time hardware-in-the-loop simulations. While a relatively simple linear model is used for lane keeping controller design, evaluation of the designed controller also uses the high fidelity, high order, realistic and nonlinear vehicle model in Carmaker HiL. A PC is used for processing video frames coming from an in-vehicle camera pointed towards the road ahead. Lane detection and tracking computations including fitting of composite Bezier curves to curved lanes are carried out in this PC. A dSpace microautobox is available for obtaining the lane data from the PC and the Carmaker vehicle data from the dSpace compact simulator used and calculating the required steering actions and sending them to the Carmaker vehicle model. Offline and real time simulation results demonstrate the effectiveness of the proposed lane keeping assistant in automatically following lanes.