The current state of the art on how the circular cross-section shape of a rigid boundary channel affects the incipience of sediment entrainment is not well investigated in the literature. Moreover, the incipience of sediment entrainment studies via artificial intelligence is very limited so far. The studies for rigid boundary channels to date have yielded different incipient motion data depending on the cross-sectional shape. Also, most of the existing works were proposed for rectangular cross-sectional channels. Conversely, this study investigated the effects induced by a circular cross-sectional shape on critical dimensionless shear stress by experimental procedures under smooth flow conditions. It was found that the circular cross-sectional channel shape significantly altered the conditions in favor of reducing the critical shear stress for the incipient sediment entrainment. This implies that the circular cross-sectional channel shape exhibits the advantage of self-cleansing open channel design. For a circular cross-sectional channel and smooth flow conditions, a new critical shear stress formula is proposed based on the obtained experimental data. Ninety-seven experimental observations were performed under hydraulically transitional flow conditions with a grain shear Reynolds number (Re*) range of 5.2-41.17. Most of the experiments were performed with Re*>10. The new proposed equation is an empirical approach to express the governing equations of incipient motion in rigid boundary channels. Furthermore, a novel hybrid-geno-fuzzy inference system (GENOFIS) and the adaptive neural fuzzy inference system (ANFIS) approaches were developed using experimentally obtained data to predict the incipient motion of sediment entrainment. The performance of the quantitative results of the novel GENOFIS, ANFIS, and empirical equations were evaluated via the root mean square error (RMSE), F-test, and the coefficient of efficient (CE) model evaluation criteria. The GENOFIS is a novel attempt to revolutionize the prediction of complicated nonlinear phenomena such as incipient motion. The results show that prediction of the incipient motion of sediment via the GENOFIS approach yielded more accurate results than those of ANFIS for hydraulically transitional flow conditions. (C) 2021 American Society of Civil Engineers.