Although biometric ear recognition has recently gained a considerable degree of attention, it remains difficult to use currently available ear databases because most of them are constrained. Here, the authors introduce a novel architecture called ScoreNet for unconstrained ear recognition. The ScoreNet architecture combines a modality pool with a fusion learning approach based on deep cascade score-level fusion. Hand-crafted and deep learning methods can be used together under the ScoreNet architecture. The proposed method represents the first automated fusion learning approach and is also compatible with parallel processing. The authors evaluated ScoreNet using the Unconstrained Ear Recognition Challenge Database, which is widely considered to be the most difficult database for evaluating ear recognition developed to date, and found that ScoreNet outperformed all other previously reported methods and achieved state-of-the-art accuracy.