The effects of silica fume and aggregate size on the softening response and brittleness of high strength concretes were investigated by measuring the fracture energy G(F), the characteristic length 1(ch) and brittleness index B. Based on the fracture tests and microscopic studies at the aggregate-matrix interface, it was concluded that, in concretes without silica fume, the cement-aggregate interface had a profusion of calcium hydroxide and also much less dense calcium silicate hydrate, hence, the cracks usually developed at this weak interface, i.e. around coarse aggregate. However, in concretes with silica fume, the interfacial zone became stronger, more homogeneous and dense, hence, the cracks usually traversed the aggregates; transgranular type of fracture was observed. In these concretes, the fracture energy decreased dramatically especially for large size of aggregate case and as a result the brittleness index increased significantly.