The purpose of this study is to develop an algorithm for real-time measurement of aggregate size by means of image processing. The tested materials were rock fragments for construction, which were sorted into five size categories by sieving. The average sizes of the categories were 7.14 mm, 14.3 mm, 25.4 mm, 34.9 mm, and 44.5 mm. Surface images of the fragments were captured by a digital video camera under 22 lighting levels. A total of 3960 images were processed by a computer. Thirty-four kinds of image features including color and texture features were measured. The effects of the lighting conditions on the variation in the image features were clarified by nonparametric analysis of variance. Color information, such as the hue and saturation, was very sensitive to variation in illuminance. The relationship between aggregate size and texture features was expressed by multiple regression equations, which showed robust accuracy under the various illuminance values. The coefficient of determination of the regression equations was more than 0.8017. This non-contact real-time measurement shortens the time and eliminates the labor required for measuring aggregate size. This method retains measurement accuracy regardless of variation in illuminance.