This paper presents an e cient automatic face recognition scheme useful for video indexing applications. In particular the following problem is addressed: given a set of known face images and given a complex video sequence to be indexed, ÿnd where the corresponding faces appear in the shots of the sequence. The main and ÿnal objective is to develop a tool to be used in the MPEG-7 standardization e ort to help video indexing activities. Conventional face recognition schemes are not well suited for this application and alternative and more e cient schemes have to be developed. In this paper, in the context of Principal Component Analysis for face recognition, the concept of self-eigenfaces is introduced. In addition, the color information is also incorporated in the face recognition stage. The face recognition scheme is used in combination with an automatic face detection scheme which makes the overall approach highly useful. The resulting scheme is very e cient to ÿnd speciÿc face images and to cope with the di erent face conditions present in a complex video sequence. Results are presented using the test sequences accepted in the MPEG-7 video content sequences set.