This paper presents an automatic video genre classification system, which utilizes several low-level audio-visual cues as well as cognitive and structural information to classify the types of TV programs and YouTube videos. Classification is performed using support vector machines. The system is integrated to our content-based video processing system and shares the same features that we have been using for high-level feature detection task in TRECVID evaluations. The proposed system is extensively evaluated using complete TV programs from Italian RAI TV channel, from French TV channels, and videos from YouTube on which 99.6%, 99%, and 92.4% correct classification rates are attained, respectively. These results show that the developed system can reliably determine TV programs' genre. It also provides a good basis for classifying genres of YouTube videos, which can be improved by using additional information, such as tags and titles, to obtain more robust results. Further experiments indicate that the quality of video does not influence the results significantly. It is found that the performance drop in classifying genres of YouTube videos is mainly due to the large variety of content contained in these videos.