The Usu mud volcanoes are the largest group of terrestrial mud volcanoes in China. The volcanoes are located in a typical arid and semi-arid region, and the group consists of 36 erupting active mud volcanoes. In this study, the prokaryotic diversity and community structure in the sediment of an active mud volcano were investigated by constructing bacterial and archaeal clone libraries of the 16S rRNA gene. A total of 100 bacterial and 100 archaeal clones were analysed and found to comprise 11 and 7 distinct phylotypes, respectively. The bacterial phylotypes were classified into three phyla (Proteobacteria, Actinobacteria, and Fusobacteria). Of these, Proteobacteria were the most abundant bacterial group, with Deltaproteobacteria dominating the sediment community, and these were affiliated with the order Desulfuromonadales. The archaeal phylotypes were all closely related to uncultivated species, and the majority of the members were related to the orders Methanosarcinales and Halobacteriales of the Euryarchaeota originating from methane hydrate bearing or alkaline sediments. The rest of the archaeal phylotypes belonged to the phylum Crenarchaeota, with representatives from similar habitats. These results suggested that a large number of novel microbial groups and potential methanogenesis may exist in this unique ecosystem.