Human tongue is one of the important organs of the body, which carries abound of information of the health status. Among the various information on tongue, color is the most important factor. Most existing methods carry out pixel-wise or RGB color space classification in a tongue image captured with color CCD cameras. However, these conversional methods impede the accurate analysis on the subjects of tongue surface because of the less information of this kind of images. To address problems in RGB images, a pushbroom hyperspectral tongue imager is developed and its spectral response calibration method is discussed. A new approach to analyze tongue color based on spectra with spectral angle mapper is presented. In addition, 200 hyperspectral tongue images from the tongue image database were selected on which the color recognition is performed with the new method. The results of experiment show that the proposed method has good performance in terms of the rates of correctness for color recognition of tongue coatings and substances. The overall rate of correctness for each color category was 85% of tongue substances and 88% of tongue coatings with the new method. In addition, this algorithm can trace out the color distribution on the tongue surface which is very helpful for tongue disease diagnosis. The spectrum of organism can be used to retrieve organism colors more accurately. This new color analysis approach is superior to the traditional method especially in achieving meaningful areas of substances and coatings of tongue.