Compression techniques for Chinese text

✍ Scribed by Phil Vines; Justin Zobel

• Publisher: John Wiley and Sons
• Year: 1998
• Tongue: English
• Weight: 98 KB
• Volume: 28
• Category: Article
• ISSN: 0038-0644
• DOI: 10.1002/(sici)1097-024x(1998100)28:12<1299::aid-spe203>3.0.co;2-e

✦ Synopsis

With the growth of digital libraries and the internet, large volumes of text are available in electronic form. The majority of this text is English but other languages are increasingly well represented, including large-alphabet languages such as Chinese. It is thus attractive to compress text written in the large alphabet languages, but the general-purpose compression utilities are not particularly effective for this application. In this paper we survey proposals for compressing Chinese text, then examine in detail the application to Chinese text of the partial predictive matching compression technique (PPM). We propose several refinements to PPM to make it more effective for Chinese text, and, on our publicly-available test corpus of around 50 Mb of Chinese text documents, show that these refinements can significantly improve compression performance while using only a limited volume of memory.