Information content in textual data: Revisited for Arabic text

✍ Scribed by Hegazi, Nadia ;Ali, Nabil ;Abed, Ehsan

• Publisher: John Wiley and Sons
• Year: 1987
• Tongue: English
• Weight: 322 KB
• Volume: 38
• Category: Article
• ISSN: 0002-8231
• DOI: 10.1002/(sici)1097-4571(198703)38:2<133::aid-asi8>3.0.co;2-p

✦ Synopsis

Arabic as opposed to English is a highly redundant language due to its morphological nature.

A study was done to measure this redundancy and compare it to its respective values in English. Samples of books, news papers, and social magazines were used to measure the entropy of the Arabic language using the n-gram method generated from a moving window of eight characters. Studies of the dependencies of characters on each other was done, as well as a study on the average distribution of word lengths. The results obtained indicated the ability of Arabic to be more compressible than English, and that of course is due to its morphological nature. The average length of Arabic words was found to be longer than English words due to the fact that Arabic words contain morphological extensions.