A metric to search for relevant words

✍ Scribed by Hongding Zhou; Gary W. Slater

• Publisher: Elsevier Science
• Year: 2003
• Tongue: English
• Weight: 664 KB
• Volume: 329
• Category: Article
• ISSN: 0378-4371
• DOI: 10.1016/s0378-4371(03)00625-3

✦ Synopsis

We propose a new metric to evaluate and rank the relevance of words in a text. The method uses the density uctuations of a word to compute an index that measures its degree of clustering. Highly signiÿcant words tend to form clusters, while common words are essentially uniformly spread in a text. If a word is not rare, the metric is stable when we move any individual occurrence of this word in the text. Furthermore, we prove that the metric always increases when words are moved to form larger clusters, or when several independent documents are merged. Using the Holy Bible as an example, we show that our approach reduces the signiÿcance of common words when compared to a recently proposed statistical metric.