Clustering and classification of large document bases in a parallel environment

✍ Scribed by Ruocco, Anthony S. ;Frieder, Ophir

• Publisher: John Wiley and Sons
• Year: 1997
• Tongue: English
• Weight: 189 KB
• Volume: 48
• Category: Article
• ISSN: 0002-8231
• DOI: 10.1002/(sici)1097-4571(199710)48:10<932::aid-asi7>3.0.co;2-2

✦ Synopsis

Development of cluster-based search systems has been ing, information retrieval systems must be prepared to hampered by prohibitive times involved in clustering process large amounts of data. As systems become inunlarge document sets. Once completed, maintaining clusdated with more and more information, it may not be ter organizations is difficult in dynamic file environpossible for people to fully understand what they have ments. We propose the use of parallel computing syscollected. The ability to produce information that categotems to overcome the computationally intense clustering process. Two operations are examined. The first is rizes the data, that is, the ability to produce metadata, is clustering a document set and the second is classifying in many cases as important as identifying specific pieces the document set. A subset of the TIPSTER corpus, speof data within a document set. cifically, articles from the Wall Street Journal, is used.

The ever-increasing size, coupled with the increasing

Document set classification was performed without the large storage requirement (potentially as high as 522M)

requirements to classify, group, and process the document for ancillary data matrices. In all cases, the time perforsets, all within nonprohibitive execution times, motivates mance of the parallel system was an improvement over the use of parallel processing computers. Parallel informasequential system times, and produced the same clustion retrieval focuses on this particular domain (Pogue, tering and classification scheme. Some results show 1988; Rasmussen, 1991;. Query pronear linear speed up in higher threshold clustering applications.

cessing assumes an organized data set as input. We, however, rely on parallel computing to organize the data by performing two cluster preprocessing operations. The first *Ophir Frieder is currently on leave from the Department of Computer Science, George Mason