TOWARDS MINING WEB CONTENT OUTLIERS

Abstract

The task of outlier detection is to find small fraction of data that are exceptional when compared with rest large amount of data. Finding outliers from huge data repositories is like finding needles in a haystack. The existing outlier detection algorithms were designed for mining numeric data which cannot be applied directly to mine outliers from Web datasets because the Web contains data of different types such as: text, hypertext, images, video, audio, etc. Web content outliers are Web documents with varying contents compared to other documents taken from the same category. Mining Web document outliers may lead to the identification of competitors, emerging business trends in electronic commerce, improving the quality of results obtained from a Web search engine, and cleaning corpus used in Web documents classification. This thesis concentrates on enhancing current approaches for detecting Web

document outliers. It introduces a Web document outlier mining system aiming t<i detect outlying Web documents from a particular category. A major advantage of the proposed system is that it does not depend on a domain dictionary. Thus, it can be used for any domain of the interest of a miner. The system achieves its objectives by passing through three phases: preparation, processing, and detection. The preparation phase converts each Web document in the collected documents into what is called a feature frequency profile. This work provides two types of features: single word (i.e., keyword) and character N-grams (i.e., 4-grams, 5-grams). The processing phase uses a vector space model to represent the document collection as a matrix of feature we,ights. Then it computes pair-wise document dissimilarities resulting document dissimilarity matrix. The detection phase exploits document dissimilarity matrix for detecting Web document outliers. In this research, we adapted a novel algorithm called FindWDO algorithm for the detection phase of our proposed system. The FindWDO algorithm uses a k-nearest neighbors' method for identifying the top outlying Web documents. It also uses a •simple pruning rule to reduce the ruiming time required for identifying the closest neighbors for every document in the collection.

The experimental results on two different datasets with embedded motifs showed that FindWDO with N-grams outperforms similar algorithms in the same domain with respect to the accuracy of results.