Knowledge-based Image Representation and Retrieval

Abstract

Analyzing complex natural language queries through image/multimedia search engines remains a big challenge. Traditional text-based retrieval systems associate textual descriptions with each image, based on subjective human perception. These descriptions are next matched lexically against the user interrogated queries. Such annotation-based paradigm does not achieve the best results, since the lexical comparison is not sufficient for matching sentences in a semantic manner. Combining image retrieval processing with rich semantics and knowledge-based modeling provide promising solutions towards better image search engines. This thesis proposes a knowledge-based image representation and retrieval which integrates external knowledge sources for obtaining a higher-level inference that can both handle complex natural language queries and increase the number of relevant retrievals for image search engines. The thesis presents two solution approaches for the purpose of enhancing image retrieval. The first solution proposes a semantic framework for image representation and retrieval that can efficiently handle complex human-wise queries. The second solution proposes a semantic evaluation for auto-generated image annotations based on similarity measurement. In the proposed image representation and retrieval framework, semantics are integrated by employing external knowledge sources and query expansion in the retrieval process. A set of developed and off the shelf parsing tools are used to obtain a full semantic understanding for relating the natural language queries and image annotations. The user query is parsed and next fused with the external knowledge sources in a query expansion process to infer supplemental knowledge about the terms of the query and hence increasing