Semantic Graph Representation and Evaluation for Generated Image Annotations

Abstract

The rapid advancement in generated image annotations, also known as captions, puts a crucial need for efficient and automated methods for those annotations’ evaluation. State-of-art in the automatic evaluation metrics has proven to be inefficient for evaluating the quality of the generated annotations because they rely on certain aspects of word matching, such as the n-gram overlap. This paper presents a semantic-based graph representation and evaluation for generated image annotations. The semantic graph explicitly encodes objects, attributes, and natural language annotations relationships, while consulting ConceptNet as an external knowledge source to provide a rich semantic generated graph. For annotations evaluation, the ConceptNet is extended for use in a proposed semantic evaluation metric, whose input is the semantic graphs. Experimental results show that, over the Flickr8k dataset, the proposed graph-based evaluation metric achieves a higher system-level correlation and rank correlation coefficient value compared to existing related works.