Semantic and Interactive Content-based Image Retrieval

✍ Scribed by Björn Barz

• Publisher: Cuvillier
• Year: 2020
• Tongue: English
• Leaves: 323
• Category: Library

✦ Synopsis

Content-based image retrieval (CBIR) aims for finding images in large databases such as the internet based on their content. Given an exemplary query image provided by the user, the retrieval system provides a ranked list of similar images. Most contemporary CBIR systems compare images solely by means of their visual similarity, i.e., the occurrence of similar textures and the composition of colors. However, visual similarity does not necessarily coincide with semantic similarity. For example, images of butterflies and caterpillars can be considered as similar, because the caterpillar turns into a butterfly at some point in time. Visually, however, they do not have much in common. In this work, we propose to integrate such human prior knowledge about the semantics of the world into deep learning techniques. Class hierarchies serve as a source for this knowledge, which are readily available for a plethora of domains and encode is-a relationships (e.g., a poodle is a dog is an animal etc.). Our hierarchy-based semantic embeddings improve the semantic consistency of CBIR results substantially compared to conventional image representations and features. We furthermore present three different mechanisms for interactive image retrieval by incorporating user feedback to resolve the inherent semantic ambiguity present in the query image. One of the proposed methods reduces the required user feedback to a single click using clustering, while another keeps the human in the loop by actively asking for feedback regarding those images which are expected to improve the relevance model the most. The third method allows the user to select particularly interesting regions in images. These techniques yield more relevant results after a few rounds of feedback, which reduces the total amount of retrieved images the user needs to inspect to find relevant ones.

✦ Table of Contents

1 Introduction
1.1 Content-based image retrieval
1.2 Instance vs. category retrieval
1.3 Challenges
1.4 Interactive image retrieval
1.5 Semantic image retrieval
1.6 Contributions of this thesis
2 Methodical Background
2.1 Fundamental concepts and definitions
2.2 Classification
2.2.1 Problem setting
2.2.2 Support vector machines
2.2.3 Linear discriminant analysis
2.2.4 Nearest neighbor classification
2.2.5 Gaussian processes
2.2.6 Neural networks
2.2.7 Active learning
2.3 Clustering
2.3.1 k-means
2.3.2 GaussianMixtureModels
2.4 Metric Learning
2.4.2 Duality between metric and feature learning
2.4.3 Learning metrics for fixed features
2.4.4 Deep metric learning
2.5 Information retrieval
2.5.1 Problem description
2.5.2 Evaluation metrics
2.5.3 Learning to rank
2.5.4 System architecture
2.5.5 Spatial verification and re-ranking
2.5.6 Query expansion and diffusion
2.5.7 Cross- and multi-modal retrieval
2.6 Image representations for CBIR
2.6.1 Hand-crafted local features
2.6.2 Hand-crafted transformationsand aggregations
2.6.3 Principal components analysis and whitening
2.6.4 Off-the-shelf CNN features
2.6.5 End-to-end learning for image retrieval
2.7 Relevance feedback
3 The Cosine Loss:A RetrievalMetricused for Classification
3.1 Introduction and motivation
3.1.1 The problem of small data
3.1.2 Weakly supervised localization
3.2 Related work
3.2.2 Learning from small data
3.2.3 Weakly supervised localization
3.3 The cosine loss
3.3.1 Objective and notation
3.3.2 Comparison with other loss functions
3.4 Dense classification andscene understanding
4 Hierarchy-based SemanticImage Embeddings
4.1 In the need of prior knowledge
4.1.1 Semantic image retrieval
4.1.2 Explaining classification decisions
4.2 Related work
4.3 Knowledge in trees: class taxonomies
4.3.1 Hierarchy-based semantic similarity
4.3.2 Tree-shaped taxonomies
4.4 Hierarchy-based semantic embeddings
4.4.1 Exact solution
4.4.2 Low-dimensional approximation
4.5 Learning semantic image embeddings
4.6 Subsequent works onsemantic embeddings
5 Experiments forCosine Loss andSemantic Embeddings
5.1 Datasets
5.1.1 Visual classification datasets
5.1.2 FGVC datasets
5.1.3 ExtremeWeather dataset
5.1.4 AG News dataset
5.1.5 MS COCO
5.2 Training details
5.3 Semantic image retrieval
5.3.1 Performance metrics
5.3.2 Competitors
5.3.3 Semantic image retrieval performance
5.3.4 Low-dimensional approximation
5.4 Learning from small data
5.4.1 Classification performance
5.4.2 Effect of semantic information
5.4.3 Effect of dataset size
5.5 Learned feature space
5.6 Dense classification
5.6.1 Weakly supervised localization
5.6.2 Explaining classifier decisions
6 Interactive Image Retrieval
6.1 Introduction
6.2 Related work
6.2.1 Multiple instance feedback
6.2.2 Active learning for image retrieval
6.2.3 Important image regions
6.3 Automatic query image disambiguation
6.3.1 Identification of image senses
6.3.2 Refinement of results
6.4 Information-theoretic active learning
6.4.1 Idea and ideal objective
6.4.2 Relevance model
6.4.3 User model
6.4.4 Approximation of mutual information
6.4.5 Greedy batch construction
6.5 Adaptive pooling
6.5.1 Learning the Exemplar-LDA model
6.5.2 Modifying the representationsof database images
7 Experimental Evaluationof Interactive Retrieval
7.1 Datasets and features
7.1.1 Natural image datasets
7.1.2 Domain-specific datasets
7.2 Experiments for AID
7.2.1 Simulation of the user
7.2.2 Evaluation metrics
7.2.3 Competitors
7.2.4 Quantitative results
7.2.5 Qualitative examples
7.3 Experiments for ITAL
7.3.1 Datasets and simulation of user feedback
7.3.2 Competitors
7.3.3 Hyper-parameter tuning
7.3.4 Quantitative performance evaluation
7.3.5 Comparison with MCMI[min] and AdaptAL
7.3.6 Effect of the user model
7.3.7 Sensitivity regarding feature dimensionality
7.3.8 Qualitative results
7.4 Experiments for adaptive pooling
7.4.1 Quantitative results
7.4.2 Qualitative examples
8 Conclusions
8.1 Summary and thesis contributions
8.2 Future work
8.2.1 Semantic embeddings
8.2.2 Learning from small data
8.2.3 Weakly supervised localization
8.2.4 Interactive image retrieval
A Appendix
A.1 Derivation of Eq. (6.5)
A.2 Derivation of Eq. (6.6)
A.3 Additional performance metrics forsemantic image retrieval
A.4 Further semantic retrieval results
A.5 Experimental setup for comparinginstance and category retrievalperformance (Fig. 1.3)
A.6 ITAL failure cases
A.7 Taxonomy used for MS COCO
A.8 Taxonomy used for CIFAR-100
Bibliography
List of Own Publications
List of Figures
List of Tables
List of Definitions, Theorems,and Lemmas
Notations
Acronyms
Index