Human Perception of Visual Information: Psychological and Computational Perspectives

✍ Scribed by Bogdan Ionescu (editor), Wilma A. Bainbridge (editor), Naila Murray (editor)

• Publisher: Springer
• Year: 2021
• Tongue: English
• Leaves: 297
• Edition: 1st ed. 2022
• Category: Library

✦ Synopsis

Recent years have witnessed important advancements in our understanding of the psychological underpinnings of subjective properties of visual information, such as aesthetics, memorability, or induced emotions. Concurrently, computational models of objective visual properties such as semantic labelling and geometric relationships have made significant breakthroughs using the latest achievements in machine learning and large-scale data collection. There has also been limited but important work exploiting these breakthroughs to improve computational modelling of subjective visual properties. The time is ripe to explore how advances in both of these fields of study can be mutually enriching and lead to further progress.

This book combines perspectives from psychology and machine learning to showcase a new,  unified  understanding  of  how  images  and  videos  influence  high-level  visual  perception  -  particularly  interestingness,  affective  values  and  emotions,  aesthetic  values,  memorability,  novelty,  complexity,  visual composition and stylistic attributes, and creativity. These human-based metrics are interesting for a very broad range of current applications, ranging from content retrieval and search, storytelling, to targeted advertising, education and learning, and content filtering.

Work already exists in the literature that studies the psychological aspects of these notions or investigates potential correlations between two or more of these human concepts. Attempts at building computational models capable of predicting such notions can also be found, using state-of-the-art machine learning techniques. Nevertheless their performance proves that there is still room for improvement, as the tasks are by nature highly challenging and multifaceted, requiring thought on both the psychological implications of the human concepts, as well as their translation to machines.

✦ Table of Contents

Preface
Contents
The Ingredients of Scenes that Affect Object Search and Perception
1 Introduction
2 Attentional Allocation in Real-World Scenes
2.1 The Role of Low-Level Features
2.2 The Role of Mid-Level Features and Objects
2.3 The Role of Meaning
2.4 The Role of Interestingness and Surprise
3 Guidance of Attention during Real-World Search
3.1 The Role of Scene Gist
3.2 The Role of Local Objects
3.3 The Role of Anchor Objects
3.4 Scene Grammar
4 Object Recognition in Scene Context
4.1 Behavioral Work
4.2 Neurophysiological Work
4.3 Which Scene Ingredients Affect Object Processing?
5 Concluding Remarks and Future Directions
5.1 Relative Contributions of Context Ingredients
5.2 Context in Human and Computer Vision
6 Conclusion
References
Exploring Deep Fusion Ensembling for Automatic Visual Interestingness Prediction
1 Introduction
2 Previous Work
2.1 Media Interestingness
2.2 Ensembling Systems
3 Deep Ensembling
3.1 Dense Networks
3.2 Attention Augmented Dense Networks
3.3 Convolutional Augmented Dense Networks
3.4 Cross-Space-Fusion Augmented Dense Networks
4 Experimental Setup
4.1 Training Protocol
4.2 Dataset
5 Experimental Results
5.1 Baseline Systems
5.2 Results
6 Conclusions
References
Affective Perception: The Power Is in the Picture
1 Introduction
2 Motivation and Emotion
3 Emotional Reactivity Database
4 Analytic Plan
5 Affective Scene Perception
5.1 Evaluative Reports
5.2 Skin Conductance
5.3 Pupil Diameter
5.4 Facial EMG
5.5 Heart Rate
5.6 Functional Amygdala Activity
5.7 Free Recall
6 Discussion
References
Computational Emotion Analysis From Images: Recent Advances and Future Directions
1 Introduction
2 Emotion Representation Models from Psychology
3 Key Computational Problems and Supervised Frameworks
3.1 Emotion Classification and Regression
3.2 Emotion Retrieval
4 Major Challenges
5 Emotion Features
5.1 Hand-Crafted Features
5.2 Deep Features
6 Learning Methods for IEA
6.1 Emotion Classification
6.2 Emotion Regression
6.3 Emotion Retrieval
6.4 Emotion Distribution Learning
6.5 Domain Adaptation
7 Released Datasets
8 Experimental Results and Analysis
8.1 Evaluation Criteria
8.2 Supervised Learning Results
8.3 Domain Adaptation Results
9 Conclusions and Future Research Directions
References
The Interplay of Objective and Subjective Factors in Empirical Aesthetics
1 Introduction
2 Are Aesthetic Preferences Shared or Unique?
3 Objective Predictors of Aesthetic Preference
3.1 Symmetry
3.2 Shape and Composition
3.3 Colour
3.4 Order, Complexity and Global Image Properties
3.5 Do Aesthetic Primitives Exist?
4 Subjective Determinants of Aesthetic Preference
4.1 Effect of Context
4.2 Effect of Artist and Process
4.3 Stability of Contextual Influences
5 Considering the Interplay Between Objective and Subjective Approaches
5.1 Aesthetic Sensitivity
6 Conclusion
References
Advances and Challenges in Computational Image Aesthetics
1 Introduction
1.1 What Makes a Picture Beautiful?
2 Dimensions in Computational Aesthetics
2.1 Input Type
2.2 Scope of the Aesthetic Problem
2.3 Aesthetic Features
2.4 Output Prediction
2.5 Applications
3 Visual Aesthetics Datasets
3.1 Number of Images and Number of Votes per Image
3.2 Image Source
3.3 Voting Methodology and Aesthetic Labels
3.4 Collection Method
3.5 Additional Labels and Attributes
4 Approaches to Computational Aesthetics
4.1 Mathematical Approaches
4.2 Hand-Crafted Features
4.2.1 Initial Works
4.2.2 Considering the Salient Object of the Picture
4.2.3 Including Semantic Information
4.2.4 Multi-Dimensional Approaches
4.2.5 Leveraging Users' Comments
4.3 Generic Features
4.4 Deep Learning Approaches
4.4.1 Preserving Global and Local Information
4.4.2 Content-Adaptive CNNs
4.4.3 Aesthetic Regression
4.4.4 Fusing Hand-Crafted and Deep Features
4.4.5 Learning an Aesthetic Ranking
5 Challenges in Computational Aesthetics: Subjectivity and Explainability
5.1 Dealing with Subjectivity
5.1.1 Predicting Score Distributions
5.1.2 Measures of Subjectivity
5.1.3 Personalized Aesthetics
5.2 Explaining Aesthetic Scores
5.2.1 Visualization Techniques
5.2.2 Generating Text Explanations
5.2.3 Datasets with Aesthetic Attributes
6 Concluding Remarks
References
Shared Memories Driven by the Intrinsic Memorability of Items
1 Introduction
2 Memorability for Visual Events
2.1 How Do We Capture and Operationalize Memorability for a Stimulus?
2.2 Why Should We Consider Memorability?
3 What Does It Mean for Memorability to Be an Intrinsic Image Property?
3.1 Memorability as a Singular Attribute
3.2 Memorability as a Combination of Attributes
3.3 Memorability as an Arrangement of Attributes
4 The Brain Mechanisms Underlying Memorability
4.1 Can We Find Memorability in the Brain?
4.2 Memorability and the Visual System
4.3 Memorability and Attention
4.4 Memorability and the Memory System
4.5 Memorability as a Prioritization Signal
5 Conclusion
References
Memorability: An Image-Computable Measure of Information Utility
1 Introduction
2 Datasets: From Visual Content to Scores
2.1 Consistency Across Participants
2.2 Natural Scenes
2.3 Diverse Photographs
2.4 Specialized Image Collections
2.5 Videos
3 Models: From Pixels to Features
3.1 Support Vector Machines
3.2 Convolutional Neural Networks
3.3 Recurrent Neural Networks
3.4 Generative Adversarial Networks
3.5 Visualizations and Model Interpretability
4 Memorability: From Low-Level to High-Level Features
4.1 Low-Level Pixel Features
4.2 Mid-Level Semantic Features
4.3 High-Level and Contextual Features
5 Applications: From Summarization to Creation
6 Future Directions
7 Concluding Discussion
References
The Influence of Low- and Mid-Level Visual Features on the Perception of Streetscape Qualities
1 Introduction to Environmental Neuroscience
2 Low-Level Visual Features
2.1 Defining Low-Level Features
2.2 The Effect of Low-Level Visual Features on Naturalness Ratings
2.3 The Effect of Low-Level Features on Aesthetic Preferences and Thought Content
2.4 The Effects of Low-Level Visual Features on Disorder Perception
3 Mid-Level Features
3.1 Defining Mid-Level Features
3.2 The Effect of Mid-Level Visual Features on Cognition
3.3 Low-Level and Mid-Level Features in Architectural Scenes
4 Street Psychology and Pedestrian Experiences in Urban Areas
5 Conclusion and Open Challenges
References
Who Sees What? Examining Urban Impressions in Global SouthCities
1 Introduction
2 Related Work
2.1 Perception of Urban Attributes using Crowdsourcing
2.2 Situated Crowdsourcing and Local Knowledge
2.3 Machine Recognition of Urban Perception Attributes
3 Methodology
4 Data: Images and Impressions
4.1 Image Dataset
4.2 Impressions by Local Observers
4.3 Impressions by Non-local Observers
5 Visual Cue Extraction
5.1 Extraction of Visual Cues via CNNs
5.2 Extraction of Visual Cues via Manual Coding and Local Annotation
6 Comparing Impressions Between local and Non-local Observers (RQ1)
6.1 Annotation Quality: Inter-Rater Reliability
6.2 Descriptive Statistics
6.3 Comparing Impressions Between Groups
6.3.1 Pair-Wise Analysis
6.3.2 Scatter Plot Analysis
6.3.3 Correlation Analysis
6.4 Qualitative Analysis of Impressions
7 Inference from Visual Cues for Local and Non-local Impressions (RQ2)
8 Discussion
9 Conclusions
References