Intended Audience......Page 6
Acknowledgments......Page 8
Contents......Page 9
List of Contributors......Page 11
Part I: Introduction......Page 14
1.1 Introduction......Page 15
1.2 Sentiment, Affect, and Emotion Analytics for Big Multimodal Data......Page 16
1.3 Unsupervised Learning Strategies for Big Multimodal Data......Page 18
1.4 Supervised Learning Strategies for Big Multimodal Data......Page 19
1.5 Multimodal Big Data Processing and Applications......Page 20
Part II: Sentiment, Affect and Emotion Analysis for Big Multimodal Data......Page 22
2.1 Introduction......Page 23
2.2 Multimodal Affect Recognition......Page 24
2.2.1 Information Fusion Techniques......Page 25
2.3 Recent Results......Page 27
2.3.1 Multimodal Sentiment Analysis......Page 28
2.3.2 Multimodal Emotion Recognition......Page 29
2.4 Proposed Method......Page 35
2.4.1 Textual Features......Page 36
2.4.3 Visual Features......Page 37
2.4.5 Contextual LSTM Architecture......Page 38
2.4.6 Fusion......Page 41
2.5.2 Speaker-Independent Experiment......Page 42
2.5.4 Generalizability of the Models......Page 45
2.5.5 Visualization of the Datasets......Page 47
2.6 Discussion......Page 48
References......Page 49
3.1 Introduction......Page 54
3.2 Related Works......Page 56
3.2.1 Text: Emotion and Sentiment Recognition from Textual Data......Page 57
3.2.2 Image: Emotion Recognition from Visual Image Data......Page 59
3.2.4 Video: Emotion and Sentiment Recognition from Video-Based Data......Page 63
3.2.5 Sentiment Recognition from a Combination of Modalities......Page 65
3.3.1 Data Preprocessing......Page 68
3.3.2 Feature Extraction......Page 73
3.4 Simulation......Page 74
3.4.1 Results and Discussions......Page 75
References......Page 76
4.1 Introduction......Page 81
4.3 Social Emotion Mining......Page 83
4.4.1 Problem Definition......Page 84
4.4.2 Network Architecture......Page 85
4.4.3 Parameter Estimation......Page 88
4.5 Experiments......Page 89
4.5.1 Experiment Design......Page 90
4.5.2 Results and Analysis......Page 91
4.6 Conclusion......Page 94
References......Page 97
Part III: Unsupervised Learning Strategies for Big Multimodal Data......Page 100
5.1 Introduction......Page 101
5.2.1 Bi-clusters in Matrices......Page 104
5.2.2 Bi-clustering Analysis Based on Matrix Decomposition......Page 106
5.3 Co-clustering Analysis in Tensor Data......Page 109
5.4.1 High-Order Singular Vector Decomposition......Page 110
5.4.2 Canonical Polyadic Decomposition......Page 112
5.5 Co-clustering in Tensors......Page 118
5.5.1 Linear Grouping in Factor Matrices......Page 119
5.6 Experiment Results......Page 120
5.6.1 Noise and Overlapping Effects in Co-cluster Identification Using Synthetic Tensors......Page 121
5.6.2 Co-clustering of Gene Expression Tensor in Cohort Study......Page 124
5.7 Conclusion......Page 127
References......Page 128
6.1 Introduction......Page 131
6.2 Bi-cluster Analysis of Data......Page 133
6.2.1 Bi-cluster Patterns......Page 134
6.2.2 Multi-objective Optimization......Page 135
6.2.3 Bi-cluster Validation......Page 136
6.3.1 AIS-Based Bi-clustering......Page 137
6.3.2 GA-Based Bi-clustering......Page 138
6.3.3 Multi-objective Bi-clustering......Page 140
6.4 Multi-objective SPEA-Based Algorithm......Page 141
6.5 Bi-clustering Experiments......Page 147
6.5.1 Gene Expression Dataset......Page 148
6.5.3 Facebook Dataset......Page 149
References......Page 153
7.1 Introduction......Page 157
7.2 Low Rank Representation on Grassmann Manifolds......Page 160
7.2.1 Low Rank Representation......Page 161
7.2.2 Grassmann Manifolds......Page 162
7.2.3 LRR on Grassmann Manifolds......Page 163
7.2.4 LRR on Grassmann Manifolds with Gaussian Noise (GLRR-F)......Page 164
7.2.5 LRR on Grassmann Manifolds with 2/1 Noise (GLRR-21)......Page 165
7.2.7 Examples on Video Datasets......Page 167
7.3.1 An Improved LRR on Grassmann Manifolds......Page 170
7.3.2 LRR on Grassmann Manifolds with Tangent Space Distance......Page 171
7.4.1 Weighted Product Grassmann Manifolds......Page 172
7.4.2 LRR on Weighted Product Grassmann Manifolds......Page 174
7.4.3 Optimization......Page 175
7.4.4 Experimental Results......Page 176
7.5 Dimensionality Reduction for Grassmann Manifolds......Page 178
7.5.2 LPP for Grassmann Manifolds......Page 179
7.5.3 Objective Function......Page 180
7.5.4 GLPP with Normalized Constraint......Page 181
7.5.5 Optimization......Page 182
7.5.6 Experimental Results......Page 183
References......Page 185
Part IV: Supervised Learning Strategies for Big Multimodal Data......Page 187
8.1 Introduction......Page 188
8.2.1 Multi-output Neural Network......Page 189
8.2.2 Special Loss Function for Missing Observations......Page 191
8.2.3 Weight Constraints by Special Norm Regularization......Page 193
8.3 The Optimization Method......Page 195
8.4.1 Simulation Study for Information Loss in Demand......Page 199
8.4.2 Simulation Study in Norm Regularization......Page 201
8.4.3 Empirical Study......Page 204
8.4.4 Testing Error Without Insignificant Tasks......Page 206
References......Page 208
9.1 Introduction......Page 211
9.2 Most Basic Recurrent Neural Networks......Page 213
9.3 Long Short-Term Memory......Page 214
9.4 Gated Recurrent Units......Page 216
9.6 Nonlinear AutoRegressive eXogenous Inputs Networks......Page 218
9.7 Echo State Network......Page 219
9.8 Simple Recurrent Unit......Page 220
9.9 TRNNs......Page 221
9.9.1 Tensorial Recurrent Neural Networks......Page 222
9.9.2 Loss Function......Page 224
9.9.3 Recurrent BP Algorithm......Page 225
9.10 Experimental Results......Page 231
9.10.1 Empirical Study with International Relationship Data......Page 232
9.10.2 Empirical Study with MSCOCO Data......Page 235
9.10.3 Simulation Study......Page 239
9.11 Conclusion......Page 245
References......Page 246
10.1 Introduction......Page 248
10.2.1 Tensors and Our Notations......Page 250
10.2.4 Coupled Matrix Tensor Factorization......Page 251
10.3.1 Joint Analysis of Coupled Data......Page 252
10.3.3 Distributed Factorization......Page 253
10.4 SMF: Scalable Multimodal Factorization......Page 255
10.4.2 Block Processing......Page 257
10.4.3 N Copies of an N-mode Tensor Caching......Page 258
10.4.4 Optimized Solver......Page 259
10.4.5 Scaling Up to K Tensors......Page 263
10.5.1 Observation Scalability......Page 264
10.5.2 Machine Scalability......Page 265
10.5.3 Convergence Speed......Page 266
10.5.5 Optimization......Page 268
References......Page 269
Part V: Multimodal Big Data Processing and Applications......Page 272
11.1 Introduction......Page 273
11.2.1 Multimodal Visual Data Registration......Page 275
11.2.2 Feature Detector and Descriptors......Page 277
11.3.3 Videos......Page 278
11.3.5 360 Cameras......Page 279
11.4 System Overview......Page 280
11.5.1 3D Feature Detector......Page 281
11.5.2 3D Feature Descriptors......Page 282
11.5.3 Description Domains......Page 283
11.5.4 Multi-domain Feature Descriptor......Page 284
11.5.5 Hybrid Feature Matching and Registration......Page 285
11.6 Public Multimodal Database......Page 286
11.7 Experiments......Page 287
11.7.1 3D Feature Detector......Page 290
11.7.2 Feature Matching and Registration......Page 292
References......Page 297
12.1 Introduction......Page 300
12.2.1 Type-2 Fuzzy Sets......Page 302
12.2.2 Fuzzy Logic Classification System Rules Generate......Page 303
12.2.3 The Big Bang-Big Crunch Optimization Algorithm......Page 306
12.3 The Proposed Type-2 Fuzzy Logic Scenes Classification System for Football Video in Future Big Data......Page 307
12.3.1 Video Data and Feature Extraction......Page 308
12.3.2 Type-1 Fuzzy Sets and Classification System......Page 310
12.3.3 Type-2 Fuzzy Sets Creation and T2FLCS......Page 311
12.4.2 Rule Base Optimization and Similarity......Page 313
12.5 Experiments and Results......Page 315
12.6 Conclusion......Page 317
References......Page 318
13.1 Introduction......Page 320
13.2.2 Data Fusion Algorithms for Traffic Congestion Estimation......Page 322
13.2.3 Data Fusion Architecture......Page 324
13.3.1 Homogeneous Traffic Data Fusion......Page 325
13.3.2 Heterogeneous Traffic Data Fusion......Page 327
13.4.1 Measurement and Estimation Errors......Page 328
13.5 Conclusion......Page 331
References......Page 335
14.1 Introduction......Page 337
14.2 Background and Literature Review......Page 339
14.2.1 Image Frameworks for Big Data......Page 341
14.3.1 Inter-Frame Parallelism......Page 344
14.3.2 Intra-Frame Parallelism......Page 345
14.4 Parallel Processing of Images Using GPU......Page 347
14.4.2 Implementation of Gaussian Mixture Model on GPU......Page 349
14.4.3 Implementation of Morphological Image Operations on GPU......Page 350
14.4.4 Implementation of Connected Component Labeling on GPU......Page 352
14.4.5 Experimental Results......Page 355
14.5 Conclusion......Page 358
References......Page 359
Chapter 15: Multimodal Approaches in Analysing and Interpreting Big Social Media Data......Page 361
15.1 Introduction......Page 362
15.2 Where Do We Start?......Page 365
15.3 Multimodal Social Media Data and Visual Analytics......Page 367
15.3.2 Emoji Analytics......Page 368
15.3.3 Sentiment Analysis......Page 374
15.4.1 Social Information Landscapes......Page 377
15.4.2 Geo-Mapping Topical Data......Page 379
15.4.3 GPU Accelerated Real-Time Data Processing......Page 381
15.4.4 Navigating and Examining Social Information in VR......Page 386
15.5 Conclusion......Page 388
References......Page 389