Probabilistic Topic Models: Foundation and Application

✍ Scribed by Di Jiang, Chen Zhang, Yuanfeng Song

• Publisher: Springer
• Year: 2023
• Tongue: English
• Leaves: 154
• Category: Library

✦ Synopsis

This book introduces readers to the theoretical foundation and application of topic models. It provides readers with efficient means to learn about the technical principles underlying topic models. More concretely, it covers topics such as fundamental concepts, topic model structures, approximate inference algorithms, and a range of methods used to create high-quality topic models. In addition, this book illustrates the applications of topic models applied in real-world scenarios. Readers will be instructed on the means to select and apply suitable models for specific real-world tasks, providing this book with greater use for the industry. Finally, the book presents a catalog of the most important topic models from the literature over the past decades, which can be referenced and indexed by researchers and engineers in related fields. We hope this book can bridge the gap between academic research and industrial application and help topic models play an increasingly effective role in both academia and industry.

This book offers a valuable reference guide for senior undergraduate students, graduate students, and researchers, covering the latest advances in topic models, and for industrial practitioners, sharing state-of-the-art solutions for topic-related applications. The book can also serve as a reference for job seekers preparing for interviews.

✦ Table of Contents

Preface
Contents
1 Basics
1.1 Linear Algebra
1.1.1 Vector
1.1.2 Matrix
1.1.3 Matrix Operations
1.1.3.1 Matrix Addition and Subtraction
1.1.3.2 Matrix Multiplication
1.1.3.3 Matrix Transposition
1.1.3.4 Matrix Inversion
1.1.4 Orthogonal Matrix
1.1.5 Eigenvalues and Eigenvectors
1.2 Probability Theory
1.2.1 Probability Distribution
1.2.2 Independence
1.2.3 Expected Value, Variance, and Standard Deviation
1.2.4 Common Probability Distributions
1.2.5 Exponential Family
1.2.6 Bayes' Theorem
1.2.7 Conjugate Distribution
1.2.8 Divergence
1.2.8.1 Kullback-Leibler Divergence
1.2.8.2 Jensen-Shannon Divergence
1.3 Bayesian Networks
1.3.1 Representation
1.3.2 Conditional Independence
1.3.3 Uncertain Reasoning
1.3.4 Parameter Learning
1.3.5 Structure Learning
References
2 Topic Models
2.1 Basic Concepts
2.2 Latent Semantic Analysis
2.3 Probabilistic LSA
2.4 Latent Dirichlet Allocation
2.5 SentenceLDA
2.6 Topic over Time Model
2.7 Topical Word Embedding
2.8 Hierarchical Topic Models
2.8.1 Pachinko Allocation Model
2.8.2 Rephil
References
3 Pre-processing of Training Data
3.1 Word Segmentation
3.1.1 Chinese Word Segmentation Tools
3.1.2 Word Segmentation Granularity
3.2 Normalization
3.3 Filtering
3.3.1 Stopword Filtering
3.3.2 Low-Frequency Word Filtering
3.3.3 Part-of-Speech-Based Filtering
3.4 Word Sorting
References
4 Expectation Maximization
4.1 Basics
4.1.1 The First Method of E-Step
4.1.2 The Second Method of E-Step
4.1.3 M-Step
4.2 Convergence of the EM Algorithm
4.3 GEM Algorithm
4.4 Applications of the EM Algorithm
4.4.1 PLSA
4.4.2 PCLSA
References
5 Markov Chain Monte Carlo Sampling
5.1 Markov Chain
5.2 Monte Carlo Method
5.3 Markov Chain Monte Carlo
5.4 Gibbs Sampling
5.4.1 Basic Concepts
5.4.2 Application of Gibbs Sampling in LDA
5.5 Metropolis–Hastings Sampling with Alias Method
5.5.1 Metropolis–Hastings Sampling
5.5.2 Alias Method
5.5.3 Application of Metropolis–Hastings Sampling in LDA
References
6 Variational Inference
6.1 Mathematical Foundation
6.2 Evidence Lower Bound
6.3 Mean Field Variational Inference
6.4 Applying Mean Field Variational Inference to LDA
6.4.1 Joint Distribution
6.4.2 Variational Factorization
6.4.3 Evidence Lower Bound
6.4.4 Variational Optimization with Partial Derivatives
6.5 Comparison of Variational Inference and MCMC
References
7 Distributed Training
7.1 Distributed Computing Architectures
7.1.1 MapReduce
7.1.2 ParameterServer
7.2 Distributed MCMC Sampling
7.2.1 Distributed MCMC Sampling with MapReduce
7.2.2 Distributed MCMC Sampling with ParameterServer
7.3 Distributed Variational Inference
References
8 Parameter Setting
8.1 Hyperparameters
8.1.1 Hyperparameter Optimization Based on MCMC Sampling
8.1.2 Hyperparameter Optimization Based on Variational Inference
8.2 The Number of Topics
8.3 Advanced Metrics for Model Evaluation
References
9 Topic Deduplication and Model Compression
9.1 Topic Deduplication
9.1.1 Precise Topic Deduplication
9.1.1.1 Topic Similarity Analysis
9.1.1.2 Topic Fusion
9.1.2 Fast Topic Deduplication
9.2 Model Compression
9.2.1 Topic-Dimension Compression
9.2.2 Word-Dimension Compression
References
10 Applications
10.1 Semantic Representation
10.1.1 Text Classification
10.1.2 Text Clustering
10.1.3 Click-Through Rate Prediction
10.2 Semantic Matching
10.2.1 Short–Long Text Matching
10.2.1.1 Advertising Page Ranking
10.2.1.2 Keyword Extraction
10.2.2 Long–Long Text Matching
10.3 Semantic Visualization
10.3.1 Basic Visualization
10.3.2 Advanced Visualization
10.3.3 General-Purpose Visualization Tools
10.3.4 Other Applications
References
A Topic Models
A.1 Common Topic Models
A.2 Topic Models with Advanced Features of Documentsor Words
A.3 Topic Models with Supervised Information
A.4 Topic Models with Word Embedding
A.5 Topic Models with Sentiment Information
A.6 Topic Models with Hierarchical Structure
A.7 Topic Models with Network Structure
A.8 Topic Models with Time Information
A.9 Topic Models with Geographic Information
A.10 Topic Models with Bayesian Nonparametrics
A.11 Distributed Training of Topic Models
A.12 Visualization of Topic Models
A.13 Applications in Recommendation System
A.14 Applications in Information Retrieval and InformationExtraction
A.15 Applications in Event Analysis
References