Preface
Contents
1 Introduction
1.1 Intelligent Systems
1.2 Computational Intelligence
1.3 About the Third Edition of this Book
Part I Neural Networks
2 Introduction to Artificial Neural Networks
2.1 Motivation
2.2 Biological Background
3 Threshold Logic Units
3.1 Definition and Examples
3.2 Geometric Interpretation
3.3 Limitations
3.4 Networks of Threshold Logic Units
3.5 Training the Parameters
3.6 Variants
3.7 Training Networks
4 General Neural Networks
4.1 Structure of Neural Networks
4.2 Operation of Neural Networks
4.3 Scale Types and Encoding
4.4 Training Neural Networks
5 Multi-layer Perceptrons
5.1 Definition and Examples
5.2 Why Non-linear Activation Functions?
5.3 Function Approximation
5.4 Logistic Regression
5.5 Gradient Descent
5.6 Error Backpropagation
5.7 Gradient Descent Examples
5.8 Variants of Gradient Descent
5.8.1 Manhattan Training
5.8.2 Momentum Term
5.8.3 Nesterov's Accelerated Gradient (NAG)
5.8.4 Self-adaptive Error Backpropagation (SuperSAB)
5.8.5 Resilient Backpropagation
5.8.6 Quick Propagation
5.8.7 Adaptive Subgradient Descent (AdaGrad)
5.8.8 Root Mean Squared Gradient Descent (RMSProp)
5.8.9 Adaptive Subgradient Descent Over Windows (AdaDelta)
5.8.10 Adaptive Moment Estimation (Adam)
5.8.11 Lifting the Derivative of the Activation Function
5.8.12 Weight Decay
5.8.13 Batch Normalization
5.9 Examples for Some Variants
5.10 Initializing the Parameters
5.11 Number of Hidden Neurons
5.12 Deep Learning
5.13 Sensitivity Analysis
6 Radial Basis Function Networks
6.1 Definition and Examples
6.2 Function Approximation
6.3 Initializing the Parameters
6.4 Training the Parameters
6.5 Example of Training
6.6 Generalized Form
7 Self-organizing Maps
7.1 Definition and Examples
7.2 Learning Vector Quantization
7.3 Neighborhood of the Output Neurons
8 Hopfield Networks
8.1 Definition and Examples
8.2 Convergence of the Computations
8.3 Associative Memory
8.4 Solving Optimization Problems
8.5 Simulated Annealing
8.6 Boltzmann Machines
9 Recurrent Networks
9.1 Simple Examples
9.2 Representing Differential Equations
9.3 Vectorial Neural Networks
9.4 Error Backpropagation Through Time
9.5 Long Short-Term Memory
10 Neural Networks: Mathematical Remarks
10.1 Equations for Straight Lines
10.2 Regression
10.3 Activation Transformation
Part II Evolutionary Algorithms
11 Introduction to Evolutionary Algorithms
11.1 Metaheuristics
11.2 Biological Evolution
11.3 Simulated Evolution
11.3.1 Optimization Problems
11.3.2 Basic Notions and Concepts
11.3.3 Building Blocks of an Evolutionary Algorithm
11.4 The n-Queens Problem
11.5 Related Optimization Techniques
11.5.1 Gradient Ascent or Descent
11.5.2 Hill Climbing
11.5.3 Simulated Annealing
11.5.4 Threshold Accepting
11.5.5 Great Deluge Algorithm
11.5.6 Record-to-Record Travel
11.6 The Traveling Salesman Problem
12 Elements of Evolutionary Algorithms
12.1 Encoding of Solution Candidates
12.1.1 Hamming Cliffs
12.1.2 Epistasis
12.1.3 Closedness of the Search Space
12.2 Fitness and Selection
12.2.1 Fitness Proportionate Selection
12.2.2 The Dominance Problem
12.2.3 Vanishing Selective Pressure
12.2.4 Adapting the Fitness Function
12.2.5 The Variance Problem
12.2.6 Rank-Based Selection
12.2.7 Tournament Selection
12.2.8 Elitism
12.2.9 Environmental Selection
12.2.10 Niche Techniques
12.2.11 Characterization of Selection Methods
12.3 Genetic Operators
12.3.1 Mutation Operators
12.3.2 Crossover Operators
12.3.3 Multi-parent Operators
12.3.4 Characteristics of Recombination Operators
12.3.5 Interpolating and Extrapolating Recombination
13 Fundamental Evolutionary Algorithms
13.1 Genetic Algorithms
13.1.1 The Schema Theorem
13.1.2 The Two-Armed Bandit Argument
13.1.3 The Principle of Minimal Alphabets
13.2 Evolution Strategies
13.2.1 Selection
13.2.2 Global Variance Adaptation
13.2.3 Local Variance Adaptation
13.2.4 Covariances
13.2.5 Recombination Operators
13.3 Genetic Programming
13.3.1 Initialization
13.3.2 Genetic Operators
13.3.3 Application Examples
13.3.4 The Problem of Introns
13.3.5 Extensions
13.4 Multi-objective Optimization
13.4.1 Weighted Combination of Objectives
13.4.2 Pareto-Optimal Solutions
13.4.3 Finding Pareto Frontiers with Evolutionary Algorithms
13.5 Special Applications and Techniques
13.5.1 Behavioral Simulation
13.5.2 Parallelization
14 Computational Swarm Intelligence
14.1 Introduction
14.2 Basic Principles of Computational Swarm Intelligence
14.2.1 Swarms in Known Environments
14.2.2 Swarms in Unknown Environments
14.3 Particle Swarm Optimization
14.3.1 Influence of the Parameters
14.3.2 Turbulence Factor
14.3.3 Boundary Handling
14.4 Multi-objective Particle Swarm Optimization
14.4.1 Leader Selection Mechanism
14.4.2 Archiving
14.5 Many-objective Particle Swarm Optimization
14.5.1 Ranking Non-dominated Solutions
14.5.2 Distance Based Ranking
14.6 Ant Colony Optimization
Part III Fuzzy Systems
15 Introduction to Fuzzy Sets and Fuzzy Logics
15.1 Natural Languages and Formal Models
15.2 Fuzzy Sets
15.3 Representation of Fuzzy Sets
15.3.1 Definition Based on Functions
15.3.2 Ξ±-Cuts
15.4 Fuzzy Logic
15.4.1 Propositions and Truth Values
15.4.2 t-Norms and t-Conorms
15.4.3 Aggregation Functions
15.5 Semantics of Membership Degrees
15.5.1 Membership Degrees as Truth Degrees
15.5.2 Membership Degrees as Similarity to a Reference Value
15.5.3 Membership Degrees as Preferences
15.5.4 Membership Degrees as Possibility
15.5.5 Consistent Interpretations of Fuzzy Sets in Applications
15.6 Operations on Fuzzy Sets
15.6.1 Intersection
15.6.2 Union
15.6.3 Complement
15.6.4 Covering and Partition
15.6.5 Linguistic Modifiers
15.7 Fuzzy Sets of Type 2
16 The Extension Principle
16.1 Mappings of Fuzzy Sets
16.2 Mappings of a-cuts
16.3 Cartesian Product and Cylindrical Extension
16.4 Extension Principle for Multivariate Mappings
17 Fuzzy Relations
17.1 Crisp Relations
17.2 Application of Relations
17.3 Logical Deduction with Relations
17.4 Simple Fuzzy Relations
17.5 Composition of Fuzzy Relations
17.6 Fuzzy Relational Equations
18 Similarity Relations
18.1 Similarity
18.2 Fuzzy Sets and Extensional Hulls
18.3 Scaling Concepts
18.4 Fuzzy Sets and Similarity Relations
19 Approximate Reasoning
19.1 Linguistic Variables
19.2 Computing with Words
19.3 Generalized Logical Inference
19.4 Approximation of Functions Using Linguistic If-Then Rules
19.4.1 Approximation of Functions by Using Rules as Constraints
19.4.2 Approximation of Functions by Solving Fuzzy Relational Equations
19.4.3 Approximation of Functions by Interpolation Between Fuzzy Points
20 Fuzzy Control
20.1 Mamdani Fuzzy Controller
20.2 Design of a Mamdani Fuzzy Controller
20.3 Mamdani Controller and Similarity Relations
20.3.1 Interpretation of a Mamdani Controller Using Similarity Relations
20.3.2 Construction of a Mamdani Controller Using Similarity Relations
20.4 TakagiβSugeno Controller
21 Hybrid Systems for Tuning and Learning Fuzzy Systems
21.1 Neuro-Fuzzy Control
21.1.1 Models with Supervised Learning Methods
21.1.2 Models with Reinforcement Learning
21.2 Evolutionary Fuzzy Control
21.2.1 Structure of an Evolutionary Fuzzy Controller
21.2.2 Optimizing Parameters of an Evolutionary Fuzzy Controller
21.2.3 Example
22 Fuzzy Data Analysis
22.1 Fuzzy Methods in Data Analysis
22.2 Fuzzy Clustering
22.2.1 Clustering
22.2.2 Presuppositions and Notation
22.2.3 Classical c-Means Clustering
22.2.4 Fuzzification by Membership Transformation
22.2.5 Fuzzification by Membership Regularization
22.2.6 Comparison
22.3 Analysis of Precise Data with Possibility Theory
22.4 Analysis of Imprecise Data Using Random Sets
22.5 Analysis of Fuzzy Data with Fuzzy Random Variables
Part IV Bayes and Markov Networks
23 Bayesian Networks
24 Elements of Probability and Graph Theory
24.1 Probability Theory
24.1.1 Random Variables and Random Vectors
24.1.2 Independences
24.2 Graph Theory
24.2.1 Background
24.2.2 Join Graphs
24.2.3 Separations
25 Decompositions
26 Evidence Propagation
26.1 Initialization
26.2 Message Passing
26.3 Update
26.4 Marginalization
27 Learning Graphical Models
27.1 Score-Based Approaches
27.1.1 Likelihood of a Database
27.1.2 K2 Algorithm
27.2 Constraint-Based Approaches
28 Belief Revision
28.1 Introduction
28.2 Revision Procedure
28.3 A Real-World Application
29 Decision Graphs
29.1 Motivation
29.2 Definition
29.3 Policies and Strategies
29.4 Finding Optimal Strategies
29.5 Example Scenario
30 Causal Networks
30.1 Causal and Probabilistic Structure
30.2 The Do Operator
Index