Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation (Lecture Notes in Computer Science, 5790)

✍ Scribed by Emiel Krahmer (editor), Mariet Theune (editor)

• Publisher: Springer
• Year: 2010
• Tongue: English
• Leaves: 363
• Category: Library

✦ Synopsis

Natural language generation (NLG) is a subfield of natural language processing (NLP) that is often characterized as the study of automatically converting non-linguistic representations (e.g., from databases or other knowledge sources) into coherent natural language text. In recent years the field has evolved substantially. Perhaps the most important new development is the current emphasis on data-oriented methods and empirical evaluation. Progress in related areas such as machine translation, dialogue system design and automatic text summarization and the resulting awareness of the importance of language generation, the increasing availability of suitable corpora in recent years, and the organization of shared tasks for NLG, where different teams of researchers develop and evaluate their algorithms on a shared, held out data set have had a considerable impact on the field, and this book offers the first comprehensive overview of recent empirically oriented NLG research.

✦ Table of Contents

Title Page
Preface
Table of Contents
Text-to-Text Generation
Probabilistic Approaches for Modeling Text Structure and Their Application to Text-to-Text Generation
Introduction
ContentModels
Computational Modeling
Applications to Text-to-Text Generation
Coherence Models
Computational Modeling
Application to Text-to-Text Generation
Conclusions
References
Spanning Tree Approaches for Statistical Sentence Generation
Introduction
Generation via Spanning Trees
Preliminiaries
The Chu-Liu/Edmonds Algorithm
Generating a Word Sequence
Argument Satisfaction and Spanning Trees
The Argument Satisfaction Problem
Modelling Argument Satisfaction
Assignment with an Argument Satisfaction Model
Assigning Words to Argument Positions
An Assignment-Based Spanning Tree Algorithm
An Example
Evaluation of Tree Building Algorithms
The String Regeneration Evaluation Framework
Algorithms and Baselines
Evaluating the Assignment-Based Algorithm
Comparing against Large N-Gram Models
Comparison to Related Work
Statistical Surface Realisers
Text-to-Text Generation
Parsing and Semantic Role Labelling
Conclusions
References
On the Limits of Sentence Compression by Deletion
Introduction
Material
Analysis
Edit Operations
Percentage of Subsequences
Problematic Non-subsequences
Semantic Relations between Aligned Phrases
Perspectives for Automatic Paraphrase Extraction
Exploring Sentence Compression for Dutch
Sentence Compression as Tree Transduction
Application to Dutch
Some General Issues
Discussion
References
NLG in Interaction
Learning Adaptive Referring Expression Generation Policies for Spoken Dialogue Systems
Introduction
Related Work
Reinforcement Learning Environment
Dialogue System
User Simulation
Environment Simulation
Reward Function
Learning REG Policies
Evaluation and Baselines
Conclusion
Future Work
References
Modelling and Evaluation of Lexical and Syntactic Alignment with a Priming-Based Microplanner
Introduction
Related Work
A Priming-Based Model of Alignment
The Alignment-Capable Microplanner SPUD Prime
Evaluation
Demonstrating Lexical and Syntactic Alignment
Empirical Evaluation Method
Corpus 1: Learning of Referring Nouns
Corpus 2: Implicit Acquisition of Referring Nouns
Discussion
Conclusion
References
Natural Language Generation as Planning under Uncertainty for Spoken Dialogue Systems
Introduction
NLG as Planning under Uncertainty
The Information Presentation Problem
MATCH Corpus Analysis
Method: The RL-NLG Model
Experiments
User Simulation
Realizer Model
Reward Function
State and Action Space
Baseline Information Presentation Policies
Results
Conclusion
References
Referring Expression Generation
Generating Approximate Geographic Descriptions
Introduction
Background
Observations on Geographic Descriptions from the Weather Domain
End Users’ Geographic Knowledge
Experts’ Descriptive Strategy
Domain Specific Constraints on Description
Generating Approximate Geographic Descriptions
Geographic Characterisation
Data Interpretation
Frame of Reference Selection
Attribute Selection
Evaluation and Discussion
Conclusions
References
A Flexible Approach to Class-Based Ordering of Prenominal Modifiers
Introduction
Related Work
Early Approaches
Computational Approaches
The Problem of Ordering Prenominal Modifiers in NLG
Towards a Solution
Materials
Code Modules
Method
Classification Scheme
Evaluation
Results
Discussion
References
Attribute-Centric Referring Expression Generation
Introduction
What Do People Do?
An Alternative Paradigm
What We Can Learn from the Data
Learning Algorithms for Description Construction
Learning Heuristics for Attribute Inclusion
What’s Missing?
Conclusions
References
Evaluation of NLG
Assessing the Trade-Off between SystemBuilding Cost and Output Quality in Data-to-Text Generation
Introduction
Data
Four Ways to Build an NLG System
Rule-Based NLG
PCFG Generation
PSCFG Generation
SMT Methods
Ten Weather Forecast Text Generators
SUMTIME-Hybrid
PCFG Generators
PSCFG Generators
PBSMT Generators
Evaluation Methods
Automatic Evaluation Methods
Human Evaluation
Results
Issues Around Input Representations
Discussion
Conclusions
References
Human Evaluation of a German Surface Realisation Ranker
Introduction
A Realisation Ranking System for German
Experiment Design
Part 1
Part 2
Part 3
Results
How Good Were the Strings?
Did the Human Judges Agree with the Original Authors?
Effects of Context
Annotator Agreement
Correlation with Automatic Metrics
String-Based Metrics
Syntactic Metrics
Related Work
Conclusion
References
Structural Features for Predicting the Linguistic Quality of Text
Introduction
Sentence Fluency and Machine Translation
Features
Feature Analysis
Distinguishing Human from Machine Translations
Pairwise Fluency Comparisons
Feature Analysis: Differences among Tasks
Applications to Human-Authored Text
Identifying Hard-to-Read Sentences in Wall Street Journal Texts
Correlation with Overall Text Quality
Predicting Linguistic Quality for Multi-document Summarization
Summarization Data
Predictors of Linguistic Quality
Experimental Setup
Results
Conclusion
References
Towards Empirical Evaluation of Affective Tactical NLG
Introduction
Measuring Emotions
Linguistic Choice
Polarity and Magnitude
Tactical Methods
Study I
Background for the Study
Test Texts
Text Validation
Experiment
Discussion
Study II
Background for the Study
Test Texts
Text Validation
Experiment
Discussion
Conclusions and Future Directions
References
Shared Task Challenges for NLG
Introducing Shared Tasks to NLG: The TUNA Shared Task Evaluation Challenges
Introduction
Background to the TUNA Tasks
The TUNA Corpus
The TUNA Challenges 2007-2009
Structure and Scope of the TUNA Shared Tasks
Datasets
Participation
Evaluation Methods
Comparing Different Evaluation Methods
Minimality versus Humanlikeness
Measures of Humanlikeness versus Task Effectiveness
Human Intrinsic Measures versus Automatic Intrinsic and Human Extrinsic Measures
Summary and Implications
Conclusion
References
Generating Referring Expressions in Context: The GREC Task Evaluation Challenges
Background
Overview of GREC Research Programme
The GREC Task in General Terms
Summary of Differences between the Two GREC Datasets (GREC-People and GREC-2.0)
The GREC-2.0 Corpus
Types of Referential Expressions Annotated
Comments on Some Aspects of Annotation
XML Format
The GREC-People Corpus
Annotation of Referring Expressions in GREC-People
Further Explanation of Some Aspects of the Annotations
XML Annotation
GREC-MSR and GREC-NEG Task Definitions
GREC-MSR
GREC-NEG
GREC Evaluation Procedures
Automatic Intrinsic Evaluations of Humanlikeness
Automatic Extrinsic Evaluation of Clarity
Human-Assessed Intrinsic Evaluation of Clarity and Fluency
Human-Based Extrinsic Evaluation of Ease of Comprehension
GREC-MSR’08/09—Participating Systems and Results
Systems
Results
GREC-NEG’09—Participating Systems and Results
Systems
Results
Discussion
Evaluation Methods
Development of the GREC Tasks
Concluding Remarks
References
The First Challenge on Generating Instructions in Virtual Environments
Introduction
Evaluating NLG Systems
Previous Work
Internet-Based Evaluation of NLG Systems
Access to Subjects
Evaluation
The GIVE Challenge
GIVE as a Special Case of the Web-Based Method
Game Worlds
Software Infrastructure
Timeline
Systems Participating in GIVE-1
Results
Demographics
Objective Measures
Subjective Measures
Further Analysis
Validating the Experimental Approach
The Laboratory Experiment
Results
Discussion
Conclusion
References
Author Index