Modern Distributed Tracing in .NET: A practical guide to observability and performance analysis for microservices

✍ Scribed by Liudmila Molkova

• Publisher: Packt Publishing
• Year: 2023
• Tongue: English
• Leaves: 336
• Edition: 1
• Category: Library

✦ Synopsis

Instrument .NET apps using OpenTelemetry and explore logs and .NET diagnostic tools to debug, monitor, and analyze the performance of complex systems in the cloud

Purchase of the print or Kindle book includes a free PDF eBook

Key Features

• Get a clear understanding of complex systems using .NET and OpenTelemetry
• Adopt a systematic approach toward performance analysis and debugging
• Explore instrumentation techniques for common distributed patterns

Book Description

As distributed systems become more complex and dynamic, their observability needs to grow to aid the development of holistic solutions for performance or usage analysis and debugging. Distributed tracing brings structure, correlation, causation, and consistency to your telemetry, thus allowing you to answer arbitrary questions about your system and creating a foundation for observability vendors to build visualizations and analytics.

Modern Distributed Tracing in .NET is your comprehensive guide to observability that focuses on tracing and performance analysis using a combination of telemetry signals and diagnostic tools. You'll begin by learning how to instrument your apps automatically as well as manually in a vendor-neutral way. Next, you'll explore how to produce useful traces and metrics for typical cloud patterns and get insights into your system and investigate functional, configurational, and performance issues. The book is filled with instrumentation examples that help you grasp how to enrich auto-generated telemetry or produce your own to get the level of detail your system needs, along with controlling your costs with sampling, aggregation, and verbosity.

By the end of this book, you'll be ready to adopt and leverage tracing and other observability signals and tools and tailor them to your needs as your system evolves.

What you will learn

• Understand the core concepts of distributed tracing and observability
• Auto-instrument .NET applications with OpenTelemetry
• Manually instrument common scenarios with traces and metrics
• Systematically debug issues and analyze the performance
• Keep performance overhead and telemetry volume under control
• Adopt and evolve observability in your organization

Who this book is for

This book is for software developers, architects, and systems operators running .NET services who want to use modern observability tools and standards and take a holistic approach to performance analysis and end-to-end debugging. Software testers and support engineers will also find this book useful. Basic knowledge of the C# programming language and .NET platform is assumed to grasp the examples of manual instrumentation, but it is not necessary.

Table of Contents

1. Observability Needs of Modern Applications
2. Native Monitoring in .NET
3. The .NET Observability Ecosystem
4. Low-Level Performance Analysis with Diagnostic Tools
5. Configuration and Control Plane
6. Tracing Your Code
7. Adding Custom Metrics
8. Writing Structured and Correlated Logs
9. Best Practices
10. Tracing Network Calls
11. Instrumenting Messaging Scenarios
12. Instrumenting Database Calls
13. Driving Change
14. Creating Your Own Conventions
15. Instrumenting Brownfield Applications

✦ Table of Contents

Cover
Title Page
Copyright and Credit
Dedicated
Foreword
Contributors
Table of Contents
Preface
Part 1: Introducing Distributed Tracing
Chapter 1: Observability Needs of Modern Applications
Understanding why logs and counters are not enough
Logs
Events
Metrics and counters
What’s missing?
Introducing distributed tracing
Span
Tracing – building blocks
Reviewing context propagation
In-process propagation
Out-of-process propagation
Ensuring consistency and structure
Building application topology
Resource attributes
Performance analysis overview
The baseline
Investigating performance issues
Summary
Questions
Further reading
Chapter 2: Native Monitoring in .NET
Technical requirements
Building a sample application
Log correlation
On-demand logging with dotnet-monitor
Monitoring with runtime counters
Enabling auto-collection with OpenTelemetry
Installing and configuring OpenTelemetry
Exploring auto-generated telemetry
Debugging
Performance
Summary
Questions
Chapter 3: The .NET Observability Ecosystem
Technical requirements
Configuring cloud storage
Using instrumentations for popular libraries
Instrumenting application
Leveraging infrastructure
Configuring secrets
Configuring observability on Dapr
Tracing
Metrics
Instrumenting serverless environments
AWS Lambda
Azure Functions
Summary
Questions
Chapter 4: Low-Level Performance Analysis with Diagnostic Tools
Technical requirements
Investigating common performance problems
Memory leaks
Thread pool starvation
Profiling
Inefficient code
Debugging locks
Using diagnostics tools in production
Continuous profiling
The dotnet-monitor tool
Summary
Questions
Part 2: Instrumenting .NET Applications
Chapter 5: Configuration and Control Plane
Technical requirements
Controlling costs with sampling
Head-based sampling
Tail-based sampling
Enriching and filtering telemetry
Span processors
Customizing instrumentations
Resources
Metrics
Customizing context propagation
Processing a pipeline with the OpenTelemetry Collector
Summary
Questions
Chapter 6: Tracing Your Code
Technical requirements
Tracing with System.Diagnostics or the OpenTelemetry API shim
Tracing with System.Diagnostics
Tracing with the OpenTelemetry API shim
Using ambient context
Recording events
When to use events
The ActivityEvent API
Correlating spans with links
Using links
Testing your instrumentation
Intercepting activities
Filtering relevant activities
Summary
Questions
Chapter 7: Adding Custom Metrics
Technical requirements
Metrics in .NET – past and present
Cardinality
When to use metrics
Reporting metrics
Using counters
The Counter class
The UpDownCounter class
The ObservableCounter class
The ObservableUpDownCounter class
Using an asynchronous gauge
Using histograms
Summary
Questions
Chapter 8: Writing Structured and Correlated Logs
Technical requirements
Logging evolution in .NET
Console
Trace
EventSource
ILogger
Logging with ILogger
Optimizing logging
Capturing logs with OpenTelemetry
Managing logging costs
Pipelines
Backends
Summary
Questions
Part 3: Observability for Common Cloud Scenarios
Chapter 9: Best Practices
Technical requirements
Choosing the right signal
Getting more with less
Building a new application
Evolving applications
Performance-sensitive scenarios
Staying consistent with semantic conventions
Semantic conventions for HTTP requests
General considerations
Summary
Questions
Chapter 10: Tracing Network Calls
Technical requirements
Instrumenting client calls
Instrumenting unary calls
Configuring instrumentation
Instrumenting server calls
Instrumenting streaming calls
Basic instrumentation
Tracing individual messages
Observability in action
Summary
Questions
Chapter 11: Instrumenting Messaging Scenarios
Technical requirements
Observability in messaging scenarios
Messaging semantic conventions
Instrumenting the producer
Trace context propagation
Tracing a publish call
Producer metrics
Instrumenting the consumer
Tracing consumer operations
Consumer metrics
Instrumenting batching scenarios
Batching on a transport level
Processing batches
Performance analysis in messaging scenarios
Summary
Questions
Chapter 12: Instrumenting Database Calls
Technical requirements
Instrumenting database calls
OpenTelemetry semantic conventions for databases
Tracing implementation
Tracing cache calls
Instrumenting composite calls
Adding metrics
Recording Redis metrics
Analyzing performance
Summary
Questions
Part 4: Implementing Distributed Tracing in Your Organization
Chapter 13: Driving Change
Understanding the importance of observability
The cost of insufficient observability
The cost of an observability solution
The onboarding process
The pilot phase
Avoiding pitfalls
Continuous observability
Incorporating observability into the design process
Housekeeping
Summary
Questions
Further reading
Chapter 14: Creating Your Own Conventions
Technical requirements
Defining custom conventions
Naming attributes
Sharing common schema and code
Sharing setup code
Codifying conventions
Using OpenTelemetry schemas and tools
Semantic conventions schema
Defining event conventions
Summary
Questions
Chapter 15: Instrumenting Brownfield Applications
Technical requirements
Instrumenting legacy services
Legacy service as a leaf node
A legacy service in the middle
Choosing a reasonable level of instrumentation
Propagating context
Leveraging existing correlation formats
Passing context through a legacy service
Consolidating telemetry from legacy monitoring tools
Summary
Questions
Assessments
Index
Other Books You May Enjoy