Build Your Own Web Server From Scratch In Node.JS : Learn network programming, HTTP, and WebSocket by coding a Web Server

✍ Scribed by James Smith

• Year: 2024
• Tongue: English
• Leaves: 132
• Category: Library

✦ Synopsis

Learn by doing: code a Web server in Node.js
Most people use HTTP daily, but few understand its inner workings. This "Build Your Own X" book dives deep, teaching basics from scratch for a clearer understanding of the tools and tech we rely on.

Network programming.
Protocols & communication.
HTTP in detail.
WebSocket & concurrency.
The project uses Node.js and TypeScript without any dependencies, but many concepts are language-agnostic, so it’s valuable for learners of any language.

Beyond coding exercises
At the end of each chapter, there are discussions about

What’s missing from the code? The gap between toys and the real thing, such as optimizations and applications.
Important concepts beyond coding, such as event loops and backpressure. These are what you are likely to overlook.
Design choices. Why stuff has to work that way? You can learn from both the good ones and the bad ones.
Alternative routes. Where you can deviate from this book.
Build your own X
Why take on a build-your-own-X challenge? A few scenarios to consider

Students: Solidify learning, build portfolio, stand out in future careers.
Developers: Master fundamentals beyond frameworks and tools.
Hobbyists: Explore interests with flexible, extensible projects.
This is part of the “Build Your Own X” book series, which includes books on building your own Redis, database, and compiler.

https://build-your-own.org

✦ Table of Contents

Cover
Contents
01. Introduction
1.1 Why Code a Web Server?
Network Programming
Protocols & Communication
HTTP in Detail
1.2 Build Your Own X From Scratch
1.3 The Book
Project-Centric
Discussions at the End of Chapter
Node.js and TypeScript
Book Series
02. HTTP Overview
2.1 Overview
2.2 HTTP by Example
2.3 The Evolution of HTTP
HTTP/1.0: The Prototype
HTTP/1.1: Production-Ready & Easy-to-Understand
HTTP/2: New Capacities
HTTP/3: More Ambition
2.4 Command Line Tools
03. Code A TCP Server
3.1 TCP Quick Review
Layers of Protocols
TCP Byte Stream vs. UDP Packet
Byte Stream vs. Packet: DNS as an Example
TCP Start with a Handshake
TCP is Bidirectional & Full-Duplex
TCP End with 2 Handshakes
3.2 Socket Primitives
Applications Refer to Sockets by Opaque OS Handles
Listening Socket & Connection Socket
End of Transmission
List of Socket Primitives
3.3 Socket API in Node.js
Step 1: Create A Listening Socket
Step 2: Accept New Connections
Step 3: Error Handling
Step 4: Read and Write
Step 5: Close The Connection
Step 6: Test It
3.4 Discussion: Half-Open Connections
3.5 Discussion: The Event Loop & Concurrency
JS Code Runs Within the Event Loop
JS Code and Runtime Share a Single OS Thread
Concurrency in Node.JS is Event-Based
3.6 Discussion: Asynchronous vs. Synchronous
Blocking & Non-Blocking IO
IO in Node.js is Asynchronous
Event-Based Programming Beyond Networking
3.7 Discussion: Promise-Based IO
04. Promises and Events
4.1 Introduction to async and await
Using Callbacks
Using Promises and await
Creating Promises
4.2 Understanding async and await
Normal Functions Yield to the Runtime by return
async Functions Yield to the Runtime by await
Calling async Functions Start New Tasks
4.3 From Events To Promises
Step 1: Analyze The Solution
Step 2: Handle the 'data' Event
Step 3: Handle the 'end' and 'error' Event
Step 4: Write to Socket
4.3 Using async and await
4.5 Discussion: Backpressure
Waiting for Socket Writes to Complete?
Producers are Bottlenecked by Consumers
Backpressure in TCP: Flow Control
Backpressure Between Applications & OS
Unbounded Queues are Footguns
4.6 Discussion: Events and Ordered Execution
4.7 Conclusion: Promise vs. Callback
05. A Simple Network Protocol
5.1 Message Echo Server
5.2 Dynamic Buffers
The Need for Dynamic Buffers
Coding a Dynamic Buffer
5.3 Implementing a Message Protocol
Step 1: The Server Loop
Step 2: Split Messages
Step 3: Handle Requests
5.4 Discussion: Pipelined Requests
Reduce Latency by Pipelining Requests
Support Pipelined Requests
Deadlock by Pipelining
5.5 Discussion: Smarter Buffers
Removing Data from the Front
Using Fixed Sized Buffer without Reallocations
5.6 Conclusion: Network Programming Basics
06. HTTP Semantics and Syntax
6.1 High-Level Structures
6.2 Content-Length
The Length of the HTTP Header
The Length of the HTTP Body
6.3 Chunked Transfer Encoding
Generate and Send Data on the Fly
Another Layer of Protocol
Chunks Are Not Messages
6.4 Ambiguities in HTTP
The Happy Cases of Body Length
Mind the Nasty Cases
6.5 HTTP Message Format
Read the BNF Language
HTTP Header Fields
6.6 Common Header Fields
6.7 HTTP Methods
Read-Only Methods
Cacheability
CRUD and Resources
Idempotence
Comparison of HTTP Methods
6.8 Discussion: Text vs. Binary
Text is Often Ambiguous
Text is More Work & Error-Prone
6.9 Discussion: Delimiters
Serialization Errors in Delimited Data
Length-Prefixed Data in Binary Protocols
07. Code A Basic HTTP Server
7.1 Start Coding
Step 1: Types and Structures
Step 2: The Server Loop
Step 3: Split the Header
Step 4: Parse the Header
Step 5: Read the Body
Step 6: The Request Handler
Step 7: Send the Response
Step 8: Review the Server Loop
7.2 Testing
Large HTTP Body
Connection Reuse & Pipelining
7.3 Discussion: Nagle's Algorithm
Optimization: Combining Small Writes
Premature Optimization
What People Actually Do in Practice
7.4 Discussion: Buffered Writer
Alternative: Make Buffering Semi-Transparent
Alternative: Add a Buffered Wrapper
08. Dynamic Content and Streaming
8.1 Chunked Transfer Encoding
The Chunked Message Format
Obscure HTTP Features
8.2 Generating Chunked Responses
8.3 JS Generators
Producers, Consumers and Queues
JS Generators as Producers
Consume from JS Generators
8.4 Reading Chunked Requests
Chunk Parser as a Producer
Decode the Chunk Format
Chunked Encoding Produces a Byte Stream
8.5 Discussion: Debugging with Packet Capture Tools
Capture & Inspect Data with tcpdump
Analyze Packets with Wireshark
Search Packet Data with ngrep
8.6 Discussion: WebSocket
Polling for Updates
Real-Time Server Push
WebSocket is Message-Based
WebSocket is NOT Request-Response
09. File IO & Resource Management
9.1 File IO in Node.JS
9.2 Serving Disk Files
Step 0: Add a Handler for Static Files
Step 1: Open and Close a File
Step 2: stat() a File
Step 3: Construct the BodyReader
Step 4: Make Sure the File is Closed
9.3 Discussion: Manual Resource Management
Resources Are Owned by Something
Resources Owned by Functions or Scopes
Ownerships Can be Transferred
Owners Are Chained
How to Do Cleanups in a Generator
9.4 Discussion: Reusing Buffers
Buffer Allocations Have Costs
Using Uninitialized Buffers
Reusing Large Buffers
Pooled Buffers
Mind the Lifetime When Reusing Buffers
10. Range Requests
10.1 How Range Requests Work
The Range Header Field Syntax
The Effective Range with Examples
Range Requests Are Optional
Range Responses Are Indicated by 206
Content-Range Returns the Effective Range
Announcing Range Support
Probing with the HEAD Method
Multiple Ranges with the Multipart Message
Discussion: Is Multipart Message Technically Necessary?
Possible Cases for Clients
10.2 Implementing Range Requests
Step 1: Parsing the Range Header Field
Step 2: Reading a Portion of a File
Step 3: Generating Range Responses
Step 4: Adding the HEAD Method
11. HTTP Caching
11.1 Cache Validator
Validate by Timestamp
Validate by ETag
Validate for Range Requests
Summary of Validation Headers
Implementing the Timestamp Validator
11.2 Discussion: Server-Side Cache Control
When to Revalidate Cache
Control the TTL
Make the Client Always Check the Server
Prevent Caching
Reduce Fresh Validations
Heuristic Caching
Non-Client Caching
Shared and Private Caches
Caching in the Cloud
12. Compression & the Stream API
12.1 How HTTP Compression Works
Negotiating with Accept-Encoding
Compress with Content-Encoding
Compression and Content-Length
Compression and Caching
Compressed Uploads
Compression and Range Requests
Compress with Transfer-Encoding
12.2 Data Processing with Pipes
Data Processing and IO
Unix Pipes for Composable Data Processors
Abstract Pipes
12.3 Exploring the Stream API in Node.JS
List of Stream Interfaces
Using Pipes on Stream Interfaces
12.4 Implementing HTTP Compression
Step 1: Check Header Fields and Enable Compression
Step 2: Understanding the Gotchas without Pipes
Step 3: Implementing stream.Readable
Step 4: Creating a Pipe
Step 5: Reading from stream.Readable
Step 6: Test It
Step 7: Flush the Compressor Buffer
12.5 Discussion: Refactoring to Stream
12.6 Discussion: High Water Mark and Backpressure
One-Item Queue
Backpressure by High Water Mark
Batching Data with Queues
High Water Mark in Node.JS
13. WebSocket & Concurrency
13.1 Establishing WebSockets
High Level Overview
Upgrade From HTTP/1.1
WebSocket Handshake
The Connection Header Field
13.2 WebSocket Protocol
Frame Format
Types of Frames
Fragmented Messages
13.3 Introduction to Concurrent Programming
Race Conditions
Atomic Operations
Mutual Exclusion with Mutexes
Multiplexing with Queues
Flow Control with Blocking Queues
Cancellation with Closeable Queues
The Blocking Queue as a Synchronization Primitive
13.4 Coding a Blocking Queue
Step 1: Analyze the Problem
Step 2: Push and Pop
Step 3: Close the Queue
Step 4: Add a Buffer
13.5 WebSocket Server
Step 1: Design the Message Interface
Step 2: Design the Server Application Interface
Step 3: Design the Server Tasks
Step 4: Start Server Tasks
Step 5: Write Messages to the Socket
Step 6: Parse Frames From the Socket
Step 7: Integrate with the HTTP Server
Step 8: Test and Debug
13.6 Discussion: WebSocket in the Browser
No Backpressure for WebSocket in the Browser
Application Level Backpressure
Message Size is Limited
13.7 Conclusion: What We Have Learned