Understanding FFmpeg with source code: FFMPEG Fundementals

✍ Scribed by Akın Yaprakgül

• Tongue: English
• Leaves: 713
• Category: Library

✦ Synopsis

In this book, we will review the ffmpeg source code and learn about ffmpeg's capabilities. In the first part, we will consider ffmpeg in general terms. In the second part, we will discuss the ffmpeg methods related to video. In this section, we will also examine the ffmpeg filters closely.
In the third part, we will learn the details and methods of audio. In the fourth and fifth part, we will learn about Interlaced video and I-B-P frames. In the sixth part, we will learn what is codec and how does ffmpeg handle it. In the seventh part, we will try to explain how video encoding and audio encoding take place in ffmpeg, with sample codes. In the eighth part, we will try to explain how decoding works in fmmpeg and how to make a sample player with the help of SDL. In parts 9, 10 and 11, we will show the video and audio multiplexing methods. In the twelfth part, how to convert a video file into another format. We will learn all the transcoding steps. In the thirteenth part, we will learn all streaming protocols. In the last part, we will learn how to manage video and audio devices with ffmpeg.

✦ Table of Contents

UNDERSTANDING FFMPEG WITH SOURCE CODE
Acknowledgements
Table of Contents
Introduction
1. FFMPEG Fundementals
FFMPEG Data Structure
AVCodecContext
AVStream
AVFormatContext
AVPacket
AVIOContext
URLContext
URLProtocol
Time information
There are two-way prediction frames:
Time information acquisition
APIs
Reading series
Codec series
Write series
Other
Several libavutil tools
AVOption (AVClass)
AVDictionary
ParseUtil
2. Video and Image basics
Raw Format
FOURCC
FOURCC Identifier
YUV Pixel Formats
Packed YUV Formats
Planar YUV Formats
RGB Pixel Formats
Bayer Data Formats
Create Sample Video
Convert Packet Format to Planar Format
Image Scale
Filter Chain
AVFilter
AVFilterPad
Picture Buffers
Permissions
Filter Links
Usage of Avfilter
Color Change Rectangular Area
Output:
Watermark
input -----------> deltapts0 --> overlay --> output
movie --> scale--> deltapts1 -------+
Output:
Color Effects
Boxblur
Hue
Output:
lut, lutrgb, lutyuv
‘E, PI, PHI’
Crop Video
Draw Box on Video
Output:
Remove Logo From Video
Output:
Draw Grid on Video
Output:
Fade Effect on Video
Output:
Horizontal Flip Video
Output:
Add Noise on Video
Rotate Video
Output:
Trim Video
All Other Filters In Sample
Write Your Own Filter (WYOF)
WYOF Filter Structure
WYOF filter_frame () call flow
WYOF decode_video //ffmpeg.c
WYOF av_buffersrc_add_frame_flags//buffersrc.c
WYOF av_buffersrc_add_frame_internal //buffersrc.c
WYOF request_frame //buffersrc.c
WYOF ff_filter_frame // avfilter.c
WYOF ff_filter_frame_framed //avfilter.c
WYOF Extract the Core Code.
WYOF filter_frame //vf_transform.c
WYOF Walk into ff_filter_frame again // avfilter.c
WYOF Third time into ff_filter_frame // avfilter.c
WYOF How to encode ffmpeg after filter
WYOF do_video_out //ffmpeg.c
WYOF Function Flowchart
Metadata
Read Metadata Sample
Write Metadata Sample
3. Audio Basics
Audio introduction
Mono
Stereo
3D Surround
Number of Samples
Frequency of Sampling
Bit Speed
VBR (Variable Bitrate)
CBR (Constant Bitrate)
MP3
Description of Audio Formats
PCM data format
PCM stream
PCM audio processing
1. Separate left and right channel audio
The audio waveform of the right channel is as follows:
2. Audio pcm resampling 48000 to 44100
3. Using ffmpeg to construct silent frames
4. Halve the volume of the left channel in the PCM16LE two-channel audio sample data
5. Double the sound speed of PCM16LE two-channel audio sampling data
6. Convert PCM16LE two-channel audio sampling data to PCM8 audio sampling data
7. A part of data will be intercepted from PCM16LE mono audio sampling data
8. Convert PCM16LE two-channel audio sampling data to WAVE format audio data
Basic Audio Effects-Volume Control
How to change PCM Sample Rate
How to change the bit depth
4. Interlaced Video
Why TV Interlaced ?
Development and Evolution of Display Technology
2x Field Frequency Display
2x Horizontal Frequency Display
Progressive Display
Multi-format Receiving Single Format Display
Flat Display Device
Psf-Progressive Segmentation
Progressive Format for Broadcast
Progressive Scan Format for Production and Exchange
Progressive Scan Format for Distribution
Digital Video Interface
Digital Display Interface
DVI
HDMI
Analog Interface
Progressive Scan DVD
Interconversion of Progressive and Interlaced Signals
Digital TV Trends-Progressive, Interlaced or Progressive
HD Production Choice: 50i or 25P?
Can I Use 50i Interlaced Scanning Equipment to Copy and Make 25Psf Progressive Scanning Programs?
Deinterlacing
5. I-B-P Frame and Timebase
I-Frame
P-Frame
B-frame
DTS and PTS
Time Base and Timestamp in FFmpeg
1. The Concept of Time Base and Timestamp
2. Three Time Bases tbr, tbn and tbc
3. Internal Time Base AV_TIME_BASE
4. Time Value Form Conversion
5 Time Base Conversion Function
6 Time-Based Conversion During the Encapsulation Process
7 Time Base Conversion During Transcoding
7.1 Video Stream
7.2 Audio Stream
6. Codecs
Video Codec
List of Open-source Codecs
Video Codec List
Audio Codec List
FFMPEG libavcodec
libavcodec/cook.c
libavcodec/avcodec.h
libavcodec/allcodecs.c
libavcodec/cook.c Init
libavcodec/cook.c Close
libavcodec/cook.c Decode
7. Encoding
Video Encoding
1. FFMpeg Required Structure for Video Encoding
2. The Main Steps of FFMpeg Encoding
(2) Initialize the Required FFMpeg Structure as Required
(3), Encoding Loop Body
(4) Finishing Treatment
H.264 Encoding
H265 Encoding
VP8 Encoding
Audio Encoding
AAC Encoding
Picture Encoding
8. Decoding
1. Connect and Open the Video Stream
2. Locate Video Stream Data
3. Prepare the Decoder Codec
4. Decode
Simple Decoder Based on FFmpeg.
Simple Video Player  Based on FFMPEG + SDL
SDL2.0 shows the flow chart of YUV
Simple Audio Player  Based on FFMPEG + SDL
9. Multiplexing
10. Demultiplexing
Demultiplexing to H264 and Mp3
11. Remultiplexing
12. Transcoding
Whole process
Demultiplexing
Decoding
Filter
Encoding
Reuse
Time Stamp Processing During Transcoding
13. Streaming
1. RTMP
1.1. General Introduction
1.2. Handshake
1.3 RTMP Chunk Stream
1.3.1 Message
1.3.2 Chunking
1.4. Different Types of RTMP Message
1.5 Push RTMP Streamer Sample
1.5.1 Introduction
1.5.2 Need to Pay Attention to Package Format
1.5.3 Delay
1.5.4 PTS / DTS Issue
1.5.5 Flowchart
1.6 Save RTMP Streaming Media as a Local FLV File
2. UDP (User Data Protocol)
2.1. Multicasting (Multicasting)
2.2. UDP Broadcast
2.3. UDP Multicast
2.4.  UDP Broadcast and Unicast
2.5.  Pull RTMP Stream to UDP Output Playback
2.6.  UDP Receiving TS Stream
3. RTP (Real-Time Transport Protocol)
3.1. RTP Working Mechanism
4. RTCP (RTP Control Protocol)
4.1.  RTCP Working Mechanism
5. RTSP (RealTime Streaming Potocol)
Analysis of MPEG2 Transport Stream (MPEG2 TS)
14. Devices
Read camera
Screen capture
2. The dshow
References
About The Author