FFmpeg, Inside Out

Overview

FFmpeg is an open-source multimedia framework. It can decode, encode, transcode, mux, demux, stream, filter, and play almost all media formats. Many applications and services use it. Examples are the VLC player, browsers, YouTube, Netflix, video editors, chat apps, and automated media pipelines.

Goal of this guide: To understand FFmpeg, you must learn two things. The first is its conceptual model: formats, codecs, streams, timestamps, and filters. The second is its software architecture: libraries, data structures, and pipelines. This guide starts with a high-level view. Then it moves to the small details. It gives you a map of FFmpeg, like the map in the article "The Illustrated Transformer".

What FFmpeg is

FFmpeg is more than one command-line program. It is a set of libraries and tools for multimedia processing. The core is written in C and is optimized for speed.

The core libraries

Library	Main job
`libavutil`	Common utilities: data structures, math helpers, pixel formats, color space utilities, logging, and memory management.
`libavcodec`	Codecs: encodes and decodes audio, video, and subtitles. Also parses bitstreams and supports hardware-accelerated codecs.
`libavformat`	Containers and I/O: demuxers, muxers, and streaming protocols (file, HTTP, RTMP, HLS, and others).
`libavfilter`	Filter framework: directed graphs of audio and video filters that work on decoded raw frames.
`libavdevice`	Capture and playback devices, such as webcams, microphones, and screens.
`libswscale`	Video scaling and pixel format conversion (for example, YUV to RGB).
`libswresample`	Audio resampling and conversion of sample format and channel layout.

The modular pipeline

Command-line tools sit on top of these libraries. The ffmpeg tool is the main processing program. The ffprobe tool shows metadata and stream information. The ffplay tool is a player that uses SDL.

Small details: how libavformat probes a file Layer 3 Depth

FFmpeg must find the format of a file. It does not depend only on the file extension. It uses a method called probing.

The libavformat library calls av_probe_input_format3(). This function goes through each registered demuxer. A demuxer reads a container and splits it into streams. Each demuxer has a read_probe function. FFmpeg reads the first kilobytes of the file, up to probesize. It gives this buffer to every demuxer.

Each demuxer looks for magic numbers, for example the ftyp atom in MP4. It returns a score from 0 to 100 (AVPROBE_SCORE_MAX). The demuxer with the highest score wins.

Formats, streams, and codecs

Containers and codecs

Many people confuse containers and codecs. They are different things.

Container (format): A file structure, like a ZIP file. It holds several streams (video, audio, subtitles). It defines how the streams are interleaved. It also stores metadata and timing. The libavformat library handles containers. Examples: MP4, MKV, AVI, WebM, TS.
Codec: An algorithm that compresses and decompresses one stream of media. The libavcodec library handles codecs. Examples: H.264, VP9, AAC, Opus.

An .mp4 file can hold an H.264 video stream and an AAC audio stream. You can move these streams into an .mkv container without a change to the streams. This is a stream copy. It causes no loss of quality and it is very fast.

Elementary streams

A container separates the media into elementary streams:

Video streams: have width, height, pixel format (for example YUV420p), and frame rate.
Audio streams: have sample rate (for example 48kHz), channels (stereo, 5.1), and sample format.
Subtitle streams: use SRT, ASS, or bitmap subtitles.

Intra frames, inter frames, and GOPs

Video compression uses temporal compression. This means it finds what stays the same between frames. The result is a Group of Pictures (GOP):

A GOP (Group of Pictures)

I-frame (Intra): A complete picture, like a JPEG. You need it to seek in a video.
P-frame (Predicted): Stores only the changes from earlier frames.
B-frame (Bi-directional): Uses earlier and later frames. It gives the best compression. The frames in the bitstream are not in time order.

Core libraries and data structures

Applications build pipelines from these libraries. The ffmpeg tool does the same. These C data structures move through the pipelines:

Key data structures

AVFormatContext & AVStream

These represent the container (for example MP4) and its streams (audio track, video track). They hold the I/O context and the format metadata.

AVPacket

A piece of compressed data, for example one encoded H.264 video chunk. It has timestamps (PTS and DTS) and a stream index.

AVFrame

A decoded, raw audio or video frame. It has pixel data (YUV, RGB) or raw audio samples. It also has metadata, such as size and format.

The developer API flow: You read an AVPacket from an AVFormatContext with av_read_frame(). You send it to a decoder with avcodec_send_packet(). You get a raw AVFrame with avcodec_receive_frame().

Small details: reference counting and AVBufferRef Layer 4 Depth

Both AVPacket and AVFrame are large data structures. A raw 4K video frame can be more than 24MB. A copy of this frame from the decoder to a filter would make FFmpeg slow.

FFmpeg uses reference counting through the AVBufferRef struct. The pixel data stays in one block of heap memory. Many AVFrame instances can point to this block. The AVBufferRef keeps an atomic counter of the frames that point to it.

When you call av_frame_free(), the counter decreases. FFmpeg frees the data memory only when the counter reaches zero. This lets decoders, filters, and encoders pass frames with zero copies.

The `ffmpeg` command line

The ffmpeg program is a pipeline builder. It parses your command line. It builds an internal graph of the libraries above. It runs until the end of the input (EOF).

Command syntax

ffmpeg [global_options] -i [input_file] [input_options] [output_options] [output_file]

The order of the options is important. An option affects only what comes after it.

The -map option

By default, FFmpeg picks the best video stream and the best audio stream. If you have more than one input or track, use -map to choose the streams yourself:

# Take video from input 0, and all audio from input 1 ffmpeg -i video.mp4 -i external_audio.m4a -map 0:v:0 -map 1:a -c:v copy -c:a aac output.mp4

Copy and re-encode

Stream copy (-c copy)

This remuxes the encoded AVPackets from the input container to the output container. It is very fast and lossless. You cannot use filters or scaling.

ffmpeg -i in.mkv -c copy out.mp4

Re-encode (-c:v libx264)

This decodes the packets to AVFrames and applies filters. Then it encodes new AVPackets. It is slower. You must use it to change the resolution or the codec.

ffmpeg -i in.mkv -c:v libx264 out.mp4

Filtergraphs and `libavfilter`

The libavfilter library changes uncompressed frames. It can scale, crop, correct color, and add watermarks.

Simple and complex filtergraphs

Simple (-vf or -af): One input and one output. The filters form a linear chain.
```
ffmpeg -i in.mp4 -vf "scale=1920:1080,hue=s=0" out.mp4
```

Complex (-filter_complex): Many inputs and many outputs. The graph can branch and merge.

ffmpeg -i bg.mp4 -i logo.png -filter_complex "[0:v][1:v]overlay=main_w-overlay_w-10:10[out]" -map "[out]" out.mp4

Graph routing

The link labels, such as [v0] or [out], connect the nodes in the graph like pipes.

Small details: AVFilterLink format negotiation Layer 3 Depth

Two connected filters, such as decode → scale, talk through an AVFilterLink. Before processing starts, libavfilter runs a configuration phase.

The output pad of the decoder negotiates with the input pad of the scale filter. It asks: I output YUV420p. Can you accept it? Suppose the next filter accepts only RGB24. Then the filtergraph adds a hidden format conversion filter in the link. This filter uses libswscale.

Two special hidden filters are the entry and exit points of the graph. The buffersrc filter receives decoded frames. The buffersink filter gives out finished frames.

Time, timestamps, and synchronization

Audio and video must stay in sync. This is a hard problem in multimedia programming. An encoder writes B-frames out of order, so timestamps are very important.

Time base (time_base)

FFmpeg stores timestamps as integer ticks, not as floating-point seconds. The time_base is a fraction, for example 1/90000. To get seconds, use PTS * time_base.

PTS (Presentation Time Stamp)

The time when the player must show the frame on the screen. The player uses it for A/V sync. The values are in time order (1, 2, 3, 4...).

DTS (Decoding Time Stamp)

The time when the decoder must receive the frame. A B-frame needs future frames to be decoded first. So the DTS order is different from the PTS order.

Why is this important? You can cut a file with a hex editor. Container metadata can also become corrupt. In these cases, PTS and DTS do not match. The player decodes the frames but shows them in the wrong order. The video then jumps back and forth.

Small details: H.264 NAL units and POC Layer 4 Depth

At the lowest bitstream level, H.264 and HEVC data is not a list of frames. It is a sequence of NAL units (Network Abstraction Layer units).

The Slice Header in a NAL unit has a field named POC (Picture Order Count). The POC is the internal version of the PTS in the codec. Sometimes an MP4 container has broken or missing PTS metadata. Then the libavcodec decoder parses the NAL units and reads the POC. It rebuilds the correct PTS values, so the video does not stutter.

Hardware acceleration

Data moves between the CPU (system RAM) and the GPU (VRAM) over the PCIe bus. This is slow. Hardware acceleration keeps the video in VRAM for the whole pipeline.

Hardware backends

NVENC/NVDEC: the encode and decode chips in NVIDIA GPUs.
VAAPI / QSV: Intel Quick Sync, and VAAPI for Intel and AMD on Linux.
VideoToolbox: the hardware framework on Apple macOS and iOS.

A zero-copy pipeline

In an ideal pipeline, the hardware decoder decodes straight into VRAM. A hardware filter works on that VRAM surface. The hardware encoder reads it directly.

# Perfect VAAPI zero-copy pipeline ffmpeg -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -i in.mp4 -vf "scale_vaapi=w=1920:h=1080" -c:v h264_vaapi out.mp4

Be careful with hwupload and hwdownload: Suppose you use a CPU filter (such as drawtext) with a GPU encoder. FFmpeg must move the frame to the GPU with the hwupload filter. Suppose you decode with hardware and encode with software. FFmpeg then uses hwdownload. Each move across the PCIe bus slows the pipeline.

Use FFmpeg as a library

You can write C or C++ code, or use bindings such as PyAV. Then you control the lifecycle yourself. This is the standard flow:

avformat_open_input(): opens the file and parses the container.
avformat_find_stream_info(): reads enough packets to find the codecs.
avcodec_open2(): starts your decoder instance.
The loop:
- av_read_frame(): gets an AVPacket.
- avcodec_send_packet(): sends it to the decoder.
- avcodec_receive_frame(): gets a raw AVFrame.
Send the frames to an AVFilterGraph if you filter them.
Send the frames to the encoder with avcodec_send_frame.
Get the encoded packets with avcodec_receive_packet. Mux them with av_write_frame.

Memory management: FFmpeg uses reference counting. You must call av_packet_unref() and av_frame_free() correctly. If you do not, your program leaks large amounts of memory.

Common FFmpeg terms

Term	Definition and context
Demuxer	Reads the container format and splits the interleaved chunks. It sends them to streams. The `libavformat` library handles it.
Muxer	The opposite of a demuxer. It takes separate streams of encoded packets and puts them in a container file (.mp4, .mkv).
BSF (Bitstream Filter)	Changes encoded packets without decoding them. For example, it converts H.264 data between the MP4 and TS formats. Use the `-bsf` option.
GOP	Group of Pictures. A sequence that starts with an I-frame. It is important for seeking and streaming.
Time base	The unit of one clock tick. FFmpeg uses it to read PTS and DTS timestamps. Each stream has its own time base.

Real-world systems

Many media systems use FFmpeg inside, but users do not see it.

VLC Player: A GUI and playback engine built on libavcodec and libavformat. When VLC plays a file, the FFmpeg libraries decode it.
HLS / DASH (streaming platforms): Netflix and YouTube use Adaptive Bitrate Streaming. FFmpeg transcodes the source video into an ABR ladder (for example 1080p, 720p, 480p, 360p). Then libavformat cuts the video into chunks of 2 to 10 seconds (.ts or .m4s). It also makes a manifest file (.m3u8). The player downloads the chunk that fits the current internet speed of the user.
Browsers: Some browsers compile FFmpeg to WebAssembly. They then decode and encode media on the client side.

Pipeline examples

Example 1: Transcode, scale, and lower the bitrate
ffmpeg -i in.mp4 -vf "scale=1280:-1" -c:v libx264 -crf 23 -c:a aac -b:a 128k out.mp4

Flow: Demux MP4 → Decode video and audio → Scale the video width to 1280 (keep the aspect ratio) → Encode video to H.264 (CRF 23 quality) → Encode audio to AAC (128k) → Mux to MP4.

Example 2: Add a watermark (filtergraph)

ffmpeg -i video.mp4 -i logo.png -filter_complex "[0:v][1:v]overlay=10:10[out]" -map "[out]" -map 0:a -c:v libx264 -c:a copy out.mp4

Flow: Read the video and the image. Send both to the overlay filter. The result is the output video. The audio comes directly from input 0. FFmpeg stream-copies it, so it does not re-encode the audio.

Overview

What FFmpeg is

The core libraries

Formats, streams, and codecs

Containers and codecs

Elementary streams

Intra frames, inter frames, and GOPs

Core libraries and data structures

Key data structures

The ffmpeg command line

Command syntax

The -map option

Copy and re-encode

Filtergraphs and libavfilter

Simple and complex filtergraphs

Time, timestamps, and synchronization

Hardware acceleration

Hardware backends

A zero-copy pipeline

Use FFmpeg as a library

Common FFmpeg terms

Real-world systems

Pipeline examples

Signup for Updates:

The `ffmpeg` command line

Filtergraphs and `libavfilter`