GStreamer - Part 2 by Deepak Sharma Sharma

GStreamer Multimedia Framework Part 2

Contents

Description

Part 1

Introduction to GStreamer

Part 2

GStreamer Plugin Internals

Part 3

Advanced GStreamer Concepts

Part 4

Concepts of GStreamer Application

Part 5

An Example Application Code

Important GStreamer Plugin Components Overview

The GStreamer Element • An Element in GStreamer is the most important class of objects. GStreamer plugins that have been discussed so far are all Elements. • Element performs one defined specific function. Example  Decode MP3  Parse AVI file  Render PCM audio  Read from file

• A sequence or combination of elements is used by GStreamer to achieve a defined multimedia scenario. Such a sequence of elements is called a graph or pipeline. Examples of multimedia scenarios could be:  Playback an audio video stream  Record a video stream using camera

• The element is represented by GstElement abstract data type. All plugins are derived from this data type.

Element Object Hierarchy

The fundamental GObject data type provided by GLib

GObject

The GStreamerâ&#x20AC;&#x2122;s base object type. Adds the following functionalities on top of Gobject: GstObject

Reference counting

Name

Locking mechanism for synchronisation

The GStreamerâ&#x20AC;&#x2122;s Element object type. Defines GstElement

the plugin interface which every module has to implement to qualify as a GStreamer plugin

Pads • Pads belong to Elements and they are the ports through which data flows in and out of elements. • GStreamer constructs graph of multimedia components by connecting the elements at their pads. • Apart from read/write properties of elements, pads are the only interface through which data is given to and taken out from elements • In any given pad, data can flow in only one defined direction. Pads are thus classified in to two types based on the direction of data flow  Sink pad: receives data  Source pad: generates data

sink

Receives Data

pad

element

src pad

Outputs Data

Element & Pad configurations

sink pad

source element

sink element

src pad

src sink pad

filter element

src

sink

pad

demuxers src pad

sink pad

multiplexers sink

src pad

pad

Pad classification based on availability

Pad Classification

Description

Examples

Always Pads

Static pads that are present always. They are created along with the element itself.

SRC pad of an MP3 decoder

Dynamic pads that could be added to an element at run time.

SRC pad(s) of any file format parser elements such as 3GP parser or AVI parser. Audio SRC pad may be added only if the audio track is detected in the media file

Sometimes Pads

Application can come to know of a pad being added through the â&#x20AC;&#x153;pad-added signal.â&#x20AC;? Request Pads

Dynamic pads that are added to an element based on request from application.

SINK pad of a 3GP parser

SRC pad(s) of a tee element

Element chaining through pads

alsasrc

src

sink

mp3 encoder

mp3 file src

sink

format

src

sink

filesink

writer

Direction of Data Flow

Bin • A bin is a container which can hold other elements • The advantage of using a bin is that, it allows for a set of related elements to be grouped together and treated as a single logical element • From a programmer’s perspective Bin is also a Element, i.e. it derives from the GstElement class. This allows for GStreamer core to treat a bin and a normal element as the same. This also allows for a bin to be included in to another bin.

GObject GstObject GstElement GstBin

Bin as a container

video bin

sink

divx decoder

src

sink

colour converter

src

sink

xvimage sink

Ghost Pad

â&#x20AC;˘ Bin does not have a pad of itâ&#x20AC;&#x2122;s own. â&#x20AC;˘ A pad belonging to an element within the bin can be made to appear as a pad for the bin. This is known as a ghost pad

Pipeline â&#x20AC;˘ In GStreamer, a pipeline is a predefined and special type of bin. â&#x20AC;˘ Pipeline is the top most level of bin. In other words, a pipeline cannot be added in to another bin.

GObject GstObject GstElement GstBin

GstPipeline

Features of Pipeline • A generic bin allows for logical grouping of elements and enables the core to treat a set of elements as a single element • In addition to this feature, a pipeline adds the following functionalities  Takes care of scheduling  Selects and distributes a clock which will be used by all elements  Provides a bus which can be used by applications to get message notifications

• A GStreamer application needs to have at least one pipeline element. The top level bin in a graph has to be a pipeline.

Component Architecture Example

Media Player Application

GTK

GStreamer Framework Core

GDK Pipeline X Window File Src

Queue

Video Decoder

Video Sink

Queue

Audio Decoder

Audio Sink

Parser

Graphics Plane (Menus and Labels)

Video/Audio (Media) Plane

Concept of Capabilities All about media representation and negotiation

“Caps” : GStreamer’s Solution for Media Representation • GStreamer is introducing the concept of capabilities or caps in short as an efficient mechanism for media representation and negotiation. • GStreamer uses the data type GstCaps to represent capabilities • GstCaps is implemented as an array of objects of type GstStructure • Each GstStructure object has a collection of key/value pairs • Each Key/Value pair can be of different type (either fundamental or derived) GstCaps Holds an array of objects of type GstStructure GstStructure

GstStructure

Collection of key/value pairs

GstStructure

Each GstStructure represents one media type

Caps …contd. • The key in each key/value pair is generally a string representing a field or property (e.g. samplingrate, channels, etc). • The value can be of any data type. Example:  INT, FRACTION, BOOLEAN, FLOAT, STRING

• GStreamer adds new types to represent values in the form of  Ranges (Int, float and fraction)  Lists (Arrays)  FOURCC codes

• Looking at the hierarchy of objects used in Caps data type, it may seem to be difficult to construct, manage and retrieve objects of GstCaps data type. However, GStreamer provides utility functions to convert from GstCaps structure to string representation and from string representation to GstCaps structure. Some sample APIs include  gst_caps_new_simple  gst_caps_from_string  gst_caps_to_string

Capabilities of an Element • In a GStreamer element, as the data passes only through pads of the element, the media handling capability of the element is entirely described by the caps associated with it’s pads. • Element capabilities:  The type of media that the element can receive as input is identified by the caps associated with the sink pad(s) of the element  The type of media that the element can generate as output is identified by the caps associated with the source pad(s) of the element

• GStreamer caps can be classified in to following based on functionality  Template caps  Allowed caps  Negotiated caps

• GStreamer caps can be classified in to following based on content  Full caps  Simple caps  Fixed caps

Caps classification based on its content • Full caps is the most generic. It… 

Contains more than one GstStructure object within, i.e. contains more than one media type



The values for the various properties (keys) in each GstStructure object can be either a range or list of values or a single/distinct value

• Simple caps is a specialized case of full caps, where in 

There is only one GstStructure object within, i.e. contains the description of only one media type



The values for the various properties (keys) for that GstStructure object can be either a range or list of values or a single/distinct value

• Fixed caps is a specialized case of simple caps, where in 

There is only one GstStructure object within, i.e. contains the description of only one media type



The values for the various properties (keys) for that GstStructure object need to have only single/distinct values (no ranges or lists)

Caps classification based on functionality • Template caps:  Applicable to a single pad  Represents the complete set of media types (universal set) that a pad is designed to handle. Template caps would be of the form “full caps.” Examples:

sink audio/x-raw-int, channels={1, 2}, rate=[16000, 96000],

audio sink

Template caps of sink pad

width=16, depth=16, audio/mpeg, mpegversion=1, layer=3, channels={1, 2}, rate=[16000, 96000]

Caps classification based on functionality • Allowed Caps:  Applicable to a connection between pads of two plugins  Represents the set of media that can flow between two plugins. This will typically be an intersection of the template caps of the pads at each end of the connection. Allowed caps would also be of the form “full caps.” mp3 decoder

src

sink

Template Caps

Template Caps audio/mpeg, mpegversion=1, layer=3, channels=2, rate=[8000, 48000]

audio sink

Allowed Caps on this Link

audio/x-raw-int, channels={1, 2},

audio/mpeg, mpegversion=1, layer=3, channels=2, rate=[16000, 48000]

rate=[8000, 96000], width=16, depth=16, audio/mpeg, mpegversion=1, layer=3, channels={1, 2}, rate=[8000, 96000]

Caps classification based on functionality • Negotiated Caps:  Applicable to pads  Represents the set of media that the pad is currently configured to handle based on the stream being processed. Pads are said to be in negotiated state when an agreement has been reached on what would be the final media type that would flow through the pipeline.  Negotiated caps would be of the form “fixed caps.”

filesrc

src

sink

mp3 parser

src

sink

mp3 decoder

src

sink

audio sink

Negotiated Caps audio/mpeg, mpegversion=1, layer=3, channels=2, rate=44100

What flow through pads? • There are two kinds of information that flows between plugin elements through the pads  Events  Buffers

• Buffers contain the actual multimedia data that the pipeline is created to handle in the first place • Events are light weight structures that carry control information  Data start indicator event (new segment event)  End of stream event  Flush events  Seek request events

Terminology: Upstream and Downstream Downstream

Plugin 1

Plugin 2

Plugin 3

Plugin 4

Upstream

• Downstream: left to right  Example downstream flow - Data (buffer) flow - Events such as flushes, end of stream indicator, beginning of data flow indicator

• Upstream: right to left  Example upstream flow - Seek request events

States in GStreamer Elements Overview

GStreamer Element States STATE

DESCRIPTION

VOID STATE

Invalid state type

NULL STATE

First state when element is just created. Initialised from GObject perspective, but not yet initialised from GStreamer perspective.

READY STATE

Run time memory requirements have been allocated. Capabilities are not yet negotiated and the plugin is not yet aware of the kind of media that would be flow.

PAUSED STATE

Plugin is aware of the final media type that is to be streamed. Plugin is ready to process any data that flows through it. Pipeline clock is not running.

PLAYING STATE

Plugin is processing data. Pipeline clock is running.

State Transitions

Upward State Transitions

PAUSED TO PLAYING

4.PLAYING STATE

PLAYING TO PAUSED READY TO PAUSED

2. READY STATE

3. PAUSED STATE

PAUSED TO READY

NULL TO READY

1. NULL STATE

READY TO NULL

Downward State Transitions

Difference between PAUSED State and PLAYING State • For SINK elements  In paused state - No data is being rendered to peripheral. Internal rendering threads, if any, will be in paused/blocked state.

- Clock is not running

 In playing state - Data rendering to peripheral is in continuous flow - Clock is running

• For NON-SINK elements  There is absolutely no difference between paused and playing state for non-sink elements.  Non-sink elements are required to process data even in paused state.

Scheduling Modes Plugin operation choices

Push mode â&#x20AC;˘ The most generic case is where one element pushes data to the next down stream element connected on its source pad GStreamer Framework Core Data Flow Management Logic 4. Audio sinkâ&#x20AC;&#x2122;s chain function is

3. Core forwards data

invoked with buffer as argument

to sink plugin through

MP3 Decoder src

2. MP3 decoder pushes data buffer on its source pad

Audio Sink

its sink pad

sink

1. Audio Sink is designed to passively receive data on its sink pad, i.e. implement chain function

Pull Mode â&#x20AC;˘ A plugin operating in pull mode will request data from upstream plugin. Upstream plugin will not send data by itself and responds only when requested. GStreamer Framework Core Data Flow Management Logic 4. filesrcâ&#x20AC;&#x2122;s _get_range() function

3. Core forwards request

gets called

to filesrc plugin through

File Source

5. When the sequence of synch apis return, the demuxer will have data

3GP Demuxer

its src pad

src

1. filesrc plugin is designed to support requests for data to be retrieved on its source pad, i.e. implement get_range function

sink

2. 3GP demuxer asks for data of given size at given offset to be pulled on its sink pad

Scheduling mode requirements • For push mode,  The downstream plugin should support _chain() function, i.e. it should passively receive data  The upstream plugin will call the api gst_pad_push() to push data on its src pad

• For pull mode,  The upstream plugin should support _get_range() function, i.e. it should support requests for data retrieval  The downstream plugin will call the api gst_pad_pull_range() to pull data through its sink pad

Thank You