Streams! ALL THE STREAMS!

Let's start with talking about what streams are. Streams at their core are a flow of media. They can be unidirectional or bidirectional and are comprised of media formats. The media formats also contain a type. To simplify things streams only carry a single type of media. Streams can also carry an identifier in the form of a name.

Asterisk currently has no explicit interface for streams and simply has a single pipe that frames are written to and read from. The negotiated formats are scoped to the entire channel as a result. Interfaces that need to manipulate media have injected themselves into this single pipe and have to take special care to not manipulate frames they do not need to. This pipe also carries control frames and other signaling related operations. This means that there is a single very loose stream for each type of media.

So what do we need to be able to do for streams?

• We need to be able to know what streams are present
• We need to be able to know the details about the streams
• We need to be able to add, remove, and change a stream (or streams)
• We need to be able to know that a frame has come from a specific stream
• We need to be able to send a frame and have it go out a specific stream
• We need to be able to negotiate two streams
• We need to be able to exchange media between two streams

All of this needs to be present and existing functionality needs to continue to work as expected.

But what about all the media stuff that exists now?

This is the complicated part. Let's take a look at impacted functionality.

Audiohooks

As they exist now audiohooks hook themselves into every audio frame that is passing through a channel. This works since there is only ever a single 'virtual' stream.

Initial Support

Audiohooks will only hook themselves into the first audio stream on a channel.

Long Term

Audiohooks will be expanded to allow selecting the stream that it is to be placed on. If no stream is specified it will select the first.

Framehooks

Framehooks currently are invoked for every frame passing through a channel.

Initial Support

Framehooks will only hook themselves into control frames and the first stream of each type. This mirrors the current behavior.

Long Term

Framehooks will allow specifying what streams are to be hooked into. The callback will provide information about what stream a frame is in relation to.

Translation

Translation currently only reliably creates a translation path for the audio portion of a channel.

Initial Support

The code will only create a translation path for the first audio stream.

Middle Term

Each audio stream will create an appropriate translation path.

File Playback/Recording

The file API is one of the oldest things in Asterisk. Given a set of formats on the channel it will decide what file (without a container) will be played.

Initial Support

The code will use the formats on the first audio and video stream to determine what file should be used for playback. It will be played out to those streams.

Long Term

A container format will be implemented which supports multiple streams within itself. Based on the characteristics of the streams the best file will be chosen and played back. Each stream within the container will be played out. For existing files without a container the initial support logic will be used.

Bridging

Initial Support

The first stream of each type will be connected. This mirrors the existing behavior. While it does not utilize streams (yet) it ensures that existing behavior works.

Middle Term

Each stream in order of type from each channel is connected. If stream changes are required (adding/removing) then they are initiated and the result taken into account. If a translation path is required it is set up for audio.

Older Channel Drivers

The older channel drivers should have a default stream representation based on the formats on the channel. If audio and video is negotiated, then there should be an audio and video stream and it should be transparent to the channel driver. Ideally _no_ changes would be required to the channel drivers. Any requests to change, manipulate, add, or remove streams will result in an error and the operation failing.

So How Do We Get There?

By not burning down what already exists so we don't go crazy updating everything so it all works again.

Legacy

The below design attempts to strike the balance between implementing the functionality we need in streams while maintaining the full media related APIs that exist today. It does this by bringing into code what logically exists within the 'virtual' streams.The API refers to these as default streams. Default streams are the first stream present on a channel for each media type. This mirrors the behavior of the 'virtual' streams present in the original single pipe. All consumers of the existing media related APIs deal with these default streams without knowing it. The internal implementation of each core API is changed to use the default stream that is applicable.

Core Concepts

As part of the addition of streams support two new core concepts have been added: streams and stream topologies.

Streams are a representation of a flow of media and contain the details described at the beginning of this wiki page.

Stream topologies are an ordered list of associated media streams. 

stream.h

This file holds the public stream API. This covers functionality for inspection by components in the system (such as applications), stream creation, stream manipulation, topology creation, and topology manipulation.

These functions can be used by anything to examine a stream after retrieving a stream (from a channel for example) and to manipulate a stream by a stream user.

As you can see a stream has various attributes: the media type being carried on, a unique (to a channel) identifier, the formats negotiated on it, its current state, a name, and an opaque data. In the case of the name this may be constructed externally (as a result of receiving an SDP offer) or generated locally if we are sending an offer.

It is expected that the originator of a stream associate data within itself using the stream numerical identifier. This allows an array (or a vector) to be used for an easy mapping.

Stream creation is up to the originator of a stream. In the case of a channel driver the channel driver may know a new stream exists, create it, and then place it on a channel. For file playback it may do something similar but place the stream in an internal structure instead. For stream destruction it is up to the holder of stream. This can be the internal channel core, or the file playback implementation. If the channel driver does not support multiple streams the internal core API will take care of the management of default streams.

A common method of describing (and passing around) a topology of streams is also present. This is an ordered list of streams that can be added to, queried, and manipulated.

stream.c

The stream implementation will hold the actual definition of a stream and the implementation of the various functions.

The contents of a stream very much mirror that of the public and internal APIs. There is not anything truly hidden away yet.

channel.h

The channel header file will be minimally changed to add multiple stream support.

To indicate support for multiple streams a property has been added for channel technologies to enable. An inspection function exists for the purpose of getting the topology of streams on the channel. Once retrieved the normal topology functions can be used to inspect each stream individually. In the case of channel drivers they can also manipulate the stream topology provided the channel lock is held.

Stream Creation

If the multiple stream property is not set the act of setting nativeformats for a channel will automatically create (or revive if removed) streams in the background. A stream will exist for each media type of formats on the channel. If the channel has both an audio and video format an audio stream and a video stream will exist. These will be set as the default stream for each media type.

If the multiple stream property is set the act of creating and adding streams is left up to the specific channel driver. As each stream is added to the channel if a default stream for that media type does not exist then the stream will be set as the default.

Writing Frames

The ast_write function will choose a default stream based on the media type of the frame being written. If no default stream exists the frame will not be written.

The ast_write_stream function will provide the given frame to the channel driver with the given stream. It is up to the channel driver to write the frame out on that stream. All frames written using either ast_write or ast_write_stream will be provided to the channel driver using the write_stream technology callback.

Reading Frames

The ast_read function will read frames from ONLY default streams and non-media. If a frame is received on a non-default stream it will be discarded and a null frame returned. This mirrors current behavior.

The ast_read_streams function will read frames from ALL streams and non-media. It is up to the caller to then decide to do based on the stream.

Changing Stream Topology

An external entity (such as a SIP phone) can request that the stream topology change at any time. The channel driver receives the change in an implementation specific fashion and queues an AST_CONTROL_STREAM_TOPOLOGY_REQUEST_CHANGE on the channel. The current handler of the channel reads the frame and acts upon the stream topology within it. The stream topology (whether it is changed or not) is provided to the channel using the ast_channel_stream_topology_changed function. For legacy reasons the ast_read() function will not allow topology changes and the current negotiated stream topology will be returned within its implementation, signaling that the topology change did not result in an accepted change.

Components in Asterisk can request that a stream topology change by providing a new topology to the ast_channel_stream_topology_request_change function. This is provided to the channel driver and is implementation specific how it is done. The existing stream topology can be used as a base by retrieving the topology from the channel and calling the ast_stream_topology_copy function to copy it. Once the stream topology change result is known in the channel driver it queues an AST_CONTROL_STREAM_TOPOLOGY_CHANGED frame.

Both of these control frames contain the complete stream topology.

frame.h

The only change to frame related stuff is mere control frame additions.

The control frame type is used to communicate a request to change the stream topology or an indication that the stream topology has changed. When written it is a request, when read it is informational.

Examples

Creating streams on a channel

This  creates an audio and video stream and places them on the channel.

Iterating

This just loops through all the streams on a channel.

Requesting a change to the stream topology

This code requests that the first stream on the channel be set to inactive.

Handling a request to change the stream topology

This code reads in a request to change the topology and accepts the topology as requested. Note that we only receive request changes if we are capable of supporting multiple streams. If this were to use ast_read() the topology request change would be internally handled.