Gstreamer on Thibault Saunier

gst-dots-viewer: A New Tool for GStreamer Pipeline Visualization

Fri, 16 May 2025 00:00:00 +0000

We’re happy to have released gst-dots-viewer, a new development tool that makes it easier to visualize and debug GStreamer pipelines. This tool, included in GStreamer 1.26, provides a web-based interface for viewing pipeline graphs in real-time as your application runs and allows to easily request all pipelines to be dumped at any time.

What is gst-dots-viewer?

gst-dots-viewer is a server application that monitors a directory for .dot files generated by GStreamer’s pipeline visualization system and displays them in your web browser. It automatically updates the visualization whenever new .dot files are created, making it simpler to debug complex applications and understand the evolution of the pipelines at runtime.

Key Features

Real-time Updates: Watch your pipelines evolve as your application runs
Interactive Visualization:
- Click nodes to highlight pipeline elements
- Use Shift-Ctrl-scroll or w/s keys to zoom
- Drag-scroll support for easy navigation
Easily deployable in cloud based environments

How to Use It

Start the viewer server:
```
gst-dots-viewer
```
Open your browser at http://localhost:3000
Enable the dots tracer in your GStreamer application:
```
GST_TRACERS=dots your-gstreamer-application
```

The web page will automatically update whenever new pipeline are dumped, and you will be able to dump all pipelines from the web page.

New Dots Tracer

As part of this release, we’ve also introduced a new dots tracer that replaces the previous manual approach to specify where to dump pipelines. The tracer can be activated simply by setting the GST_TRACERS=dots environment variable.

Interactive Pipeline Dumps

The dots tracer integrates with the pipeline-snapshot tracer to provide real-time pipeline visualization control. Through a WebSocket connection, the web interface allows you to trigger pipeline dumps. This means you can dump pipelines exactly when you need them during debugging or development, from your browser.

Future Improvements

We plan on adding more feature and have this list of possibilities:

Additional interactive features in the web interface
Enhanced visualization options
Integration with more GStreamer tracers to provide comprehensive debugging information. For example, we could integrate the newly released memory-tracer and queue-level tracers so to plot graphs about memory usage at any time.

This could transform gst-dots-viewer into a more complete debugging and monitoring dashboard for GStreamer applications.

Demo

GStreamer: one repository to rule them all

Wed, 29 Sep 2021 00:00:00 +0000

For the last years, the GStreamer community has been analysing and discussing the idea of merging all the modules into one single repository. Since all the official modules are released in sync and the code evolves simultaneously between those repositories, having the code split was a burden and several core GStreamer developers believed that it was worth making the effort to consolidate them into a single repository. As announced a while back this is now effective and this post is about explaining the technical choices and implications of that change.

You can also check out our Monorepo FAQ for a list of questions and answers.

Technicall details of the unification

Since we moved to meson as a build system a few years ago we implemented gst-build which leverages the meson subproject feature to build all GStreamer modules as one single project. This greatly enhanced the development experience of the GStreamer framework but we considered that we could improve it even more by having all GStreamer code in a single repository that looks the same as gst-build.

This is what the new unified git repository looks like, gst-build in the main gstreamer repository, except that all the code from the GStreamer modules located in the subprojects/ directory are checked in.

This new setup now lives in the main default branch of the gstreamer repository, the master branches for all the other modules repositories are now retired and frozen, no new merge request or code change will be accepted there.

This is only the first step and we will consider reorganizing the repository in the future, but the goal is to minimize disruptions.

The technical process for merging the repositories looks like:

foreach GSTREAMER_MODULE
 git remote add GSTREAMER_MODULE.name GSTREAMER_MODULE.url
 git fetch GSTREAMER_MODULE.name
 git merge GSTREAMER_MODULE.name/master
 git mv list_all_files_from_merged_gstreamer_module() GSTREAMER_MODULE.shortname
 git commit -m "Moved all files from " + GSTREAMER_MODULE.name
endforeach

This allows us to keep the exact same history (and checksum of each commit) for all the old gstreamer modules in the new repository which guarantees that the code is still exactly the same as before.

Releases with the new setup

In the same spirit of avoiding disruption, releases will look exactly the same as before. In the new unique gstreamer repository we still have meson subprojects for each GStreamer modules and they will have their own release tarballs. In practice, this means that not much (nothing?) should change for distribution packagers and consumers of GStreamer tarballs.

What should I do with my pending MRs in old modules repositories?

Since we can not create new merge requests in your name on gitlab, we wrote a move_mrs_to_monorepo script that you can run yourself. The script is located in the gstreamer repository and you can start moving all your pending MRs by simply calling it (scripts/move_mrs_to_monorepo.py and follow the instructions).

You can also check out our Monorepo FAQ for a list of questions and answers.

Thanks to everyone in the community for providing us with all the feedback and thanks to Xavier Claessens for co-leading the effort.

We are still working on ensuring the transition as smoothly as possible and if you have any question don’t hesitate to come talk to us in #gstreamer on the oftc IRC network.

Happy GStreamer hacking!

GStreamer Editing Services OpenTimelineIO support

Mon, 22 Apr 2019 00:00:00 +0000

GStreamer Editing Services OpenTimelineIO support

OpenTimelineIO is an Open Source API and interchange format for editorial timeline information, it basically allows some form of interoperability between the different post production Video Editing tools. It is being developed by Pixar and several other studios are contributing to the project allowing it to evolve quickly.

We, at Igalia, recently landed support for the GStreamer Editing Services (GES) serialization format in OpenTimelineIO, making it possible to convert GES timelines to any format supported by the library. This is extremely useful to integrate GES into existing Post production workflow as it allows projects in any format supported by OpentTimelineIO to be used in the GStreamer Editing Services and vice versa.

On top of that we are building a GESFormatter that allows us to transparently handle any file format supported by OpenTimelineIO. In practice it will be possible to use cuts produced by other video editing tools in any project using GES, for instance Pitivi:

At Igalia we are aiming at making GStreamer ready to be used in existing Video post production pipelines and this work is one step in that direction. We are working on additional features in GES to fill the gaps toward that goal, for instance we are now implementing nested timeline support and framerate based timestamps in GES. Once we implement them, those features will enhance compatibility of Video Editing projects created from other NLE softwares through OpenTimelineIO. Stay tuned for more information!

WebKitGTK and WPE gain WebRTC support back!

Tue, 31 Jul 2018 00:00:00 +0000

WebRTC is a w3c draft protocol that “enables rich, high-quality RTP applications to be developed for the browser, mobile platforms, and IoT devices, and allow them all to communicate via a common set of protocols”. The protocol is mainly used to provide video conferencing systems from within web browsers.

A brief history

At the very beginning of the WebRTC protocol, before 2013, Google was still using WebKit in chrome and they started to implement support using LibWebRTC but when they started the blink fork the implementation stopped in WebKit.

Around 2015/2016 Ericsson and Igalia (later sponsored by Metrological) implemented WebRTC support into WebKit, but instead of using LibWebRTC from google, OpenWebRTC was used. This had the advantage of being implemented on top of the GStreamer framework which happens to be used for the Multimedia processing inside WebKitGTK and WebKitWPE. At that point in time, the standardization of the WebRTC protocol was still moving fast, mostly pushed by Google itself, and it was hard to be interoperable with the rest of the world. Despite of that, the WebKit/GTK/WPE WebRTC implementation started to be usable with website like appr.tc at the end of 2016.

Meanwhile, in late 2016, Apple decided to implement WebRTC support on top of google LibWebRTC in their ports of WebKit which led to WebRTC support in WebKit announcement in June 2017.

Later in 2017 the OpenWebRTC project lost momentum and as it was considered unmaintained, we, at Igalia, decided to use LibWebRTC for WebKitGTK and WebKitWPE too. At that point, the OpenWebRTC backend was completely removed.

GStreamer/LibWebRTC implementation

Given that Apple had implemented a LibWebRTC based backend in WebKit, and because this library is being used by the two main web browsers (Chrome and Firefox), we decided to reuse Apple’s work to implement support in our ports based on LibWebRTC at the end of 2017. A that point, the two main APIs required to allow video conferencing with WebRTC needed to be implemented:

MediaDevices.GetUserMedia and MediaStream: Allows to retrieve Audio and Video streams from the user Cameras and Microphones (potentially more than that but those are the main use cases we cared about).
RTCPeerConnection: Represents a WebRTC connection between the local computer and a remote peer.

As WeKit/GTK/WPE heavily relies on GStreamer for the multimedia processing, and given its flexibility, we made sure that our implementation of those APIs leverage the power of the framework and the existing integration of GStreamer in our WebKit ports.

Note that the whole implementation is reusing (after refactoring) big parts of the infrastructure introduced during the previously described history of WebRTC support in WebKit.

GetUserMedia/MediaStream

To implement that part of the API the following main components were developed:

RealtimeMediaSourceCenterLibWebRTC: Main entry point for our GStreamer based LibWebRTC backend.
GStreamerCaptureDeviceManager: A class to list and manage local Video/Audio devices using the GstDeviceMonitor API.
GStreamerCaptureDevice: Implementation of WebKit abstraction for capture devices, basically wrapping GstDevices.
GStreamerMediaStreamSource: A GStreamer Source element which wraps WebKit abstraction of MediaStreams to be used directly in a playbin3 pipeline (through a custom mediastream:// protocol). This implementation leverages latest GstStream APIs so it is already one foot into the future.

The main commit can be found here

RTCPeerConnection

Enabling the PeerConnection API meant bridging previously implemented APIs and the LibWebRTC backend developed by Apple:

RealtimeOutgoing/Video/Audio/SourceLibWebRTC: Passing local stream (basically from microphone or camera) to LibWebRTC to be sent to the peer.
RealtimeIncoming/Video/Audio/SourceLibWebRTC: Passing remote stream (from a remote peer) to the MediaStream object and in turn to the GStreamerMediaStreamSource element.

On top of that and to leverage GStreamer Memory management and negotiation capabilities we implemented encoders and decoder for LibWebRTC (namely GStreamerVideoEncoder and GStreamerVideoDecoders). This brings us a huge number of Hardware accelerated encoders and decoders implementations, especially on embedded devices, which is a big advantage in particular for WPE which is tuned for those platforms.

The main commit can be found here

Conclusion

While we were able to make GStreamer and LibWebRTC work well together in that implementation, using the new GstWebRTC component (that is now in upstream GStreamer) as a WebRTC backend would be cleaner. Many pieces of the current implementation could be reused and it would allow us to have a simpler infrastructure and avoid having several RTP stack in the WebKitGTK and WebKitWPE ports.

Most of the required APIs and features have been implemented, but a few are still under development (namely MediaDevices.enumerateDevices, canvas captureStream and WebAudio and MediaStream bridging) meaning that many Web applications using WebRTC already work, but some don’t yet, we are working on those!

A big thanks to my employer Igalia and Metrological for sponsoring our work on that!

gst-validate: A suite of tools to run integration tests for GStreamer

Mon, 21 Apr 2014 00:00:00 +0000

Collabora ltd. has been developing a tool that allows GStreamer developers to check that the GstElements they write behave the way they are supposed to: GstValidate. The tool was started first to provide plug-ins developers with a tool to check that they use the framework the proper way. Since the beginning, it has been available in gst-devtools, a gst module where we collect a set of tools to facilitate GStreamer development and usage.

Well, what is it about?

The GstValidateMonitor

Basically gst-validate allows us to monitor everything that is happening inside a GstPipeline. For example if you want to check that every component of a pipeline is properly behaving, you can create a GstValidatePipelineMonitor that will track that pipeline. Then each time a GstElement is added to the pipeline, a GstValidateElementMonitor will be instantiated and start tracking that element. Then when a GstPad is added to that GstElement a GstValidatePadMonitor will be monitoring that pad.

This monitoring logic allows us to check that what those elements do respect some rules GStreamer components have to follow so all the elements of a pipeline can properly interact together. For example, a GstValidatePadMonitor will make sure that if we receive a GstSegment from upstream, an equivalent segment is sent downstream before any buffer gets out. You can find the whole list of tests currently implemented here.

The GstValidateRunner

Then there is an issue reporting system so that each issue found during the execution of the pipeline is reported with as much details as possible so that users understand what the detected misbehaviour is about and can fix it efficiently.

In term of code the only thing to do in order to get a pipeline monitored is:

int
main (int argc, gchar ** argv)
{
 GstElement *pipeline;
 GstValidateRunner *runner;
 GstValidateMonitor *monitor;

 gboolean ret = 0;

 /* Initialize GStreamer and GstValidate */
 gst_init (&argc, &argv);
 gst_validate_init ();

 /* Create the pipeline and make sure it is
 * monitored */
 pipeline = gst_pipeline_new (
 "monitored-pipeline");
 runner = gst_validate_runner_new ();
 monitor = gst_validate_monitor_factory_create (
 GST_OBJECT (pipeline),
 runner,
 NULL);

 /* HERE you can do anything you want with that
 * monitored pipeline */

 /* Now print the errors on stdout.
 * The return value of that function
 * is != 0 if if critical errors occured
 * during the execution of the pipeline */
 ret = gst_validate_runner_printf (runner);

 /* Cleanup */
 gst_object_unref (pipeline);
 gst_object_unref (monitor);
 gst_object_unref (runner);

 return ret;
}

The result of gst_validate_runner_printf will look something like:

issue : buffer is out of the segment range Detected on theoradec0.srcpad at 0:00:00.096556426
 
Details : buffer is out of segment and shouldn't be pushed. Timestamp: 0:00:25.000 - duration: 0:00:00.040 Range: 0:00:00.000 - 0:00:04.520
Description : buffer being pushed is out of the current segment's start-stop range. Meaning it is going to be discarded downstream without any use

Here we can see that an issue occurred on the src pad of theoradec as it outputted a buffer that was not inside the segment it last pushed. This is an interesting piece of information and clearly shows an error in the element. (Note: This issue does not actually exist)

How should it be used?

GstValidate command line tools

In order to make gst-validate usage simple, we created command line tools that allow plugin developers test there elements in many use cases from a high level perspective.

The gst-validate pipeline launcher

This is a command line pipeline launcher similar to gst-launch. That tool uses the gst-launch pipeline description syntax and make sure that the pipeline is monitored and that the users will have all the reported information from the GstValidate infrastructure. As you can expect, you can monitor the playback of a media file using playbin as follow:

gst-validate-1.0 playbin uri=file:///.../file.ogg

You will then be able to see all the issues GstValidate found.

The gst-validate-transcoding tool

A command line tool allowing to test media files transcoding with a straight forward syntax. You can for example transcode any media file to vorbis vp8 in webm doing:

gst-validate-transcoding-1.0 
 file:///./file.ogg 
 file:///.../transcoded.webm 
 -o 'video/webm:video/x-vp8:audio/x-vorbis'

It will report what issues happened during the execution of that pipeline.

The gst-validate-media-check tool

A command line tool checking that media files discovering works properly with gst-discoverer. Basically it needs a reference text file containing valid information about a media file (which can be generated with the same tool) and then it will be able to check that those informations correspond to what is reported by gst-discoverer over new runs. For example, given that we have a valid reference.media_info file, we can run:

gst-validate-media-check-1.0 
 file:///./file.ogv 
 --expected-results reference.media_info

That will then output found errors if any and return an exist code different from 0 if an error was found.

GstValidateScenarios

As you can notice, those tools let us test static pipelines execution and not that the pipeline reacts properly during execution of actions from the end user such as seeking, or changing the pipeline state, etc… In order to make that possible and easy to use we introduced the concept of scenarios.

A scenario is a set of actions to be executed on the monitored pipeline serialized in a text file. An action (GstValidateAction) is just a function call executed at a precise time (usually based on the playback position of the pipeline).

An example of scenario:

# Just some metadatas describing the scenario
# The format is the GstStructure serialization 
# synthax
description, seek=true, duration=3.0

# When the pipeline reaches 1.0 second of
# playback it launches an accurate flushing
# seek to 10.0 seconds
seek, playback_time=1.0, start=10.0, 
flags=accurate+flush

# Send EOS to the pipeline
# so it stops and the application
# knows about it.
eos, playback_time=12.0

You can find more examples of scenarios here

gst-validate-launcher

This looks all right but shouldn’t those tests be executed automatically on a large number of samples and with the various existing scenarios? This is where gst-validate-launcher comes into play. It is basically a python application that launches the tools described above with the proper parameters. It then monitors them, check their results and serializes those results into a junit xml file (more formatter could obviously be implemented). That tools is pretty simple and it is only a matter of setting the media samples to run the tests on and set what scenarios you want to run.

Where is it used?

GStreamer Editing Services

As part of the GStreamer Editing Services project, I have made sure that ges-launch (a command line tool that launches timelines, video editing projects etc…) works with GstValidate, if compiled against it. This means that we can launch scenarios and test GES sharing the same infrastructure as the rest of GStreamer. It is also very interesting to be able to monitor dynamic pipeline (within GES the GstPipeline changes very dynamically) to discover elements misbehavior in that stressful use case. For the time being we do not have many GES specific GstValidateActions implemented, but we will implement more as needed (mostly implementing timeline edition actions, ie. moving clips around the timeline, changing effect properties, etc…) As part of the Pitvi fundraiser, we are also investigating how to serialize users action in Pitivi so that we could easily reproduce users issues outside of the application (and thus outside the python interpreter).

The GStreamer jenkins integration server

On the GStreamer continuous integration server, we are running gst-validate-launcher on a set of media sample in major formats after (almost) each commit on any of the component of the GStreamer stack. This is just the beginning and we are slowly adding more tests there making sure they pass and tracking regressions.

A lot of work has been done around that tool. We still need to clean up some part, review the few APIs we have, and a particular effort as to be made around the documentation. But now that good basis are there, we should just keep adding more tests to detect regressions in GStreamer as soon as possible. If you are interested in using that tool please come talk to us in #gstreamer on freenode!

Pitivi master finaly gets effects!

Wed, 22 Sep 2010 00:00:00 +0000

So as suggested by the title, the work I did during the Google Summer Of Code has been Merged! It means that the next Pitivi release which should arrive quite soon will gain effects.

I haven’t been blogging for a long while so sorry about not telling you what was happening but I did it in order to make a surprise :P

Now that it is merged, you are very welcome to test it and report all the bugs you will find so we have a great release soon to propose the world…. with effects!

Sorry for not making a screencast but as you can test it quite easily, it shouldn’t matter so much ;) (and nekohayo is telling me he should make one soon)

As I enjoyed a LOT working in the pitivi community during this summer, I am going to keep working on it in my free time fixing the bugs you are going to report. And of course I am planing to enhance effects support and add features such as keyframes handling…

And just a little screenshot so you can have an idea about how the UI looks like now:

Pitivi get effects, part 6

Sun, 08 Aug 2010 00:00:00 +0000

Sorry for not blogging in such a long time!

Now that I am back in chile afer the GUADEC, I have finally been able to hack again on Pitivi. So has I said during my presentation of my project during the GUADEC, I implemented everything I wanted to. So I am now working on debugging everything, cleaning code, and even if I have a few remaining critical bugs (which could probably be bugs in gnonlin), you can use it for testing purposes.

I also started to work on a new branch[0] in order to save the undocked windows state, but since I found new bugs on the effect implementations, I am working on them.

So for now I don’t have a screencast, but I promise I will make one when I can show everything working smoothly.

You can give it a try if you have the latest gstreamer from git using my git repo[1]

[0]: http://github.com/thiblahute/Pitivi/commits/save_undock_windows_state
[1]: http://github.com/thiblahute/Pitivi

Pitivi get effects, part 5

Mon, 12 Jul 2010 00:00:00 +0000

I didn’t blog for almost 3 weeks but it doesn’t mean I didn’t progress with this project. I thing it actually means more that I prefer to spend my time on coding than on blogging!

Last week was pretty full because of the end of the semester but tomorrow I have my last examen for university. Next week I am planning to go skiing so it may not be the week I will be hacking the most, but anyway, I will code!

The UI is almost the same as the mockup I showed you last time apart from the key-frame thing since I am not sure I want to do it this way, and because I want to have effects stable before implementing this feature.

So for now the UI looks exactly like:

What is working:

Adding effects
Configuring effects, the UI is generated automatically depending on the effect property so all effects are configurable. But there is also a system which will permit to create a special UI for more complicated effects.
Enabling/disabling effects even if a few bugs still remain concerning it
Removing effects

What doesn’t work:

Seeking, right now when I add an effect and try to seek, it always go to the first frame of the video. It is a pretty critical bug on which I am actively working, and that’s the reason why I didn’t do a screen-cast (sorry about that). When this is solved, I will make a screen-cast and post it here for sure.
Drag’n drop directly to the effects list. This is almost nothing but I want to fix the first issue before implementing it.
Keyframe based effect configuration
Saving/opening projects containing effects. I have already started this work but it is still not working. It should be done quite easily.

So I don’t miss so many things to complete the plan I had made for this Google Summer of Code, but the idea is to get something almost bug free and merge and release it (something I have been told should be possible quite soon… I hope :))

I also wanted to talk about a tool I use for refactoring python code called ‘Bicycle repair man’. It permits to rename variable/classes very easily and this through all python files of the project, cut functions into several ones and make the proper calls, and probably much more but those are the two features I actually use and abuse. I use it inside vim and noticed a real gain of productivity and style.

The code is still available in my GIT repository here: http://github.com/thiblahute/Pitivi

Hoping I can show you a screencast of all that and tell you that I need tester soon.

Pitivi get effects, part 4

Tue, 22 Jun 2010 00:00:00 +0000

I didn’t blog for almost 2 weeks now because I wanted to have something interesting to say before saying it!

So the last 2 weeks I started an experimental branch to handle effects in the timeline and of course corrected all the bugs I found during this time. Then I rethought this design with nekohayo and got this pretty cool mockup from him:

So as you can expect, I am now implementing that. I already rewrite the effect list and it looks the same as you can see in the mockup. And I am working on making effects being per clip or per track instead of being considered as clips themselves. This should actually be pushed by the end of the week.

So afterward I am going to work on this new middle section for clips properties and effect configurations. I think I will get it working in the next two weeks. I am quite in advanced on the planning I proposed for the postulation but this is a good thing because I am still at university and may have to spend more time for it in the coming weeks.

You still can grab my latest code here and play with it!

Pitivi gets effects, Part 3

Fri, 04 Jun 2010 00:00:00 +0000

This week I have been debugging everything and I final (Thanks to my mentor Edward Hervey) got effects working!

So better show you how it works for now. It is still quite buggy, and not really ergonomic, but it is the beginning!

Here is the video (of course edited in Pitivi and using effects). The original host (people.collabora.co.uk) is gone, so the video is no longer available:

You need a modern browser to see this video (chromium, firefox, epiphnay, midori…)

You can grab the code here if you want to give it a try even if it’s not ready for production…
So next week I am going to clean up the code and of course fix the bugs I find. If I have time, I will start working on the UI part.

[Edit] I tried the only website I found that permits to upload ogv files in order to use it with the video tag, but it seems no to work well. So you can find the video here and in better quality here

Pitivi is getting effects, Part 2

Thu, 27 May 2010 00:00:00 +0000

Here is my second report about the implementation of effects in Pitivi. I am going quite well with this project, and I now have almost everything ready to be able to add effects to videos. The dragging of effects to the timeline is almost ready, but I need to work on it a bit more to be able to show you a screen-cast! I hope it can be done next week.

Sorry for this kind of useless post, but I just wanted to tell you I make good progress, and I should be able to show you more soon!

You still can have a look at my git repository

Pitivi is getting effects, Part 1

Fri, 14 May 2010 00:00:00 +0000

Here is the first report about my Google summer of code project. I have been working quite a lot on it for the last 2 weeks! So here is the summary of what have been done so far:

Code reading, analyze of Pitivi architecture (Thanks to the pitivi wiki)
Some reading about Gstreamer to understand better the way it works
Creation of my git branch to start working on effects implementation
Implementation of the core Pitivi modifications to be able to create effects (With unit tests)
See how I can implement effect files as described in my Gsoc application, and discussions about it in the Pitivi community. We actually didn’t take a fixed decision about this for now.
Creation of the UI to show the list of effects detected thanks to the gstreamer registry (the list was already retrieved in Pitivi)
Some talking to know how this UI should be done. I will need to talk more about that to the Pitivi community to take a good decision about how it should look like. For now, it looks like:

I am now working on drag’n droping effects to the timeline.

That is all for now, and I will give you frequent news about how things are going.