ffmpeg backend?

[Matthew Macdonald-Wallace]

Hi all,

I'm new to Pyglet and I need the ability to overlay text on top of video.  I've tried doing this with libavbin, however it appears that the avbin bindings are unmaintained (see previous posts by others to this group).

I'm happy to explore what's involved in creating bindings for FFMpeg instead, as this seems to be included in pretty much all modern linux distros, however I've never written bindings before for anything, so if anyone is willing to help me do this that would be great.

Work commitments mean that I probably won't be able to make a real start on this until mid-February, but if there's benefit in it for the community please let me know and send me links that might help me with background reading before I start on this!

Cheers,

Matt

[Benjamin Moran]

Hi Matthew,

I was the guy talking to you on Reddit earlier. This might be a pretty gnarly thing to get into, but it's certainly a learning excercise!
The first step would be to familarize yourself with the ctypes library: https://docs.python.org/3.5/library/ctypes.html
I also found this video to be useful as an introduction:
https://www.youtube.com/watch?v=9CPOjOQNr7Y

Avbin was pretty nice in that it offered a simple interface to work with, but working with ffmpeg directly doesn't look to be terribly difficult.
I was looking over this guide to writing a simple video player in SDL with the ffmpeg libs, and it looks pretty informative: http://dranger.com/ffmpeg/tutorial01.html

[Matthew Macdonald-Wallace]

Brilliant, thanks.

I'll check them out.

I've done a lot of Python dev in the past, but never anything UI/Media related as it's all been serverside (OpenStack Cloud mainly!)

I'll come back to the group once I've had a chance to get started :)

Matt

[Matthew Macdonald-Wallace]

Hey,

So I've been reading around and it looks like there are already a couple of options that we might be able to use:

http://mikeboers.github.io/PyAV/index.html - basic functionality, need to play more but at least supports newer (3.x and above) versions of FFMPEG

https://avpy.readthedocs.io/en/stable/ - seems to contain everything we need, but doesn't support FFMPEG > 2.8 yet (see https://github.com/sydhds/Avpy/issues/1 for more on that!)

I'll keep digging, but in the meantime these give me some hope! :)

Matt

On Friday, 20 January 2017 14:39:08 UTC, Benjamin Moran wrote:

[Matthew Macdonald-Wallace]

In fact, it gets better!

https://bitbucket.org/proffalken/pyglet/commits/b051687f54654d2a3b4aaff3e93363a55a14a1b2?at=feature/additional_media_loader_support uses PyAV and loads the video, now I just need to work out how to get it to play!

The following code will run without errors when using the above commit installed into a virtualenv:

===================================

import pyglet

pyglet.options['debug_media'] = True

window = pyglet.window.Window(width=800, height=600)

label = pyglet.text.Label('Hello, world',

                          font_name='Times New Roman',

                          font_size=36,

                          x=window.width//2, y=window.height//2,

                          anchor_x='center', anchor_y='center')

video = pyglet.media.load('../SampleVideo_1280x720_20mb.mp4')

@window.event

def on_draw():

    window.clear()

    label.draw()

pyglet.app.run()

====================

[Benjamin Moran]

Hi Matthew,

Using those libraries directly in pyglet might be difficult due to licensing differences, but it's an interesting first step to work out using them as external modules. Going forward, it will be better for pyglet to have its own direct wrapper.

I would recommend looking over the pyglet.media.sources.Source and StreamingSource base classes, and looking at the avbin module for an example of how it can be subclassed. There are basically only a few methods needed for seeking, getting audio, and getting video packets. The hard work, of course, is working out how to fetch this data behind the scenes.

[Paul Everitt]

On a barely-related note...I pitch in with the Arcade project (2d games for Python) and keeping avbin packaging working has not been fun.

I personally would throw in some money for a bounty, either to keep pyglet+avbin viable, or switch to something more maintained. Arcade currently only wants avbin for audio.

[Benjamin Moran]

Hey Paul,

I know that library. I've maybe chatted with you (or the other contributors) on Reddit in the past. It's a nice project.

The avbin issue has been the elephant in the room for a while now. No one is currently maintaining it, and some linux distributions have dropped it from their repos. Today, there is a large number of avbin related bugs on the issue tracker, so it's definitely causing trouble for users. To me, just writing bindings for ffmpeg seems like the most logical solution.

Speaking of just audio, I've been working on a pure-python Ogg Vorbis decoder. I've made a lot of progress so far, but this is one of those projects that might never be completed. It's pretty mathimatically intensive stuff. I think that going forward, having libraries like this in the ecosystem is the best way. It might be slow today, but the portability makes it valuable. For pyglet, it would probably be fine if your're doing StaticSources.

[Paul Everitt]

Thanks for writing back. I don’t know enough about ffmpeg and packaging…can it be built into a binary wheel for systems that don’t have it?

Yes, I think a pure Python Ogg Vorbis might be enough for Arcade. You’re right that Arcade’s audio needs are pretty basic. OTOH, it’s a trillion hours of work for you over the next year, not just completing but the (like avbin) pile of annoying bugs. if I did a bounty, I’d have to get some more folks putting money in. Still, that’s the most attractive option.

—Paul

--
You received this message because you are subscribed to a topic in the Google Groups "pyglet-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyglet-users/u1OTO2aYFLo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyglet-users...@googlegroups.com.
To post to this group, send email to pyglet...@googlegroups.com.
Visit this group at https://groups.google.com/group/pyglet-users.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Moran]

I don't know enough about packaging to answer your question. I've only dealt with bundling standalone packages myself, in which case the so/dlls were just bundled with it.

As far as bugs and maintenace for something like an Ogg Vorbis decoder, I actually think it would be quite low maintenance. The reason being is that the spec is well defined, and isn't changing. Once all decoding is fully implemented, you won't really have to modify it, since there aren't any new versions coming out or features being added to the Vorbis specification. 

[Daniel Gillet]

Hello Matthew,

This is an interesting topic. I've been reading about it and here is what I figured out.

Current situation:

- Pyglet binds into AVbin which is a thin C wrapper around Libav. This guarantees a stable ABI (Application Binary Interface).

- Pyglet uses AVbin in order to implement class AVbinSource(StreamingSource) found in pyglet/media/sources/avbin.py

- Libav is a fork from FFmpeg (see here ). 

- It seems that today FFmpeg is better maintained than Libav.

So in order to use FFmpeg over Libav, the path of least resistance would be to update AVbin in order to link to FFmpeg instead of Libav. The function names seem pretty identical. Maybe some changes might be required due to the new features of FFmpeg, but in general the code could be left pretty untouched. This has also the advantage that you would not have to change the pyglet code for AVbin.

I'm no expert in this area. But this what I came up with so far. Let me know what you think.

Dan

[Benjamin Moran]

I think you have all that right. If someone were to take over maintenance of avbin, that would certainly be the simplest thing.

The reason I would consider wrapping ffmpeg directly, despite the ABI difficulty, is only because avbin seems to be dropped or unavailable from many Linux distributions. If a maintainer stepped up, that could change.

[Daniel Gillet]

Hello Benjamin,

I see a couple of options here. Again trying to find the simplest solution first.

If we drop AVbin and try to wrap FFmpeg directly through ctypes, we could then start by re-creating the functions from AVbin, but written in pure Python. This way, the rest of the code from avbin.py should still work.

Another option would be to rewrite AVbin using Cython. But I'm not sure this will give a real advantage as the C code already exists.

I also noticed this bit in AVbin build.sh:

# Either "libav" or "ffmpeg"

BACKEND=libav

So it seems it is even possible to switch to FFmpeg when compiling AVbin. Also in avbin.h, the struct AVbinInfo has a member called backend:

    /**

     * Which backend we are using: "libav" or "ffmpeg"

     */

    char *backend;

I'm on Windows, so I cannot build easily AVbin. But it would be interesting to see if we can build using FFmpeg. 

[Benjamin Moran]

I'm running Linux, so I gave this a quick try. Unfortunately it looks like the build script is missing the ffmpeg bits, so it fails in it's current state. For a quick test, I checked out the old AVbin 8.1 tag, and cloned the ffmpeg revision it was looking for. It does try building, but fails halfway through (which makes sense since these are all pretty old revisions at this point). I didn't spend any time troubleshooting, but it seems possible to support ffmpeg again (providing that the actual ffmpeg specific code still exists inside AVbin).

I like the idea of rewriting the AVbin functionality in pure python. All other components of pyglet wrap the so/dlls directly, and gives us the ability to run without any compilation just by dropping the pyglet folder into a project. I think this is a major advantage, so we shouldn't break this.

[Daniel Gillet]

Hello,

Thank you for trying it out. Too bad it's not that easy. :)

I'm currently trying to port AVbin in a full Python version, wrapping FFmpeg with ctypes. I already have 700+ lines of code. I'm more aiming at a proof of concept first. I haven't coded enough to have anything substantial to be seen. But I can already read the type of media, how many streams it has, and what kind of decoder is needed to play the stream.

I hope by the end of the week I will have something working on my machine.

At the moment, it's a bit of a mess. Wrapping C code is very error-prone.

I'll keep you posted.

Dan

[Benjamin Moran]

That sounds fantastic!  If this eventually becomes usable, it will make a lot of people happy. A large number of bugs on the issue tracker are AVbin related, but noone has stepped up to tackle it so far.  I'm more than happy to assist with testing and things on Linux, once you're ready.

[Daniel Gillet]

Hello,

Just a little update on my endeavour in binding FFmpeg with ctypes.

After hours of work, I finally managed to have a working video and sound! But there are still tons of problems I need to fix. For instance, one mp4 video just freezes. I need to find out what's the problem.

The good news is that I managed for the audio to play also planar audio format, which was not yet supported by AVbin (see here). I had no idea what it was, but I found out that this format is used more and more with the more recent videos / audios. Or at least, this is my understanding.

My main problem now is that the video does not stop at the end. I'm still investigating. If I play only a sound, it stops correctly. But as soon as there's a video, the Video Thread wants more video packet and the whole program freezes.

I'm not ready yet to give out any code. It still needs a lot of clean-up. Besides this only works on Windows, as I have currently hard-coded the *.dll names. This will need to be changed obviously.

I thought you might like to hear that something is going in the right direction. :)

[Benjamin Moran]

I haven't heard of planar audio format either until now, but better compatibility with modern files is great. I'm happy to hear that you've been able to make progress. Hopefully this pans out, and we can get pyglet into a bit better shape in the audio/video department.

I'm not very familiar with this part of the codebase, but I'll try to help if I can once you're ready to share. Pyglet has a lib loader function built in, so it should be pretty easy to un-hardcode everything towards the end.

Once things stabilize a bit in the future, we can think about making a new "ffmpeg" branch to get more testing.

[Daniel Gillet]

Here is a quick update on my progress.

I can now play audio and video using most formats. I tried mp4, avi, ogg, mp3, and a few other formats. What is NOT working at the moment is to correctly detect the end of the video. I will look into it, but this was not my priority yet.

The more I progress, the more I realize the code found in pyglet/media/sources/avbin.py might also need some work. I'm learning as I progress. So I'm sure there are tons of things I'm not doing properly.

For the sake of giving people a go at it, I'll push on my repo on a new branch. This code is still changing A LOT. I appreciate any feedback.

How to use it?

- Clone the repo

- Go in pyglet/examples/video_ffmpeg

- Place in this folder the *.dll/*.so from FFmpeg

- Place also there your media file

- run python media_player.py my_media.mp4

It will probably crash. :)

[Benjamin Moran]

Looks great. I'll give this a try today and let you know how it goes.

[Benjamin Moran]

Hey Dan,

I gave this a try on my Arch Linux box, and it runs fine. I had to do a small tweak to get it to run, which was to remove the specifc library versions from the lib.load_library() statements in av.py (change "avformat-57" to "avformat", etc). After that, the loader function was able to properly pick up the system ffmpeg libraries. Fantastic work. I've looked over your files, and it's quite a lot of work that you've done with the structures and function definitions.

I ran a few different formats, such as mpeg2/ts, webm, ogg/vorbis, and h264/mp4. All worked OK, with very minor issues that were likely causes by my poor graphics card on this test box. (Some minor stuttering with the mpeg2/ts, but It's a semi-non-standard format that local DTV is broadcast in here).

There's nothing pretty about wrapping libraries with ctypes, but you've written some clean code here.  Let me know if there is anything you need help on, once you start to stabilized things

-Ben

[Daniel Gillet]

Hello,

This is all excellent news! Thanks for taking the time to give it a try.

Le mardi 14 mars 2017 03:23:11 UTC+1, Benjamin Moran a écrit :

I had to do a small tweak to get it to run, which was to remove the specifc library versions from the lib.load_library() statements in av.py (change "avformat-57" to "avformat", etc). After that, the loader function was able to properly pick up the system ffmpeg libraries. 

This is indeed something I (we ?) need to think about. When I downloaded the binaries from FFmpeg website, they had dll files with names containing the lib version at the end. As we anyway only provide a wrapper to these lib, but it's up to the user to provide the said libraries, should we maybe provide a way in the API to give the appropriate file names ? Or should we add in the documentation that pyglet expects the libs to have any version number stripped off from the file name?

 

All worked OK, with very minor issues that were likely causes by my poor graphics card on this test box. (Some minor stuttering with the mpeg2/ts, but It's a semi-non-standard format that local DTV is broadcast in here).

Maybe your graphic card is not the problem. I can see a few possible problems. Currently the code syncs the video to the audio. Depending how the packets are coming from the stream, it could be that some frames are skipped because the time moved passed the supposed presentation timestamp. Also the code only decodes one frame at a time. Maybe with this format this takes a bit too much time. Begin able to decode multiple frame ahead of time and buffering them would probably help. But this requires some drastic changes in avbin.py code. 

There's nothing pretty about wrapping libraries with ctypes, but you've written some clean code here.  Let me know if there is anything you need help on, once you start to stabilized things

 Thank you very much for these kind words. It's always good for the motivation! I was wondering if I should split the av.py file into multiple smaller files. Maybe one per lib I'm wrapping, and then one for the AVbin functions I re-created? Or should I just leave it like this?

My current goal is to correctly detect the end of the media. Then I'll look again at the seeking functions, because I'm not sure it's correct.

One thing you might do is run a video and see how memory is doing. I'm pretty sure I must have some leaks here and there. Some testing could help.

Cheers,

Dan

[Daniel Gillet]

By the way, as I need to venture in the pyglet/media/drivers code in order to understand why the end of the audio is not detected correctly when the video is playing, if Rob van der Most who did some great work in that area could help me understand what's going on, this could prove useful.

Dan

[Daniel Gillet]

Good news! I could finally find the problem with detecting the end of the video. I'm now fixing the seeking feature which crashes once in a while. I'll also start to look at the memory usage and try to find if there are any leaks.

Dan

[Benjamin Moran]

That's great, I'm glad you figured it out. Was it something to do with the drivers, or something else?

That DTV file I mentioned before that stuttered a bit, It's most likely because it's an interlaced file. It almost seems as if it's only rendering every other frame. Honestly, I think supporting interlaced files is a very low priority, and wouldn't spend too much time on it at the moment.

Regarding the so/dll names, on Linux we can safely assume the name is just libxxxx.so. The version number gets tagged on the end for installed libraries, such as "libavutil.so.55", but there is always a symlink to "libavutil.so" available. If a developer is bundling these libs with their application, they can be expected to ensure the lib name just ends in .so.  On Windows, I'm not sure how the libraries appear in the path. If you have it installed system wide, can ctypes.util.find_library locate the DLL just by it's name (without version tag)? If so, I think it's safe to ask users to rename the dll if they are bundling it. For the time being, I wouldn't worry about bundled libs at all. The pyglet.lib.load_library code already has a bunch of work towards this, so it shouldn't be anything that goes into the bindings themselves.

It might make sense to split it up into multiple files, and create an ffmpeg package. Something similar to how the pyglet/media/drivers packages are organized. For those, the bindings are in seperate lib_x files, with the python code by itself in it's own files. This also makes sense if we explore automatic binding generation, using some scripts like we do for wrapping the OpenGL and other system libraries. This makes it easier to automatically replace the binding fiels without touching the other code.

-Ben

[Daniel Gillet]

No the drivers were not the issue. The problem was in the Player. Without going into too much details, let's say that the Player instantiates an audio driver which will spawn a thread. That thread starts to get audio data, decode it and place it in a buffer for the driver to play it. The Player also schedules a function to get the next video frame, normally at a pace of 30 fps.

When reading packet from the media stream, audio and video can come in any order. So when the audio thread requests more audio data, it reads new packet, and if they are video data, it places them on a queue for the update_texture function to process. It does so until it finds some audio data,

The update_texture function also reads packets when called until video data is found. If it gets some audio data, it places it on queue for the audio thread to process.

The audio is the master clock when it comes to synchronization. The video asks the audio what time it is to know which frame it should display. So it's the audio who is responsible to tell everyone that the end has come.

The problem was that the audio thread put an event on the application loop, but in the meantime the update_texture was called, and it was polling infinitely the media to find some video data, which will never come.

I made some fixes, but as I already said, I think I should re-design the way the Player gets its video and audio data. But I still need to think about it. I will of course keep the API the same. I might add some methods if need be.

Regarding the dll files on windows, the names end always with .dll. I have for instance 

avcodec-57.dll

avdevice-57.dll

avfilter-6.dll

avformat-57.dll

avutil-55.dll

swresample-2.dll

swscale-4.dll

They have to be placed either in the application folder to be found or its PATH should be added to the PATH environment variable.

pyglet.lib.load_library allows to try a couple of different names. But I don't see how to use something like a glob. At the end, it call ctypes.util.find_library() to try to find the required files. Here is a note for Windows:

On Windows, find_library() searches along the system search path, and returns the full pathname, but since there is no predefined naming scheme a call like find_library("c") will fail and return None.

I won't worry yet about this particularity, but we'll have to keep it in mind.

Good point about structuring the code in multiple files in a fmpeg package, which would be placed under pyglet/media/sources I guess.

So there is still a lot to do. But I'm happy to see where it's heading.

Dan

[Daniel Gillet]

Hello,

I've pushed some new changes on my branch. I think I could fix the threading issues I encountered before. If someone is willing to give the Media Player a try, I would be grateful. As a reminder, it's located at pyglet/examples/video_ffmpeg/media_player.py. 

You should now be able to play, pause and seek the video. When the video finishes, you cannot play again the video as the default behaviour for the Media Player was to get the next source which is inexistant. This obviously could be changed.

I will now start to slowly tackle the smaller issues, like the creation of a ffmpeg package, etc.

Cheers,

Dan

[Benjamin Moran]

Thanks for the explaination above on how things are organized, with regards to fetching packets and frame timing.

I just gave this a try on my box, and it seems like everything works as expected. I think that the video not playing again once it's finished is OK for now. It seems like something that would be better handled in the application, rather than changing how the Player class works. Speaking of which, the media_player example is a little crusty at this point. It might be nice to monernize that a bit using a batch and vertex lists for the controls. It's functionally fine, but as an example it should show off the modern way to do things. Maybe I'll do that when I have a moment (now that it's actually useful again!).

I did notice that the video playback is a bit choppy, but I'll have to test it again on my home box. CPU load wasn't too bad, so If it still seems strange I'll run it through a profiler.

[Daniel Gillet]

Hello,

Thank you Benjamin for taking the time to test this. It would be great indeed to update the media_player code. If you can take care of the graphic part, that would be awesome.

I made a new commit. I'm afraid I put 2 changes in that commit. As I'm not super familiar with mercurial, I didn't know how to commit only the changes I want, in order to split the changes in 2 commits. With Git, there is git add --interactive. Anyway, I didn't want to spend too much time on this, so I just committed those changes in a single commit.

One change is about determining the average frame rate of the video and using that number to refresh the texture at an appropriate time. I hope this would help to reduce the "choppiness" you describe. I must say I don't really notice the same here. But it probably depends also on the video you're playing.

The second change is about re-organizing the code in a sub-package. I'm not entirely happy with it. You'll notice under ffmpeg/__init__.py that I commented out some imports. If I uncomment them, there is a problem with some imports. I don't fully understand why. If you could have a look at it, that would help too.

Dan

[Benjamin Moran]

Hi Dan,

I still get confused with mercurial sometimes myself, so you're definitely not alone there.

I finally found a few minutes to give this a try. Unfortunately it looks like the latest version doesn't play any audio at all for me. The video also seems to stutter on every video now, which maybe makes sense since something is up with the audio.

I also ran a few profiles using vmprof (awesome tool, if you've not tried it). You can see some of my results here with two different videos, a 1080p mkv and 720p mp4, both x264 codec:
(python -m vmprof --web media_player <video>)
http://vmprof.com/#/88671822-4aed-4d22-8093-4ea844b79fb0
http://vmprof.com/#/19aa7b4f-a178-4cff-ab70-ce9942611ac9
As far as I can tell it looks normal. The time seems to be spent pretty much where you would expect, since it's mainly just moving data around and updating textures.

Nice work on the package. I'll have a look at it and see if I can figure out the import issue.

[Daniel Gillet]

Hi Benjamin,

This is unfortunate you don't get the audio with the last changes. Indeed without the audio, the video gets crazy. This is because the SilentAudioPlayerPacketConsumer doesn't really behave as it should. This is probably something else to fix.

I've made actually some more changes, which are more related to how the ffmpeg package gets imported. So I could fix my problem regarding the imports. This is commit 748ae51. I don't know if you tried from that commit. If not, would you give it a try?

If this commit fails too, in the media_player, you could maybe turn on the debugging info with this line pyglet.options['debug_media'] = True. If you can then copy here the beginning, so I can figure out why the audio is not detected correctly.

I'm at the moment busy rewriting how the player works for synchronizing audio with video. These are pretty big changes. So I'm doing that on the side. If it turns out to be successful, I'll push it and let you try it too.

Thanks for the profiling info. Very interesting! I think the stutter comes from how the Player synchronizes the video. In the file pygelt/media/player.py, you will find in the Player class the method update_texture. You will find in the code, those lines:

        while ts is not None and ts < time:

            self._groups[0].get_next_video_frame() # Discard frame

            ts = self._groups[0].get_next_video_timestamp()

If you would add a print in this loop, you will notice that frames get discarded from time to time. This is because the method is called regularly, but not exactly when the next frame should be shown. This is probably why you see this stutter. What I'm working on tries to change that, by calling the update_texture when it's required to show the next frame.

Let me know how it goes with the audio!

Dan

[Benjamin Moran]

Hi Dan,

I tried another video just now, and was able to hear audio. The working one was AAC, while the non-working one was DTS. I also tested for dropped frames, and they seem to only happen every once in a while, so that probably doesn't seem to be the cause. It's strange, but I'll just chalk it up to an incompatible file for now.
I was just thinking, maybe we can find a few freely available sample videos online that we can use for testing, just so that everyone gets the same results. Something like those Blender made ones like "Big Buck Bunny"?

Also, about the library names, we can probably check "pyglet.compat_platform" to see which library names to load. Linux is just the names without the "-xx" version tag on the end. I'm not sure about OSX, but don't worry about this for now. I just strip the names off manually when I test it.

-Ben

[Daniel Gillet]

Hi Benjamin,

I would be interested in trying that video that causes trouble on your side.

It's an excellent idea to try a free video in common. I didn't have much time recently. Plus I just pushed a fix for pyglet font. :)

From my tests, I had a *.ts file that shows also some stutter. It seems that the video decoding takes a lot of time, so a frame is skipped quite regularly, causing this unpleasant visual effet. I will need to properly time this, to make sure I'm not going in the wrong direction.

Dan

[Daniel Gillet]

Hello,

I made some changes to my branch. I hope this will allow the video to play without too many stutters. Instead of updating the texture at a regular interval, I determine when to schedule the next texture update based on the next video frame timestamp.

I also tried to make the libraries loading a bit more platform compatible. At the moment, for Windows, the file names are hard coded. I'm thinking we could give a hook for the user to set its own lib filename at pyglet startup. Something along the lines of pyglet.options. What do you think?

Dan

[Benjamin Moran]

Sorry for the slow reply. Very promising progress!

I just gave it a try, and it looks like the audio problems I was seeing are fixed now. It turns out that they were 5.1 audio, which I can see you added support for in your commits. It also looks like my test videos all play smoothly now! I've been using these Big Buck Bunny trailers to test: https://peach.blender.org/trailer-page/
My other DTV recordings also play smoothly now!  One issue I did notice was that skipping around seems to freeze the video in the 1080p .mov BBB trailer above. Other than that, this is really shaping up nicely.

The library loading works well now. If you can, however, I would suggest switching to pyglet.compat_platform instead of sys.platform. It's actually the same thing, but pyglet.__init__ tweaks it a big for BSD, etc. support. 

I like the idea of something like pyglet.options for specific library names. It's too bad there are so many different individual libraries names. Maybe something like pyglet.options.ffmpeg_libraries = ["avcodec-99", "swresample-9", etc....]?
In the actual loader code, maybe a more elegant version of this:

for lib_name in pyglet.options.ffmpeg_libraries:
    if lib_name.startswith("avcodec"):
        new_lib_name = lib_name
avcodec = pyglet.lib.load_library("new_lib_name")

[Daniel Gillet]

Hi Benjamin,

Thank you for giving this a try. I'm glad things are looking better.

I pushed a few more commits which tackle some of the points you made. In no particular order:

• Using pyglet.options to define the FFmpeg dll names under Windows. I didn't use it for the other platforms, using the default names. This would ensure you could distribute your code for all platforms. For Windows you would have to provide the FFmpeg binaries. For Linux / OS X, you could rely on a default install. And if not, you could still provide the binaries. But with these 2 platforms, the version in the filename is not a problem, and it comes after the filename extension, if I understood it correctly.
• I used pyglet.compat_platform as you suggested.
• I tried the video you provided and I could reproduce the bug. It was a subtle bug, ie. Heisenbug, as it would disappear as soon as I would turn on the debug messages. But I could finally find out where it came from. So please, let me know how the seeking is now working for you.
  As a side note, it's normal that seeking only brings you to an approximate point in time. It's the application responsibility to skip the first frames until the correct point in time is found. This is something the media player is not doing. We could try to implement it to illustrate how to do it.
• I removed all the references to AVbin. I thought it was confusing to refer to this lib which is not used. I can imagine the confusion for someone trying to read the code. He would not understand why there are some classes called AVbinXXX while we're wrapping FFmepg.

I guess there should be some work going to the documentation. I never built pyglet documentation, so I guess I'm good for some more reading. :)

Dan

[RedDevil91]

Hi Dan,

Firstly thank you for your work with the new ffmpeg backend. I have tested your implementation and it worked for me flawlessly on videos which has audio. However on videos without audio stream the seeking and EOF events don't work. Question: Is it possible to solve the problems with the SilentTimeAudioPlayer? Thanks in advance!

Best regards,

RedDevil91

[Daniel Gillet]

Hello RedDevil91,

Thank you for trying out the code. It's motivating to know people have an interest in this!

Sure, I will try to fix the SilentTimeAudioPlayer. It's indeed the goal that Pyglet can play video with and without any audio. :) But I'm pretty busy at the moment. So it might take some more time.

And I have to ask you with such a pseudo: are you Belgian? :)

Dan

[RedDevil91]

Hi Dan,

Thank you again for your hard work. If you need a tester (for Windows platform) I am ready to help.

RedD

p.s: No I am not Belgian :D

[Benjamin Moran]

The changes look good, Dan. It's too bad there isn't a better way to handle the library names on Windows, but I think the code looks fairly clean as you've written it.
In the case where a developer is going to bundle the binaries, they can just name them whatever they want.

I'll have to get that media_player rewritten soon, and see about implementing proper frame catch-up functionality.

[Daniel Gillet]

Hello,

Just a quick update. I'm still working on this project. :) I'm currently wrestling with the SilentAudioPlayerPacketConsumer. The progress is promising though. I haven't pushed my changes yet, but you will soon be able to test it.

The seeking functionality now goes to the exact location in time. The drawback is that this creates a bit of a delay before the video can play again, as the program is busy going through the packets until it finds the correct one. This is something you will have to test for yourself and decide whether this delay is acceptable or not.

I hope I'll be able to give you something to try by the end of next week.

Cheers,

Dan

[Benjamin Moran]

Thanks for letting everyone know where you're at.

Personally, I like the seeking behaviour you've described. From a users standpoint, I've always found it a bit annoying when media players force the scrollbar to the nearest key frames. It makes the seeking behavior feels a bit clunky, or notchy somehow. I've never liked that, and prefer to wait a moment for playback to resume.

Speaking of media players, I started refactoring the example player last week, but it might be cleaner to just rewrite it from scratch. It does work fine, but It would be nice to get rid of all the legacy OpenGL code in the examples. They should all be using the pyglet.graphics module at this point.

[Daniel Gillet]

Hi Benjamin,

Looking forward to seeing the new media player!

I've pushed a new commit which implements the new seeking behaviour. Also the Silent Audio driver saw some fixes. So video without any sound stream should now play (hopefully) fine.

I had to touch some code in the different audio drivers. I only have here DirectSound. I could not test OpenAL. So if someone could test that and report any problems, that would be very helpful. I didn't have the time yet to implement this in the Pulse audio driver. So this will have to wait a bit. If you are using Pulse, the code will crash, complaining that a seek method should be implemented.

Let me know how it goes. Also let me know what you think about the delay when you seek.

Dan

[Benjamin Moran]

Sounds good, I'll give this a try this evening.
I can at least confirm if OpenAL works on Linux, and will also try to test a file without audio track.

I'm not very familiar withe driver code, but let me know if you need help on the Pulse side.

-Ben

[Daniel Gillet]

Thanks Benjamin,

I've pushed more changes with an attempt at a Pulse Audio implementation. If you have the chance to test it, I would appreciate. I honestly don't expect it to work just like that. That would be too easy. :) But I cross my fingers.

Dan

[Benjamin Moran]

I'm happy to report that it works, even on PulseAudio!

I ran through the Big Buck Bunny trailers on both OpenAL and Pulse. Playback is identical.
As for seeking, the delay is quite small. For the BBB trailers, it's only a fraction of a second. I think that's perfectly acceptable.  Have you tested any videos that are any worse than this? If not, I'd say it's OK. We can maybe get some other opinions on this as well.

One thing I have noticed, is that the video playback seems very slightly stuttery with the 1080p trailers only. I also tested a 1080p trailer (the *.mov one) with the audio track removed, and that one seemed to play completely smoothly. Do you see this at all on Windows? I wonder if there is some slight rounding errors somewhere that are not apparent in the SilentAudio player.

Also, just wanted to say that you've done a fantastic job on this.

-Ben

[Daniel Gillet]

Hi,

Well this is very good news!

I'm glad you find the seeking delay small enough. Truth is, there is still room for improvement. But my aim was to first make it work. 

I have downloaded the whole BBB movie in avi format (in full HD) from here: https://peach.blender.org/download/ This is where I find the biggest delay when I seek.

 

One thing I have noticed, is that the video playback seems very slightly stuttery with the 1080p trailers only. I also tested a 1080p trailer (the *.mov one) with the audio track removed, and that one seemed to play completely smoothly. Do you see this at all on Windows? I wonder if there is some slight rounding errors somewhere that are not apparent in the SilentAudio player.

I don't experience this on Windows. I believe it's not really a Windows - Linux problem, but more the audio driver which is different. So in order to trouble shoot this a bit more, could you let me know if you experience the same problem on OpenAL and PulseAudio? Try first to watch the video without touching anything, then try the same with some seeking and report if it's the same or it's getting worse. I've just found a small bug in OpenAL and pushed the fix on my repo. But this should not affect how the video plays when you don't touch it at all, only if you stop or seek.

 

Also, just wanted to say that you've done a fantastic job on this.

Thanks a lot! I appreciate. I must say I'm also proud of the work that has been accomplished. The code is far from perfect. As I stated several time, it would be better (understant more efficient) to re-write it completely because it's doing things in very complicated ways for no good reasons. The worse problem in my opinion is the use of Threads for decoding audio and video which is totally unnecessary because of the GIL (global interpreter lock). It's actually even decreasing performances!

If I find the motivation to carry on after this phase is done, I will maybe try to rewrite the player to use pyglet event loop rather than threads. This should improve things a little. For a real performance boost, I could change the Threads into Processes which would then indeed use the other cores to decode the video and audio. But we will loose time passing data around the processes. I have no idea if this would in the end improve performances so much.

By the way, regarding the seeking, maybe I could try to add a keyword argument to the player seek method to let the user choose if he wants a precise seeking behaviour or if he prefers the regular seeking which is less precise but quicker.

I think I read you're living somewhere in Japan. So we have quite a big time difference as I'm in Europe. But maybe we could try to arrange a Skype meeting and see if we can find what the problem is with the stutter on Linux.

Dan 

 

[Benjamin Moran]

I've tried downloading the full BBB movie in mp4/avi format, but unfortunately both mirrors seem unresponsive at the moment. I'll give it another try in the morning and see how it performs.

About the micro-stuttering, sorry I wasn't clearer before. I've tried with both PulseAudio and OpenAL, and the result is the same. I then tried remuxing the video without the audio stream, and the playback was smooth. (So It does not appear to happen with the SilentAudio player). The stuttering is very very minor, but it's noticable.  Seeking around in the video doesn't appear to have any affect on the behavior.

It's an interesting idea to add an option to the seek method. We probably want to lean towards keeping it simple, but lets see what kind of other feedback we get first and then consider if its worth it.

I agree on the usage of Threads. The idea of using Processes is very appealing to me. I've done a lot of work on the procedural audio module this year, and I've also thought about using processes there as well. In my experience with the multiprocessing module, the IPC overhead is really not bad at all for this kind of job. There is some overhead, but it's sub-milisecond in my experience. Obviously this adds up fast if you try to do a million function calls, but we're only talking about arrays of video packets for full frames of video. Also, since we know in advance how big the video frames are, we could possibly use a multiprocessing.Array for potentially faster access to the data. This would be a fun one to experiment with in the future!

I'm certainly up for a Skype call if we can make it work. Weekends are best for me.

-Ben

[Daniel Gillet]

Hello,

I think I could only download the movie from the torrent link.

Thanks for the clarifications. I understand it happens with both OpenAL and PulseAudio. The silent audio driver is obviously less CPU intensive, which might be why the video plays nicely. Unless it's the audio time which is creating problems. But it means it would happen both with OpenAL and PulseAudio.

For a skype call, my pseudo is danb737. Let's try to organise something there.

In the meantime, could you try the following:

• On lines 273-273, remove the if _debug: so that the next print statement will happen without debugging. This is easier than turning debugging on, as you will be flooded by other debug messages.

Let me know if you get a lot of print statements telling that the frame was skipped. And let me know by how much time is was missed. This would help in determining if that is what makes the stuttering.

Also have a look at your CPU usage. Is it close to full on a single core?

Thanks for helping!

Dan

[Benjamin Moran]

Hey Dan,

It doesn't seem CPU related. None of my cores crested 20% or so while testing. I've also done a few VMProf profiles as well, and it looks like resource usage is really low (much lower than expected, really).

With the debug prints, it looks like both the silent and openal/pulse playbacks have some skipped frames. Not many though. Generally they looked like this:
Frames discarded at timestamp 0.04. Missed by 0.06. time 0.10
Frames discarded at timestamp 10.68. Missed by 0.10. time 10.78
Frames discarded at timestamp 10.72. Missed by 0.06. time 10.78
Frames discarded at timestamp 10.80. Missed by 0.05. time 10.85

I also looked at Radeontop (a linux graphics profiling tool), and it looked like GPU usage was almost zero. Even so, maybe it's an issue with the Mesa graphics stack I'm using. It would be great to have someone else with perhaps an Nvidia card on Linux try it. It would be a good sanity check, to make sure it's not just a specific thing to my systems (both of which are running Arch Linux).

I've sent you a request on Skype as well, so I'll see if I can catch you there.

-Ben

[Benjamin Moran]

Hi Dan,

I just had a thought. I'm now running the Wayland display server on both of my Linux machines, since Gnome has recently moved to it by default. Wayland itself is great, but legacy Xorg applications (of which pyglet is one), run inside a thin Xorg instance on top of Wayland.  I'm not sure if this is having any effect, but It's definitely worth a test. I'll test this later today under a pure Xorg display server and let you know if there is any difference. It doesn't account for the fact that it runs smoothly with the silent audio player, but it's still worth a test.

-Ben

[Daniel Gillet]

Hello,

Claudio, Bethany and I have been busy with the FFmpeg bindings. We have seen some nice progress. While busy with the development, a couple of questions arose for which we might need Rob or Benjamin to help answering them.

First question is about the inclusion of the ffmpeg binding code into pyglet. I wanted to make sure that the idea is still to have in pyglet/media/sources a ffmpeg.py file which would implement a FFmpegSource class deriving from StreamingSource. Or do you want this to be a separate project?

The second question is about the location of FFmpeg itself. On Linux and I guess on Mac OS, the binaries could be installed on the machine (in the default location) and pyglet would find them easily. But on Windows, this is another story. We need a mechanism where the developer can tell pyglet where to look for the FFmpeg binaries (the dll files). At the moment, I've added a pyglet option which is documented like this:

#: ffmpeg_libs_win

#:      A tuple containing the FFmpeg dll filenames for Windows. As on this

#:      platform there is no standard way to find the dll files, and the

#:      FFmpeg dll names have version number appended at the end, it's easier

#:      for the developer to state what are the filenames, and hence giving

#:      the version for each dll files. By default, the filenames are

#:      'avcodec-57', 'avformat-57', 'avutil-55', 'swresample-2', 'swscale-4'

I'm facing a couple of hurdles. First of all each dll have its version appended to the end of the file name. This is probably anyway a good thing as we have to explicitly state which version we want to use. The other problem is the location of those files. Maybe we could create an empty dll folder in the pyglet structure to allow developers to drop any dll they need and this would be a location where pyglet would look for binaries? More info can be found here: dlls placement and licences, integration

I understand you don't want to create any dependencies. But we also need to provide the users with an "easy" way of using those external libraries.

And a final question: Claudio developed a really awesome utility for measuring video player quality including numbers of frames dropped, difference between video and audio times and other things. Do you think it would be valuable to include it in pyglet for future development with the media player? Where should this utility suite live? In the tools folder maybe?

Thanks,

Daniel

[Benjamin Moran]

Hi Dan,

I'm happy to hear you guys are making progress.

For your first question, I do think it makes sense to keep them in pyglet/media/sources/, because this is where the avbin bindings are currently located. I think it make sense to keep all of the media stuff together as it's currently done.

For the second question, it seems like there might already be an easy way. I was looking over the pyglet.lib module, and I discovered a few things I was unaware of. First of all, it's possible to pass multiple library names to the pyglet.lib.load_library() method. For an example, pyglet/media/drivers/openal/lib_openal.py shows this in action. Basically, we can use kwargs for additional win32 library names. This might make "ffmpeg_libs_win" unnecessary.
In addition to that, pyglet already has the pyglet.options['search_local_libs'] option. If you look in the pyglet.lib module, it shows that this option will make pyglet also check for a subdirectory in the project path named "lib". If the ffmpeg libraries are placed there, it should be a solution for bundling ffmpeg libraries with applications. I'm interested if this works as expected on Windows.

For development tools, I think the "tools" folder is a good choice. We already have a few other similar things in there, such as the DDS viewer and font viewers. It would be valuable to include Claudio's tools if it can help future developments.  The tools folder itself needs a major cleanup (such as upgrading to Python 3), but that's another discussion :)

If anyone else has any additional comments or oposing views, please feel free to chime in!
-Ben

[Daniel Gillet]

Hi,

 

For the second question, it seems like there might already be an easy way. I was looking over the pyglet.lib module, and I discovered a few things I was unaware of. First of all, it's possible to pass multiple library names to the pyglet.lib.load_library() method. For an example, pyglet/media/drivers/openal/lib_openal.py shows this in action. Basically, we can use kwargs for additional win32 library names. This might make "ffmpeg_libs_win" unnecessary.
In addition to that, pyglet already has the pyglet.options['search_local_libs'] option. If you look in the pyglet.lib module, it shows that this option will make pyglet also check for a subdirectory in the project path named "lib". If the ffmpeg libraries are placed there, it should be a solution for bundling ffmpeg libraries with applications. I'm interested if this works as expected on Windows.

Thanks for that Benjamin. I've gone down that road. I had to fix something in lib.py for Windows to add the local lib folder to the PATH. See commit fcd693 I've hard-coded the FFmpeg version for Windows. My idea is that when a new version comes out, we need to make sure the ctypes wrapping is still valid and that there are no API changes that would require a rewrite of pyglet.media.

We still have a couple of bugs, as you are aware. But going forward I've also tried to fix the GroupSource class. It's basically used for queuing source on the Player. But I must say that the original intent is not entirely clear to me. It seems like they were trying to make a GroupSource behave like a normal Source, except it provides data from the first source in its queue. There were some configurations possible to ask to automatically switch to the next source in the queue when the preceding reached the end of its stream or to loop over that single source. They had a restriction that you could only queue Source with a similar audio and video format. But this is something I find rather surprising for the user then. He might not know that 2 media don't share the same audio format. And all of a sudden, they're not part of the same SourceGroup. 

So I retained the class just for the sake of not re-instantiating the audio player and re-creating the texture when Sources share the same audio and video format. But for the user, he does not see or care that sources are in a SourceGroup or different ones. For him the Player has different Sources on it. The default Player behaviour is to play the next media when the preceding has finished. The Player has a `loop` property which by default is False. If set to True, the Source will loop until we call on the Player its `next_source()` method. This obviously could be mapped to a GUI button or a key. The media player could demonstrate that. I just wanted to know if you guys had different opinions on the SourceGroup class and its usage. If my approach is sufficient, I might even re-consider this limitation about audio and video format. The SourceGroup would be the Player queue. It would be smart enough to tell the Player when changing Source if the Player should also re-instantiate a new AudioPlayer or re-create a new Texture.

For something completely different, I have written a small media manual which is more intended to developers rather than users. I will try also to make a small document which explains briefly the changes I made from the original code. But I wanted already to share a bit with you, because I want to know if it's not a no-go for you.

In the AudioPlayers (DirectSound and OpenAL) there was some threading madness going on there. There was also some threading madness going on in the Player itself, with a worker decoding video frames asynchronously. I have remove all that. The code is now single threaded! And guess what? It goes faster. :) But more importantly it is easier to reason about and to maintain. The threading was not bringing anything because of the GIL. Maybe when this code was written, the people didn't know or care about the GIL. But anyway, all these locks, threads and such are a thing of the past.

The other thing is the Source.get_audio_data method has a change in its API. Before it was get_audio_data(self, nbytes). Now it is def get_audio_data(self, bytes, compensation_time=0.0). This compensation time is about synchronizing the audio to the master clock. For a given audio sample rate, a given number of bytes will provide you with a given duration. Say you're asking for 1 second of audio data. But at the same time, the Player has noticed that the audio is too far ahead of the master clock. So that 1 second of audio data should play in let's say 0.9 sec. So compensation time will equal to -0.1 in this case. And the Source will shrink the audio samples to make the 1 second data fit in 0.9 seconds of real time.

I've added this optional kwargs to the other Sources, like the Procedural source. But there, it does not do anything with this parameter. The reason being that there is no audio synchronization necessary for those Sources. But I might be wrong.

That's it for the moment. I will come back later with further update while we are consolidating things.

Dan

 

[Benjamin Moran]

Hi Dan,

Sorry for the slow reply (busy weekend here!).  Your changes with lib loading look great. This way looks a lot cleaner and simpler. I also agree with you about the hardcoded library names. I believe ffmpeg is a lot better with regards to API stability these days (maybe as a result of the whole libav split?). However, we still want to confirm new versions, as you said. We might also want to see about adding an internal ffmpeg lib version check at some point, for Mac/Linux.

I've looked over the SourceGroup, and I'm also unsure of it's benefit. It seems that users are not intented to use it directly. It's also not well documented, and has some "TODO"s left in the code. Perhaps it was never finished? Since you're already doing a major cleanup, please go ahead with removing it completely if you think it would be best. It's such a small bit of code, so we can reimplement it easily if we realize some limitation later.

The removal of the unnecessary extra Threading is a great thing! It should make it easier for other developers in the future as well.

Your get_audio_data() addition looks like a perfectly reasonable way to handle the drift issue. For the ProceduralSource, as you said, it doesn't need it. It doesn't really need the seek() or some other methods either. I've actually been thinking a lot about the Procedural classes. Maybe this is another topic, but I have a few things I'd like to fix/change. Firstly the name, because these days "procedural" makes people think of random procedural generation. Secondly, the module wastes a lot of code to support both 8bit and 16bit sample sizes (which doesn't really save any memory in Python). I think I'll start another thread about that!

Thanks a lot for writing out your updates. This is really going to benefit pyglet a lot to have this functionality returned, and I think a lot of users will be happy to have it.
-Ben