Playing video using VLC in psychopy/pyglet window in full screen

[Daniel Schreij]

I'm working on a project which requires me to render video that is
delivered by the vlc python module into a psychopy/pyglet window.
So more specifically, I have a program which creates a vlc based video
player using the python vlc bindings (http://wiki.videolan.org/
Python_bindings). These bindings allow to output the video to a
specific window using a function which you just specify the OS
internal reference id of the desired window.

You can find the reference id of the used window in pyglet under the
variable:
win_id = pyglet.window.winHandle._hwnd # for windows
win_id = pyglet.window.winHandle._window # for linux

I then specify this id in

vlc_player.set_hwnd(win_id) #Windows
vlc_player.set_x_window(win_id) #Linux

to indicate that vlc has to render its output to this window.

When I run my psychopy experiment in windowed mode, this works
perfectly! I get perfect video output in the psychopy window with
sound.
The only problem is that when I run the experiment full screen, the
vlc bindings can't find the specified window anymore and opens a new
one of its own to play the video in. _hwnd (or _window) however still
contain a reference id to a window when psycopy runs in full screen
mode.

I know about the AVbin class of psychopy, which offers similar
functionality, but this project requires me to use VLC as psychopy
runs as a backend that can be exhanged (with for instance pygame or
pyOpenGL). I need the video player to be compatible with all backends,
and AVbin is of course psychopy-specific. The solution I posted above
works beautifully with pygame and pyOpenGL in both full screen and
windowed mode. Only the full screen psychopy mode is giving me
trouble.

Does anybody know how to get a correct reference to the full screen
psychopy/pyglet window (or surface) so an external program like vlc
can output to it?

For those interested, the full source code of my video_player plugin
can be found at: https://github.com/dschreij/media_player_vlc. It is
written as a plug-in of OpenSesame experiment builder, so it cannot be
run independently.

[Jonathan Peirce]

Thanks for the info Daniel. That's really interesting and might be a
better alternative to avbin anyway, partly for independence between the
backends and partly because avbin has a slight tendency to fall over,
and doesn't support as many different formats as I'd like.

I don't know the answer to your question, but maybe under fullscreen
mode you want to provide the display handle instead of the window
handle? Alternatively, is there a way having loaded a movie using this
lib that you can simply fetch the image data for the current frame? That
way it could be inserted into a texture and PsychoPy's own rendering can
take over. That also means we can do things like rotate, flip or make
semi-transparent. This is the method I've used with the avbin movies.
Maybe you'd take a look at MovieStim and see if there's a way to create
a VLC replacement?

How does installation work? To use the library do you simply get the
user to install the VLC application themselves somewhere? Or is there a
dll that ships separately? I'm wondering how distribution would work. (I
don't actually think it would be too bad to say users have to install
VLC to use movies)

Jon

--
Jonathan Peirce
Nottingham Visual Neuroscience

http://www.peirce.org.uk

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

[Daniel Schreij]

Hi John,

Thanks for your reply: The installation of and referencing to vlc is
relatively simple. the python bindings indeed simply reference to
installed versions of VLC.
As long as vlc is installed at its default location of the specific OS
that is used, the vlc.py module will automatically locate and link to
the libvlc.dll (or linux equivalent) to use all of vlc's functions. It
is also possible to reference to libvlc at other locations, but this
requires a bit more configuration (see http://wiki.videolan.org/Python_bindings#Win32_initialization
for details).

As far as retrieving single frames form vlc, I think this is is a bit
more problematic. I think these bindings (and vlc in general) are only
designed for simple video playback and not to perform complex
operations on the video and audio streams. As far as my experience
goes, I have found it really hard to use these bindings for frame-per-
frame operations (I also requires something like this at the moment).
Basically you just tell vlc to play the video, and it goes... The only
thing that I have found to have control over, is specifying in which
window it renders its video output. VLC does provide some transform
filters of its own though, with which you can manipulate movie output
(specified in http://www.advene.org/download/python-ctypes/doc/).

As for the full screen problem: could you tell me which variable of
winHandle contains the reference to the screen or display surface
used?

Hope this was of help.



On Feb 3, 6:17 pm, Jonathan Peirce <jonathan.pei...@nottingham.ac.uk>
wrote:

[Jonathan Peirce]

On 03/02/2012 20:16, Daniel Schreij wrote:
> As far as retrieving single frames form vlc, I think this is is a bit
> more problematic. I think these bindings (and vlc in general) are only
> designed for simple video playback and not to perform complex
> operations on the video and audio streams. As far as my experience
> goes, I have found it really hard to use these bindings for frame-per-
> frame operations (I also requires something like this at the moment).
> Basically you just tell vlc to play the video, and it goes... The only
> thing that I have found to have control over, is specifying in which
> window it renders its video output. VLC does provide some transform
> filters of its own though, with which you can manipulate movie output
> (specified in http://www.advene.org/download/python-ctypes/doc/).

This is a shame. The ability to render to an opengl texture seems a
really natural, flexible solution.

> As for the full screen problem: could you tell me which variable of
> winHandle contains the reference to the screen or display surface
> used?

Have a look at the pyglet.window module for various different sorts of
handles on this. e.g.::
display = pyglet.window.get_platform().get_default_display()
screen = display.get_default_screen()

> Hope this was of help.

Yes, thanks, it's very interesting to know about anyway.

Jon

--
Dr. Jonathan Peirce
Nottingham Visual Neuroscience

http://www.peirce.org.uk/

[Daniel Schreij]

> Have a look at the pyglet.window module for various different sorts of
> handles on this. e.g.::
>      display = pyglet.window.get_platform().get_default_display()
>      screen = display.get_default_screen()> Hope this was of help.

Thanks for these, but unfortunately the objects these return do not
carry any id's, which can be used as a reference for vlc to render its
contents to. I think the _hwnd or _window handles are still the way to
go, as a full screen display is still internally referenced as a
window by the OS as far as I know? It simply is a window that fills
the screen and has no borders.
I think the problem is that vlc does not manage to create a directx
plane to render to in the full screen psychopy window. It does manage
to do so with pygame and pyOpenGL windows (both fullscreen and small),
in which vlc temporally creates a layer on top of the pygame surface.
I can see this is the case, as when movie playback is finished and vlc
destroys its drawing plane, the the original contents of the window
that were drawn by pygame return (as is the case with vlc movie
playback in a non-full screen psychopy window). I think when in full
screen, psychopy/pyglet somehow blocks input by other applications,
causing vlc to spawn a new window to render its movie in.
Nevertheless, this might hopefully be solved by simply tweaking some
parameters in psychopy...