Problem with video:still / video:player / gstreamer toughts

[Pavel Šorejs]

Hi all

First i would like to say Praxis Live is awesome. 

I have used it for a multimedia part of an escape game - 4 different video screens playing video loops and rendering text and still images, 6 audio channels for main music and sound effects. All running in headless mode controlled using OSC protocol. All running great.

I am writing this because i have found some problems, some of them has workarounds, some don;t.

First - i am now running version 3.5 but this also happens on 4.1 - there is memory corruption problem when using video:still in OpenGL. I have multiple Video patches - all of them OpenGL, all of the running full screen, each running on different monitor (actually have 4 of them). In one of them i have video:still, into which i am dynamically loading images. When everything is running, after multiple image changes (depends, but ranges from 10 - 15 image changes)  it looks like the texture memory in other patches got badly corrupted and the corruption worsens with each image load. All the graphics in other patches starts jumping all over the place, the textures got badly distorted and flashing, weird random blocks and noise appears. The patch with video:still is fine though. If i switch the one patch with video:still to software renderer, the problem completely disappears (that is workaround i am using now). I was able to isolate the problem to video:still element, nothing else does it, but wasn't digging deeper for now.

Second - in version 4.0 and beyond, there is some optimizations regarding video:player and not processing video output when it is not rendered. In couple of places i am using 2 video players going into xfader, with animator connected to xfader's mix input. Lets say video 1 is playing, video2 is stopped and xfader mix is at 0. I send OSC command to start video2 and send 1 to animator input to gradually crossfade to the playing video (sometimes the input1 of xfader is till image, but it is behaving the same). In version 3.5 this works great. On version 4.0 onwards the video2 randomly doesn't start. More often it doesn't than it does but it doesn't look deterministic. Looks like some kind of race condition or something. Workaround for me for this is to stay on 3.5.

Thirs - this is not a bug, rather maybe some points for discussion. I am playing couple of 1080p videos in the patches. It works, but the frame rate is not the best. Acceptable, but not the best. I was trying various things and found out this. If i modify praxis live to use some kind of direct renderer in the gstreamer pipeline - for example xvimagesink - it runs super smooth. So looks like the pixel copying in java is the problem here. So i tried to look into it and found the OpenGL context sharing thing in gstreamer. As i understand, the support for this is missing in gst1-java-core - https://github.com/gstreamer-java/gst1-java-core/issues/82 - i can volunteer to make it happen probably, but i will need at least some hints on where to start  as i am not very familiar with gst1-java-core codebase. These guys sort of did that https://github.com/gohai/processing-glvideo for processing - but they completely evaded the Java  <-> C interop by writing the whole player in C so it is probably of no use.

Thank you again for the great work.

Pavel Sorejs

[Neil C Smith]

Hi,

On Tue, 9 Apr 2019 at 15:03, Pavel Šorejs <sor...@gmail.com> wrote:
> First i would like to say Praxis Live is awesome.

Thanks!

> I have used it for a multimedia part of an escape game - 4 different video screens playing video loops and rendering text and still images, 6 audio channels for main music and sound effects. All running in headless mode controlled using OSC protocol. All running great.

Would love to see some documentation of this project. Even better if
it's somewhere it can be shared! :-)

> First - i am now running version 3.5 but this also happens on 4.1 - there is memory corruption problem when using video:still in OpenGL. I have multiple Video patches - all of them OpenGL..

First question - how are you running the headless project(s)? Are the
video patches all running in one JVM or across multiple processes /
machines? Unfortunately, Processing has some serious issues running
multiple outputs in one VM which might explain what you're seeing, in
particular with resource (texture) cleanup affecting other outputs.

> Second - in version 4.0 and beyond, there is some optimizations regarding video:player and not processing video output when it is not rendered.

video:player was completely rewritten in 4.0 to be recodeable,
although the behaviour of not processing video when not rendering
hasn't changed. However, I noticed a bug the other day whereby
position and playback state are updated inside draw() so the component
state doesn't correctly update, although the video is playing fine.
Are you sure it's not this? Are you sure playback isn't actually
working? If so, please share a simple patch that replicates the issue
and I'll have a look.

> Thirs - this is not a bug, rather maybe some points for discussion. I am playing couple of 1080p videos in the patches. It works, but the frame rate is not the best. Acceptable, but not the best. I was trying various things and found out this. If i modify praxis live to use some kind of direct renderer in the gstreamer pipeline - for example xvimagesink - it runs super smooth. So looks like the pixel copying in java is the problem here. So i tried to look into it and found the OpenGL context sharing thing in gstreamer. As i understand, the support for this is missing in gst1-java-core - https://github.com/gstreamer-java/gst1-java-core/issues/82 - i can volunteer to make it happen probably, but i will need at least some hints on where to start as i am not very familiar with gst1-java-core codebase. These guys sort of did that https://github.com/gohai/processing-glvideo for processing - but they completely evaded the Java <-> C interop by writing the whole player in C so it is probably of no use.

There isn't any pixel copying in Java, or at least there shouldn't be
(just check with setting the renderer to OpenGL:GL3). However, the
pixel data is potentially pulled down to the CPU and passed back to
the GPU outside of the JVM. The support for OpenGL context sharing is
coming soon, although you're welcome to hop on that issue and help out
if you can. I'm aware of Gohai's work with gl-video for this (and
helped him with updating the Processing Video library for
gst1-java-core too). There is potentially work going into the
Processing Video library as part of GSoC which may also touch on this,
so hopefully all the projects can work together to make this happen.
The gl-video library is somewhat more limited than the support we
require. It should actually be possible to use the gl-video library
within a custom PraxisLIVE component if it works on your setup and you
need it now.

Best wishes,

Neil

--
Neil C Smith
Artist & Technologist
www.neilcsmith.net

PraxisLIVE - hybrid visual live programming
for creatives, for programmers, for students, for tinkerers
www.praxislive.org

[Pavel Šorejs]

Hi

On Wed, Apr 10, 2019 at 6:01 PM Neil C Smith <ne...@neilcsmith.net> wrote:

Hi,

On Tue, 9 Apr 2019 at 15:03, Pavel Šorejs <sor...@gmail.com> wrote:
> First i would like to say Praxis Live is awesome.

Thanks!

> I have used it for a multimedia part of an escape game - 4 different video screens playing video loops and rendering text and still images, 6 audio channels for main music and sound effects. All running in headless mode controlled using OSC protocol. All running great.

Would love to see some documentation of this project.  Even better if
it's somewhere it can be shared! :-)

Well, i can't share the assets, that's not possible - i probably can share the patches, but in some rather limited way - i mean i can't just stick them on GitHub, but i probably can send them so some limited amount of people under some conditions. Don't get me wrong, if this was 100% my project, i would share them no problem - but IP on this thing is a bit more complicated.

 

> First - i am now running version 3.5 but this also happens on 4.1 - there is memory corruption problem when using video:still in OpenGL. I have multiple Video patches - all of them OpenGL..

First question - how are you running the headless project(s)?  Are the
video patches all running in one JVM or across multiple processes /
machines?  Unfortunately, Processing has some serious issues running
multiple outputs in one VM which might explain what you're seeing, in
particular with resource (texture) cleanup affecting other outputs.

Yes, it is 4 video + 1 audio + 1 OSC patch in single JVM.

 

> Second - in version 4.0 and beyond, there is some optimizations regarding video:player and not processing video output when it is not rendered.

video:player was completely rewritten in 4.0 to be recodeable,
although the behaviour of not processing video when not rendering
hasn't changed.  However, I noticed a bug the other day whereby
position and playback state are updated inside draw() so the component
state doesn't correctly update, although the video is playing fine.
Are you sure it's not this?  Are you sure playback isn't actually
working?  If so, please share a simple patch that replicates the issue
and I'll have a look.

I am pretty sure the playback didn't work. I will have some time alter this week to test it again, i will try to upgrade to latest PraxisLIVE and do deeper tests. Will report.

 

> Thirs - this is not a bug, rather maybe some points for discussion. I am playing couple of 1080p videos in the patches. It works, but the frame rate is not the best. Acceptable, but not the best. I was trying various things and found out this. If i modify praxis live to use some kind of direct renderer in the gstreamer pipeline - for example xvimagesink - it runs super smooth. So looks like the pixel copying in java is the problem here. So i tried to look into it and found the OpenGL context sharing thing in gstreamer. As i understand, the support for this is missing in gst1-java-core - https://github.com/gstreamer-java/gst1-java-core/issues/82 - i can volunteer to make it happen probably, but i will need at least some hints on where to start  as i am not very familiar with gst1-java-core codebase. These guys sort of did that https://github.com/gohai/processing-glvideo for processing - but they completely evaded the Java  <-> C interop by writing the whole player in C so it is probably of no use.

There isn't any pixel copying in Java, or at least there shouldn't be
(just check with setting the renderer to OpenGL:GL3).  However, the
pixel data is potentially pulled down to the CPU and passed back to
the GPU outside of the JVM.  The support for OpenGL context sharing is
coming soon, although you're welcome to hop on that issue and help out
if you can.  I'm aware of Gohai's work with gl-video for this (and
helped him with updating the Processing Video library for
gst1-java-core too).  There is potentially work going into the
Processing Video library as part of GSoC which may also touch on this,
so hopefully all the projects can work together to make this happen.
The gl-video library is somewhat more limited than the support we
require.  It should actually be possible to use the gl-video library
within a custom PraxisLIVE component if it works on your setup and you
need it now.

Looks like i misunderstood the code flow. But than it is strange why is there noticable difference when the video is running through Praxis Live as opposed to when it is rendered directly to some window (or even glimagesink) 

 

Best wishes,

Neil

--
Neil C Smith
Artist & Technologist
www.neilcsmith.net

PraxisLIVE - hybrid visual live programming
for creatives, for programmers, for students, for tinkerers
www.praxislive.org

--
You received this message because you are subscribed to the Google Groups "PraxisLIVE software discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to praxis-live...@googlegroups.com.
To post to this group, send an email to praxi...@googlegroups.com.
Visit this group at https://groups.google.com/group/praxis-live.
For more options, visit https://groups.google.com/d/optout.

[Neil C Smith]

Hi,

On Wed, 10 Apr 2019, 22:25 Pavel Šorejs, <sor...@gmail.com> wrote:

Well, i can't share the assets, that's not possible - i probably can share the patches, but in some rather limited way 

To be clear, I was thinking photos, videos, posts about the overall project, not the patches. Just interested in finding out (and sharing if possible) the sort of thing people are building. 

Yes, it is 4 video + 1 audio + 1 OSC patch in single JVM.

OK, in which case it's sort of expected unfortunately. I am considering whether to fork the Processing library to fix this. But at the moment distributed hubs is your only way to workaround this. 

Looks like i misunderstood the code flow. But than it is strange why is there noticable difference when the video is running through Praxis Live as opposed to when it is rendered directly to some window (or even glimagesink) 

Not really. It is still doing more stuff on the CPU, including colour conversions. But in GStreamer code not in Java.

Best wishes, 

Neil