Updating examples

[Neil C Smith]

Hi All,

I'm finally finding some time to look at updating the examples
repository. I'm moving each example into a separate Gradle project,
and each example now includes code to set up default GStreamer paths
for macOS and Windows, so hopefully a bit easier to get started.
Should all work with ./gradle run across each OS assuming GStreamer is
installed.

I've updated 4 examples so far, and tried to add more documentation in
the source for each step. Overall README will be added before
merging.

Seem a good step forward? Help testing and checking text would be
great, as well as (once merged) additional examples.

Repo branch at :
https://github.com/codelerity/gst1-java-examples/tree/examples-rev
WiP PR : https://github.com/gstreamer-java/gst1-java-examples/pull/15

Thanks and best wishes,

Neil

--
Neil C Smith
Codelerity Ltd.
www.codelerity.com

Codelerity Ltd. is a company registered in England and Wales
Registered company number : 12063669
Registered office address : Office 4 219 Kensington High Street,
Kensington, London, England, W8 6BD

[Scooter Willis]

Neil

Looks great and very interested in the RTSP example. Still trying to figure out what is possible with java gstreamer.

If I have a Java app as an image sink from a webcam using gstreamer want to do object detect box and/or text overlay of the image then have that java bufferedImage and the stream of following images be the source for the h264 encoding RTSP stream via streamers.

Any code examples you can reference?

Along similar odd/advanced use case if the webcam is an 8 megapixel camera want to have a 640x480 crop and be able to have a moving PTZ. Want to modify existing pipeline that is doing the crop and be able to change the parameters via java applicatio in real time for existing pipeline. 

If possible any code examples to get me started?

Thanks

Scooter

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5Hi_BLsapre1TNvANa4xsyBVmUKg34jxzA%2BsMhjoXpXww%40mail.gmail.com.

[Davide Perini]

Very nice done,
I think that it's way easier to start using gstreamer-java with those
examples.

Fixed some parts of my code thanks to that examples.

Thank you
Davide

[Neil C Smith]

On Fri, 5 Feb 2021 at 00:21, Scooter Willis <will...@gmail.com> wrote:
> Looks great and very interested in the RTSP example. Still trying to figure out what is possible with java gstreamer.

Thanks. At the moment, just looking at updating the main key examples
that I originally wrote, and adding one for the JavaFX integration.
Then I'll take a look at updating the contributed ones. What exactly
with rtsp are you looking for?

Also planning on porting and updating an old webrtc example.

> If I have a Java app as an image sink from a webcam using gstreamer want to do object detect box and/or text overlay of the image then have that java bufferedImage and the stream of following images be the source for the h264 encoding RTSP stream via streamers.
>
> Any code examples you can reference?

Not directly. I've done something with two pipelines, and feeding the
second with an AppSrc, but in the context of a larger framework. I'll
have a think on this one, and options to achieve.

Unless anyone else here as such an example already?! :-)

> Along similar odd/advanced use case if the webcam is an 8 megapixel camera want to have a 640x480 crop and be able to have a moving PTZ. Want to modify existing pipeline that is doing the crop and be able to change the parameters via java applicatio in real time for existing pipeline.
>
> If possible any code examples to get me started?

To clarify, cropping in the GStreamer pipeline not controlling the hardware?

Using the videocrop element you could set the four point properties
directly. Might be better with Controllers though, which should allow
for the control to be atomic and interpolated if wanted. There is a
controller example that moves text around already. May look at
another to control random cropping and demonstrate the other control
sources.

Best wishes,

[Neil C Smith]

On Fri, 5 Feb 2021 at 16:36, Davide Perini <perini...@dpsoftware.org> wrote:
>
> Very nice done,
> I think that it's way easier to start using gstreamer-java with those
> examples.

Thanks. Is it obvious I'm also trying to stop answering so many
questions about UnsatisfiedLinkError?! :-)

[Scooter Willis]

Need to take camera feed and overlay a graffic based on object detected and then send the frame on in the pipeline for h264 encoding and then RTSP stream to some number of clients.

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5EwNFmnsTXkMnNMOWqA0Q0MqZ1iYZuz4wGhMO-xCsuwVg%40mail.gmail.com.

[Scooter Willis]

Focusing on the problem of being able to source an image do some
processing on it and then adding annotations on the image and passing
it on.

The use case of drawing a bounding box around an object in a live
video stream is a good example and found good others asking the same
question on NVIDIA message boards where the complexity is the pipeline
is in the GPU memory space. Did not appear to be any easy ways to do
it and an example was given of passing in a paint method that would be
called to update an image on GPU.

Did find a plugin on windows rsvgoverlay that you could pass in a SVG
file or SVG text to overlay on an image so close.

For my particular example I am tracking QR codes in the video and for
debug purposes want to show the bounding box and decoded value in the
video stream for testing. Currently I have code that I can source the
gstreamer image into an appsink process the image, find the QR code
and setup a motion jpeg stream for debugging.

Ideally, I want to take the image in the Java application and provide
it as a source to a gstreamer pipeline to do h264 encoding and stream
as RTSP or write a file.

Any thoughts on how to use gstreamer java as an image source in a
gstreamer pipeline without being an official gstreamer plugin?

[Neil C Smith]

On Sun, 7 Feb 2021 at 18:14, Scooter Willis <will...@gmail.com> wrote:
> Any thoughts on how to use gstreamer java as an image source in a
> gstreamer pipeline without being an official gstreamer plugin?

It's potentially doable with an AppSrc, or with a Pad probe.

Pad probes covered here -
https://gstreamer.freedesktop.org/documentation/application-development/advanced/pipeline-manipulation.html?gi-language=c

And there's a Java test that might lead you in the right direction -
https://github.com/gstreamer-java/gst1-java-core/blob/master/test/org/freedesktop/gstreamer/PadTest.java#L269

Need to check the buffer is writable.

Hope that helps. It's something I intend to find some time to write
an example of.

[Scooter Willis]

It looks like TestProcessVideoRTSPToMP4.java is close in that you
connect to an RTSP stream, press enter to start, get images then send
to pipeline for conversion to MP4.

Put in a RTSP feed and get the following error when doing the setup
work to write a captured image. The exception gets thrown on the
following

info.setFlags(NativeFlags.toInt(srcBuffer.getFlags()));

Any thoughts?

GST finished initialization.
Processing of RTSP feed started, please wait...
Press ENTER to start processing footage from the cam, or type 'QUIT'
and press ENTER to exit...

Press ENTER to stop the process...
Feb 07, 2021 1:43:13 PM com.sun.jna.Native$1 uncaughtException
WARNING: JNA: Callback
org.freedesktop.gstreamer.elements.AppSink$2@4524411f threw the
following exception
org.freedesktop.gstreamer.GstException: Not supported by requested
GStreamer version
at org.freedesktop.gstreamer.Gst.checkVersion(Gst.java:613)
at org.freedesktop.gstreamer.Buffer.getFlags(Buffer.java:255)
at org.freedesktop.gstreamer.examples.TestProcessVideoRTSPToMP4$AppSinkListener.newSample(TestProcessVideoRTSPToMP4.java:253)
at org.freedesktop.gstreamer.elements.AppSink$2.callback(AppSink.java:232)
at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:520)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:551)

> --
> You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5HEd-%3Di5F-hRkzxEi1hnLQSFt7pojbB91-E0QW94bH11A%40mail.gmail.com.

[Neil C Smith]

On Thu, 4 Feb 2021 at 19:13, Neil C Smith <ne...@codelerity.com> wrote:
> I've updated 4 examples so far, and tried to add more documentation in
> the source for each step. Overall README will be added before
> merging.

OK, I've added one more example based on the JavaFX sink, added a
readme, and updated the main readme to give more prominence to the
examples.

This is just a starting point, but I've merged them in as it's still
in a better shape than what was there!

Please report any issues. Certain to be a few teething problems - mac
in particular seems to have issues with the camera ones, which might
just be the new permissions system, although I have cameras working
fine in packaged Java applications.

Many thanks and best wishes,

[Neil C Smith]

On Sun, 7 Feb 2021 at 18:47, Scooter Willis <will...@gmail.com> wrote:
> org.freedesktop.gstreamer.GstException: Not supported by requested
> GStreamer version
> at org.freedesktop.gstreamer.Gst.checkVersion(Gst.java:613)

If you ever see that it means you didn't pass at least the version
required by that feature in to Gst.init(...)

Try with Version.of(1, 10) at least for that - see the @since
information where it's throwing on getFlags. We use selective
enablement of features based on GStreamer version, even if the native
version you're running on is higher. I am considering a more helpful
exception message here - seems to confuse quite a few people.

Everything now in archive is old by the way, and as you've seen, might
have issues! One reason this task is underway.

[Scooter Willis]

Version resolved that problem.

Now working through the code as it creates a file of 0 size with the
press enter to start recording but pressing enter again per the
instructions to stop doesn't seem to be working.

> --
> You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5HORSFuwdbWVSP3PCRyFOVrU6VR%2BzLcHPiiQs%3DxNx_jdw%40mail.gmail.com.

[Scooter Willis]

In the program logic appears to be getting stuck on the following
pileline.sendEvent(new EOSEvent()) call not returning in the
while(true) loop that handles when to start and stop recording.
Commented it out and hung on the next line. Commented that out but
still getting a file output size of 0.

Any thoughts on what I should try?

pipeline.sendEvent(new EOSEvent());
gotEOSPipeline.acquire(1);

[Neil C Smith]

On Sun, 7 Feb 2021 at 19:35, Scooter Willis <will...@gmail.com> wrote:
> In the program logic appears to be getting stuck on the following
> pileline.sendEvent(new EOSEvent()) call not returning in the
> while(true) loop that handles when to start and stop recording.
> Commented it out and hung on the next line. Commented that out but
> still getting a file output size of 0.
>
> Any thoughts on what I should try?

Sorry, no. I didn't write that code and haven't looked at it much.
It's possible there are some wrong assumptions around concurrency
somewhere.

I will look at a simpler example of saving a stream to a file at some point.

I had a quick play with reading/writing to the buffer in a probe using
Java2D. Seems OK on Ubuntu, but haven't tested it on other OS yet.
Probably not highly performant!

https://github.com/codelerity/gst1-java-examples/blob/bi-probe/BufferProbe/src/main/java/org/freedesktop/gstreamer/examples/BufferProbe.java

[Scooter Willis]

Thanks! Will do some testing

> --
> You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5H9aVYT6T09Ay89cNYxNzsPKgyeO7t-%2B_AuGuGCvQmvUA%40mail.gmail.com.

[Scooter Willis]

Wanted to get your guidance on how gstreamer handles buffering pipeline data not being consumed fast enough.

For encoding or streaming video would seem to would buffer as much as you can hoping the downstream pipeline can catch up.

In my use case running on Jetson nano a low resource device with gpu that can do deep learning. 

What I am seeing in my java code is a 3-5 second delay in the video stream for detecting QR codes and can process a 4k image at 5 fps. The pipeline based on current config in theory is doing 4k at 30fps. This 25 fps will need to be dropped. 

In reality I want to always process the latest image because detecting the QR code position that location information needs to get sent out immediately. 

I will do some testing with the camera pointed at a clock and do a timestamp overlay of when I first see the image. The delay is consistent over time so suspect something in gstreamer pipeline is buffering. 

Any thoughts on how to have gstreamer elements drop images if pipeline has images in the queue? 

[Neil C Smith]

On Wed, 10 Feb 2021 at 03:57, Scooter Willis <will...@gmail.com> wrote:
> Any thoughts on how to have gstreamer elements drop images if pipeline has images in the queue?

Firstly, please start a new thread for these sort of questions.

There's a limit to what can be answered generically here, and it's
also worth asking upstream.

But, have a look at the elements with gst-inspect, eg. queue - check
docs on properties such as leaky, max-size-**. Also look at
properties on your source and sink elements. If at the (app)sink
level, look at how FXImageSink always processes only the last buffer
received, and try with sync=false to get the sink to ignore the clock.

[Scooter Willis]

Works well on windows and will use it as a starting point for the project I am working on. 

Thank you for taking the time to put a working example together. I don't know the API/Gstreamer framework well enough that I would have figured i out. 

Let me know how I can contribute to the Dinner and Drink fund as a way of saying thank you!!!!

[Neil C Smith]

Hi,

On Wed, 10 Feb 2021 at 17:07, Scooter Willis <will...@gmail.com> wrote:
> Works well on windows and will use it as a starting point for the project I am working on.

Good to know it works. Will do a little more testing myself and merge
it in to the examples.

> Thank you for taking the time to put a working example together. I don't know the API/Gstreamer framework well enough that I would have figured i out.

No problem! It's a useful example in general for the newer probe API.

> Let me know how I can contribute to the Dinner and Drink fund as a way of saying thank you!!!!

Thanks! :-) There's no direct way to do that. Obviously keep us in
mind if you or anyone you know needs commercial development work.

Or there's always donation options over at
https://www.praxislive.org/community/ - the project that first led to
me working on the bindings.

Thanks and best wishes,

[Phong Tran]

Dear Neil, 

I have tried you this example with SwingPlayer project:

But the problem is the video player loading when click play and pause many time. 

I tried it with env below:

Gstreamer: 1.18.3

Mac OS:  11.2.1

gst1-java-core: 1.4.0.

Pls find more info in the attachment below.

Thanks and Best Regards.

Vào lúc 11:14:13 UTC+7 ngày Thứ Sáu, 26 tháng 2, 2021, Phong Tran đã viết:

Dear Neil, 

I have tried you this example with SwingPlayer project:

But the problem is the video player loading when click play and pause many time. 

I tried it with env below:

Gstreamer: 1.18.3

Mac OS:  11.2.1

gst1-java-core: 1.4.0

Thanks and Best Regards.

Vào lúc 23:09:32 UTC+7 ngày Thứ Năm, 11 tháng 2, 2021, neil đã viết:

[Neil C Smith]

Hi All,

On Thu, 4 Feb 2021 at 19:13, Neil C Smith <ne...@codelerity.com> wrote:
> I'm finally finding some time to look at updating the examples
> repository.

I'm working on some more examples again. I've just merged an example
of WebRTC send and receive - forked from upstream example, and updated
to handle received video and audio streams. Feedback on this one
would be useful.

Examples repo at -

https://github.com/gstreamer-java/gst1-java-examples/

The original example is at -

https://gitlab.freedesktop.org/gstreamer/gst-examples/-/tree/master/webrtc/sendrecv/gst-java

Best wishes,

[Davide Perini]

Just playing with the WebRTC example, at a first glance it works awesome,
at a second one it works awesome.

Can't wait to find some time for integrating it in some ideas I have...
As always I will share them with the community.

Awesome bindings, thank you so much for adding such a good powerup to
Java and GStreamer.

Davide

[Neil C Smith]

Hi All,

On Wed, 16 Jun 2021 at 13:31, Neil C Smith <ne...@codelerity.com> wrote:
> I'm working on some more examples again.

And another - put together an example integrating a live pipeline and
hlssink2 with Javalin framework to demonstrate streaming to the
browser.

> Examples repo at -
>
> https://github.com/gstreamer-java/gst1-java-examples/

Again, feedback welcomed!

Thanks and best wishes,

[Scooter Willis]

Neil

Will find some time do some testing. I know when I tried to get something similar working on the Jetson for streaming via RTSP it appeared to be a harder than it should be related to others asking and NVIDIA providing a C application that took you out of the gstreamer pipeline. It would only work as a C executable.   https://forums.developer.nvidia.com/t/rtsp-rtmp-with-gstreamer-on-tx2/69941/7

Having a WebRTC solution will open up lots of interesting options. 

Would be helpful to post a youtube video of the process as a demo to help make it real for those evaluating the potential of gstreamer-java

Thanks

Scooter

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CAPxOS5E_GwnODszFxnKOubcaBDxGe4futb1OXFQ_eFOcHT5rXw%40mail.gmail.com.

[Neil C Smith]

On Tue, 22 Jun 2021 at 17:20, Scooter Willis <will...@gmail.com> wrote:
> Will find some time do some testing. I know when I tried to get something similar working on the Jetson for streaming via RTSP it appeared to be a harder than it should be related to others asking and NVIDIA providing a C application that took you out of the gstreamer pipeline. It would only work as a C executable. https://forums.developer.nvidia.com/t/rtsp-rtmp-with-gstreamer-on-tx2/69941/7
>
> Having a WebRTC solution will open up lots of interesting options.

Thanks. Yes, the RTSP option is harder - there's a separate library
which we don't have mappings for. We've had WebRTC in the bindings
for a while now, but no example except an incomplete Java one in the
upstream GStreamer repo, which this is derived from.

HLS is higher latency - in fact, the example is about 20s - different uses.

> Would be helpful to post a youtube video of the process as a demo to help make it real for those evaluating the potential of gstreamer-java

I'll look into it - a backlog list of videos to create at the moment!

In the meantime, some still images at -

https://twitter.com/neilcsmith_net/status/1402306161152516096/photo/1
https://twitter.com/neilcsmith_net/status/1405584079131754498/photo/1

Best wishes,