Having problems in screen capturing 3440x1440 resolution, other resolutions wokrs well.

[Davide Perini]

Hi all,
I'm screen capturing using GStreamer bindings for Java without problems
since a lot of time,
it works super awesome at all screen resolutions.

I'm using a pipeline like this:
d3d11screencapturesrc monitor-handle={0} ! d3d11convert ! d3d11download
! video/x-raw(memory:SystemMemory),width=3440,height=1440
this pipeline works as expected.

I recently switched to this "new pipeline" that should be more optimized
for the purpose:
d3d11screencapturesrc monitor-handle={0} ! d3d11convert !
video/x-raw(memory:D3D11Memory),width=3440,height=1440

this "new pipeline" works well at every screen resolutions except
3440x1440,
I even tried another 21:9 resolution like 3840x1080 and it works well at
2560x1080 but not on 3440x1440.

This is so weird. My code should be ok since if I switch to the "older
pipeline" all works well.

The author of the bindings is always right so the problem should be
found in my understanding of the bindings :D

Is there someone who can shed some light please?

Thanks
Davide

[Davide Perini]

I created a JPEG file from the captured image, I wrote the IntBuffer to a jpg file to see what it is capturing.

At a different resolution than 3440x1440 the resulted image is a perfect screenshot of my screen.

I attach the result at 3440x1440, it's a completely scrambled image. O_o

boolean write = false;
int skip = 0;

private class AppSinkListener implements AppSink.NEW_SAMPLE {

    public void rgbFrame(int width, int height, IntBuffer rgbBuffer) {


        // If the EDT is still copying data from the buffer, just drop this frame
        if (!bufferLock.tryLock()) {
            return;
        }

        skip++;

        BufferedImage img =
                new BufferedImage(3440, 1440, 1);
        int[] rgbArray = new int[rgbBuffer.capacity()];
        rgbBuffer.rewind();
        rgbBuffer.get(rgbArray);
        img.setRGB(0, 0, img.getWidth(), img.getHeight(), rgbArray, 0, img.getWidth());

        try {
	    // skip some frames
            if (!write && skip == 90) {
                write = true;
                ImageIO.write(img, "jpg", new File("screenshot.jpg"));
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

Il 05/03/2023 15:22, Davide Perini ha scritto:

Hi all,
I'm screen capturing using GStreamer bindings for Java without problems since a lot of time,
it works super awesome at all screen resolutions.

I'm using a pipeline like this:
d3d11screencapturesrc monitor-handle={0} ! d3d11convert ! d3d11download ! video/x-raw(memory:SystemMemory),width=3440,height=1440
this pipeline works as expected.

I recently switched to this "new pipeline" that should be more optimized for the purpose:
d3d11screencapturesrc monitor-handle={0} ! d3d11convert ! video/x-raw(memory:D3D11Memory),width=3440,height=1440

this "new pipeline" works well at every screen resolutions except 3440x1440,

I even tried another 21:9 resolution like 2560x1080 and it works well at 2560x1080 but not on 3440x1440.

[Christophe LAFOLET]

The problem seems to be related to padding 

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/8fe58dc4-ad16-094d-08fd-94bfe75e3c9a%40dpsoftware.org.

[Davide Perini]

Mmm... Searching the internet for GStreamer padding, I somewhat understood what it is but have no idea on how to solve the problem.

Is it something that can be done by adding params to my pipeline?

Thanks

Davide 

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/CABYxBqpW71rXKgq%2Bkx-yfuEMXcfD3%2Bj4_WJWf3X0%2BsENwaw4cA%40mail.gmail.com.

[Scooter Willis]

If you are doing screen capture make a black desktop(no icons) and a white square box in the middle. Would help you detect a pattern shift indicating a resolution problem in the file. 

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/186b3655118.2883.3c8a1e3a2388806d058c6e4a23c6ecab%40dpsoftware.org.

[Davide Perini]

ok, following your suggestion I can see that the image is sliced when using some resolutions like 3440x1440 or 480x720 on a pipeline like this:

./gst-launch-1.0 d3d11screencapturesrc ! d3d11convert ! "video/x-raw(memory:D3D11Memory),width=3440,height=1440,sync=false" ! autovideosink

a less optimized pipeline like this works well.

./gst-launch-1.0 d3d11screencapturesrc ! d3d11convert ! d3d11download ! "video/x-raw(memory:SystemMemory),width=3440,height=1440,sync=false" ! autovideosink

The weird thing is that using the D3D11Memory pipeline on AMD cards creates this problem on other resolutions too.

For example I can correctly capture at 480x720 with an NVIDIA card but not on an AMD card.

I think that this issue should be related to GStreamer Java because running the same pipeline using the gst-launch.1.0 command has no problems, 

no matter the resolution, no matter the GPU vendor.

I attach two images, a test image with a black background and a white rectangle, 

and the image that is captured when the problem occurs.

At 480x720 that image is sliced on an AMD card, same code, same settings works well on Nvidia.

Other resolutions like 3440x1440 creates the problem on both AMD and Nvidia.

Is there someone that can help me understanding what can be the cause of this problem?

I asked on the upstream channel and someone answered me:
"Is this resolution in need of a bit of alignment? The slice length must be

divisible by a number. some are 32 and you have to pad the buffer or it warps (or worse)"

The thing that confuses me is why 480x720 works well on Nvidia and not on AMD and why it works well on gst-launch-1.0 and not when using GStreamer Java...

[Christophe Lafolet]

with the image screenshot_KO (480x270), we can see that lines are translated around 30 RGB pixels, giving a size of 30x3=90 bytes

the real size of buffer would be 480x360

d3d11screencapturesrc is in BGRA

are you sure the alpha channel have been correctly removed ?

your old pipeline was OK with d3d11download ?

did you read  the source code of d3d11download ?

Christophe

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/c7de7227-b768-48ce-a3b4-fcfc35c79b95n%40googlegroups.com.

[Christophe Lafolet]

sorry, the real slice of buffer would be 270x3+90=900

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/78801434-21e9-5f9b-5258-9c7e1086a304%40laposte.net.

[Davide Perini]

Yes, I wrote 480x720 but I was referring to a 480x270 screen capture, sorry, my fault.
I'm using format=BGRx so this should be good.

how can I be sure that the alpha channel is removed and why should I remove it?

this pipeline works ok (the old one):

./gst-launch-1.0 d3d11screencapturesrc ! d3d11convert ! d3d11download ! "video/x-raw(memory:SystemMemory),width=3440,height=1440,sync=false" ! autovideosink

this one not:

./gst-launch-1.0 d3d11screencapturesrc ! d3d11convert ! "video/x-raw(memory:D3D11Memory),width=3440,height=1440,sync=false" ! autovideosink

no I haven't read the d3d11download source code :)

can you explain me what is this slice please?
270x3+90=900
how do you calculated it?
how can I use this number?

thank you very much.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/2adc2169-1158-882d-611a-a8daa58b87a0%40laposte.net.

[Davide Perini]

I add a question to the previous...
Is there a way to detect if the captured image needs "alignment" in order to switch to the less efficient pipeline that does not resize the image on the GPU?

Thanks
Davide

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/c6a8ce1d-6c0d-4663-beac-c5d8cc86ae92%40dpsoftware.org.

[Neil C Smith]

On Sun, 3 Dec 2023 at 15:30, Davide Perini <sblan...@gmail.com> wrote:
> I think that this issue should be related to GStreamer Java because running the same pipeline using the gst-launch.1.0 command has no problems,
> no matter the resolution, no matter the GPU vendor.

Are you using autovideosink or appsink with the GStreamer Java
pipeline? Using autovideosink should not differ when called from
Java. Please share exactly how you're configuring things?

On the other hand, appsink doesn't behave like autovideosink, so a
difference would be expected.

What exactly are you doing with the capture? What data do you want
and how are you processing / displaying it?

Best wishes,

Neil

--
Neil C Smith
Codelerity Ltd.
www.codelerity.com

Codelerity Ltd. is a company registered in England and Wales
Registered company number : 12063669
Registered office address : 3rd Floor Suite, 207 Regent Street,
London, W1B 3HH, England

[Davide Perini]

Hi Neil,

thank you very much for the time spent answering me, I appreciate it.

Sorry for the late replay, I don't know why but I lost this message and I haven't replied to it in time.

No,
I'm not using autovideosink nor appsink with the GStreamer Java.

Basically I capture the screen in an rgbBuffer thanks to your wonderful binding, 

then I use that rgbBuffer (IntBuffer) to send the color information of the screen to a led strip to act like an ambilight.

This is the final result just to give you the idea:

https://youtu.be/PO37JC7-Nn4?si=nuLmVAB1938WV4K5

This is the interesting code that launches the screen capture pipeline:

/**
* Launch Advanced screen grabber (DDUPL for Windows, ximagesrc for Linux)
*
* @param imageProcessor image processor utility
*/
public void launchAdvancedGrabber(ImageProcessor imageProcessor) {
  imageProcessor.initGStreamerLibraryPaths();
  Gst.init(Constants.SCREEN_GRABBER, "");
  AtomicInteger pipelineRetry = new AtomicInteger();
  pipelineRetry.getAndIncrement();
    log.info("Starting a new pipeline");
    GrabberSingleton.getInstance().pipe = new Pipeline();
    if (NativeExecutor.isWindows()) {
      DisplayManager displayManager = new DisplayManager();
      String monitorNativePeer = String.valueOf(displayManager.getDisplayInfo(MainSingleton.getInstance().config.getMonitorNumber()).getNativePeer());
      bin = Gst.parseBinFromDescription("d3d11screencapturesrc monitor-handle={0} ! d3d11convert\"".replace("{0}", monitorNativePeer), true);
    }
    vc = new GStreamerGrabber();
    GrabberSingleton.getInstance().pipe.addMany(bin, vc.getElement());
    Pipeline.linkMany(bin, vc.getElement());
    JFrame f = new JFrame(Constants.SCREEN_GRABBER);
    f.add(vc);
    vc.setPreferredSize(new Dimension(MainSingleton.getInstance().config.getScreenResX(), MainSingleton.getInstance().config.getScreenResY()));
    f.pack();
    f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    GrabberSingleton.getInstance().pipe.play();
    f.setVisible(false);
}

  

this is the GStreamerGrabber:

/**
* Creates a new instance of GstVideoComponent
*/
public GStreamerGrabber(AppSink appsink) {
  this.videosink = appsink;
  videosink.set(Constants.EMIT_SIGNALS, true);
  AppSinkListener listener = new AppSinkListener();
  videosink.connect(listener);
  String gstreamerPipeline;
  gstreamerPipeline = "video/x-raw(memory:D3D11Memory),width=3440,height=1440,sync=false,";
  gstreamerPipeline = setFramerate(gstreamerPipeline);
  StringBuilder caps = new StringBuilder(gstreamerPipeline);
  videosink.setCaps(new Caps(caps.toString()));
  setLayout(null);
  setOpaque(true);
  setBackground(Color.BLACK);
}

NewSample:

@Override
public FlowReturn newSample(AppSink elem) {
  Sample sample = elem.pullSample();
  Structure capsStruct = sample.getCaps().getStructure(0);
  int w = capsStruct.getInteger("width");
  int h = capsStruct.getInteger("height");
  Buffer buffer = sample.getBuffer();
  ByteBuffer bb = buffer.map(false);
  if (bb != null) {
    rgbFrame(w, h, bb.asIntBuffer());
    buffer.unmap();
  }
  sample.dispose();
  return FlowReturn.OK;
}

This is the callback where I process the captured image

private class AppSinkListener implements AppSink.NEW_SAMPLE {

  public void rgbFrame(int width, int height, IntBuffer rgbBuffer) {

    // If the EDT is still copying data from the buffer, just drop this frame
    if (!bufferLock.tryLock()) {
      return;
    }
 

    HERE IS WHERE I PROCESS the IntBuffer (rgbBuffer) that contains the problematic buffer

    
    bufferLock.unlock();
  }
}

All the code is open source in the Luciferin project here:

https://github.com/sblantipodi/firefly_luciferin

By further investigating the problem it seems that when the problem happen, the received IntBuffer is bigger than what I expect.

At 3440x1440 resolution for example, 

I would expect an IntBuffer with a capacity of 

3440 x 1440 = 4.953.600 

but I receive a capacity of 4976640

Other resolutions like 1920x1080 seems to works well on NVIDIA GPUs but not on AMD GPUs.

This is pretty weird...

I have tried to screen capture a full screen white image, I attach the result.

As you can see from that image, 16 black pixels are added every 3440 pixels. 

Considering the fact that we are capturing a 3440 x 1440 display resolution,

if you do simple math:
16 * 1440 you get 23040

23040 is exactly the difference between the expected capacity and the received capacity

4.976.640 -  4.953.600 = 23040

I opened an issue on the GStreamer upstream branch but they pointed out that it may be an issue on the bindings/app.

Thank you Neil for this super awesome binding.

Davide

[Christophe Lafolet]

hello Davide,

if you try to get pixels from a gst buffer, you need to know the stride / offset for each plane of this buffer (for RGB, only one plane)

These informations can be retrieve from the video meta when this meta is attached the the buffer

https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/gstvideometa.c

example

3440+16=3456 -> stride

3456*1440=4976640

Christophe

--

You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/0495bcc8-c477-4a9a-8a19-f6499de43b66n%40googlegroups.com.

[Davide Perini]

I discovered why I lost the Neil message before... GMAIL is putting the GStreamer Java mails in my SPAM folder, I don't know why Google should put in the SPAM folder a message from it's own Google Groups but...

Returning in topic.

Ok, now that I'm able to calculate the stride,

how can I use this infos to fix the screen capture problem using GStreamer Java?

I tried to do it "manually" with a dirty workaround like this that basically shifts the pixels by 16 for every row (every 3440 pixel):

IntBuffer rgbBuffer = IntBuffer.allocate(buffer.capacity());
for (int i=0; i<buffer.capacity(); i++) {
  rgbBuffer.put(buffer.get(i));
  if (i%3439 == 0) {
    i+= 16;
  }
}

The situation improved by a lot but there is still some "cut" in the final image.

I attach the completely scrambled image without my "workaround" (KO_gstreamer_screenshot.jpg)

and the same captured image with my workaround. (OK_gstreamer_screenshot.jpg)

Thanks
Davide

Image 

[Christophe Lafolet]

https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/video-frame.c

for (h = 0; h < height; ++h) {
        for (w = 0; w < width; ++w) {
          guint8 *pixel = pixels + h * stride + w * pixel_stride;
      }
 }

I suppose pixel_stride=3 for RGB (4 for RGBA)

Christophe

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/0512a7a6-f153-45b0-a39a-a48ec6011e73n%40googlegroups.com.

[Davide Perini]

Hi Christophe,
that made the trick, thank you very much, I really appreciate it.

It made the trick by using "manual math",
I'm not really good in using the Java binding.

What I'm not able to do is to retrieve the stride.

What I have from the lovely Java Binding is a callback like this:

private class AppSinkListener implements AppSink.NEW_SAMPLE {

    public void rgbFrame(int width, int height, IntBuffer rgbBuffer) {

with my "IntBuffer rgbBuffer" that contains all my RGBs pixels.

How can I get the stride inside that callback?
The only "useful" number that I'm able to retrieve in that callback is the capacity from the rgbBuffer.

From the capacity I can do some math to get the stride but I'm not sure if it's correct.

Suppose that we are using a 3440x1440 resolution, this is the calcultions I have done to retrieve the stride.

int capacity = rgbBuffer.capacity(); // at 3440x1440 resolution this number is 4.976.640 
final int exectedCapacityWithoutStride = 3440*1440; // this constant equals to 4.953.600
int tmp = capacity - exectedCapacityWithoutStride; // this equals to 23.040
int tmp2 = tmp / 1440; // that equals to 16
int stride = 3440 + tmp2; // that equals to your stride 3.456

is there a better or smarter way to get the stride using the Java Binding?

Thanks
Davide

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/dafc4bc7-ff42-9ff5-5c57-04d8c3382950%40laposte.net.

[Neil C Smith]

On Sat, 16 Dec 2023 at 13:08, Davide Perini <sblan...@gmail.com> wrote:
> No,
> I'm not using autovideosink nor appsink with the GStreamer Java.

Your code shows that GStreamerGrabber is using an AppSink.

Do you see any performance improvement from using D3D11 memory vs
system memory in your application (ie. without autovideosink)? I
suspect that you're trying to solve problems here that won't give you
any improvement.

If anything you might want to look at removing the memory location
from the caps completely. Move the caps to set width, height and
format between d3d11convert and d3d11download. Keep the appsink using
system memory.

[Davide Perini]

Hi Neil,
thanks, again, for the answer.

I tried some benchmarks that shows how the screen capture influences a
3D rendering
in terms of absolute performance and resource usage and no, there is
nearly no difference between
using D3D11 Memory vs System Memory.

I haven't understood why, isn't D3D11 Memory supposed to scale the image
in video memory over system memory?
Isn't supposed to be faster?

I think that I lack some basic GStreamer knowledge here, it's never late
to learn something new
but this is obviously something for another mailing list... :)

Thanks for your input Neil, I'll start from there to move forward.

Davide

[Christophe Lafolet]

To retrieve the stride with the video meta, you can use the following classes and buffer.getMeta(VideoMeta.API)

Don't forget to register VideoMeta

public interface GstVideoMetaAPI extends com.sun.jna.Library {
    
    GstVideoMetaAPI GSTVIDEO_API = GstNative.load("gstvideo", GstVideoMetaAPI.class);
    
    @Structure.FieldOrder({"meta", "buffer", "flags", "format", "id", "width", "height", "n_planes", "offset", "stride", "mapFn", "unmapFn", "alignment" })
    class GstVideoMetaStruct extends Structure {
        public volatile GstMetaAPI.GstMetaStruct.ByValue meta;
        public volatile Pointer buffer; // Pointer to GstBuffer
        public volatile int flags; // GstVideoFrameFlags
        public volatile int format;
        public int id;
        public int width;
        public int height;
        public int n_planes;
        public NativeLong[] offset = new NativeLong[4];
        public int[] stride = new int[4];
        public volatile Pointer mapFn; // map method
        public volatile Pointer unmapFn; // unmap method
        public volatile int alignment;
        
        public GstVideoMetaStruct() {
        }

        public GstVideoMetaStruct(Pointer p) {
            super(p);
            read();
        }
    }

}

public class VideoMeta extends Meta {
    
    private final GstVideoMetaStruct metaStruct;

    /**
     * Meta.API for VideoMeta.
     */
    public static final API<VideoMeta> API =
            new API<>(VideoMeta.class, "GstVideoMetaAPI");
    
    /**
     * Underlying GType name.
     */
    public static final String GTYPE_NAME = "GstVideoMeta";

    VideoMeta(Initializer init) {
        super(init);
        this.metaStruct = new GstVideoMetaStruct(init.ptr.getPointer());
    }

    public int getPlaneCount() {
        return this.metaStruct.n_planes;
    }
    
    /**
     * The plane line stride is the number of bytes between two consecutive lines in the buffer.
     * This number will vary depending on the frame's {@code VideoFormat} and decoder output.
     * @param planeIndex which plane to get the stride for, valid range is zero
     *            to {@link #getPlaneCount() getPlaneCount()} non-inclusive
     * @return the line stride for the specified plane
     */    
    public int getStrideForPlane(int planeIndex) {
        return this.metaStruct.stride[planeIndex];
    }

    /**
     * @param planeIndex which plane to get the offset for, valid range is zero
     *            to {@link #getPlaneCount() getPlaneCount()} non-inclusive
     * @return the line stride for the specified plane
     */
    public int getOffsetForPlane(int planeIndex) {
        return this.metaStruct.offset[planeIndex].intValue();
    }

}

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/e1535b8d-c356-4090-a2a3-7cf6b620dad1%40dpsoftware.org.

[Davide Perini]

Hi Christophe,
buffer.getMeta(VideoMeta.API).getStrideForPlane(0)
returns 0.

Does the problem depends on the fact that I haven't registered the VideoMeta?
What do you mean to register VideoMeta?
How can I register VideoMeta?

Thanks
Davide

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/cb85d888-a4b9-7949-d8a3-ecca5446524f%40laposte.net.

[christoph...@laposte.net]

There is an example here

https://github.com/gstreamer-java/gst1-java-core/blob/master/src/org/freedesktop/gstreamer/video/Video.java

Envoyé depuis mon téléphone

-------- Message original --------
De : Davide Perini <perini...@dpsoftware.org>
Date : mar. 19 déc. 2023 à 10:10
À : gstream...@googlegroups.com
Objet : Re: [gstreamer-java] Having problems in screen capturing 3440x1440 resolution, other resolutions wokrs well.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/de2fdeb0-40a7-42d5-a95f-3fd1b37dc5e7%40dpsoftware.org.

[Davide Perini]

Thanks

Stream.of(registration(VideoMeta.class, VideoMeta.GTYPE_NAME, VideoMeta::new));
buffer.getMeta(VideoMeta.API).getStrideForPlane(0)
buffer.getMeta(VideoMeta.API).getStrideForPlane(1)        

stride is still 0 for every plane.

To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/-2wx3ru-uhotvq-vdk0xycsel3d-8jqnp179webh-dv6m7jdst7qg-xphb95-i67me7nzvae29qukq8g0qqq9-ysiwma7bek2ecyip52isapqu-oeh934fgq7ml3qfznd95vcxwwjkd83-o5fvdx69teqi.1702988854411%40email.android.com.

[Davide Perini]

Following the Neil's suggestions fixed the problem completely,
no need to set the correct stride by following that suggestion...

Thank you all.
Davide

You received this message because you are subscribed to a topic in the Google Groups "gstreamer-java" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gstreamer-java/q0juwEUJiWc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gstreamer-jav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/2ba134fd-2609-43a6-a1df-e2126283a5df%40dpsoftware.org.