GStreamer java and low performance on screen capture

[Davide Perini]

Hi all GStreamer Java fans,
before this merge request
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1855

GStreamer was a bit slow in screen capturing at 4K when there is some heavy content on the GPU like a videogame or a 3D Demo.

That merge request completely solved the GStreamer slowdown.

Unfortunantly GStreamer Java seems to not benefit from those performance improvements at its terribly slow.
I have a RTX2080 Ti GPU with an Intel i9 and I still capture at 6FPS when some 3D content is running on the GPU along with the screen capture.

This video shows a really weird behaviour:
https://www.youtube.com/watch?v=i6nUnS5Ap9g&feature=youtu.be

If you see the video, I have a 3D demo running in a small window (the pendulum nvidia demo)
and IntelliJ is running GStreamer Java screen capturing at 30 FPS.

As you can see, if I focus IntelliJ by clicking its windows, it captures at 30FPS,
as soon as I focus the 3D demo window by clicking it, the screen capture drops at 6FPS.

I hoped that a real DXGI implementation is GStreamer would have solved this problem and infact it solved it,
GStreamer is super fast after that merge request but the problem is that GStreamer Java is super slow.

This is the code I'm using

GStreamerGrabber vc = new GStreamerGrabber();
Bin bin;

    bin = Gst.parseBinFromDescription( "d3d11desktopdupsrc ! queue max-size-buffers=1",true);

pipe = new Pipeline();
pipe.addMany(bin, vc.getElement());
Pipeline.linkMany(bin, vc.getElement());
JFrame f = new JFrame(Constants.SCREEN_GRABBER);
f.add(vc);
vc.setPreferredSize(new Dimension((int)screenSize.getWidth(), (int)screenSize.getHeight()));
f.pack();
f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
pipe.play();
f.setVisible(false);

public GStreamerGrabber(AppSink appsink) {

    this.videosink = appsink;
    videosink.set(Constants.EMIT_SIGNALS, true);
    AppSinkListener listener = new AppSinkListener();
    videosink.connect(listener);
    String gstreamerPipeline = "video/x-raw(memory:D3D11Memory),";
        gstreamerPipeline += framerate=30/1,";
    StringBuilder caps = new StringBuilder(gstreamerPipeline);
    videosink.setCaps(new Caps(caps.toString()));
    setLayout(null);
    setOpaque(true);
    setBackground(Color.BLACK);

}

/**
 * Listener callback triggered every frame
 */
private class AppSinkListener implements AppSink.NEW_SAMPLE {

    public void rgbFrame(int width, int height, IntBuffer rgbBuffer) {

        // If the EDT is still copying data from the buffer, just drop this frame
        if (!bufferLock.tryLock()) {
            return;
        }

        int intBufferSize = (width*height)-1;

        try {
            Color[] leds = new Color[ledMatrix.size()];
            // We need an ordered collection so no parallelStream here
            ledMatrix.forEach((key, value) -> {
                int r = 0, g = 0, b = 0;
                int skipPixel = 5;
                // 6 pixel for X axis and 6 pixel for Y axis
                int pixelToUse = 6;
                int pickNumber = 0;
                int xCoordinate = value.getX();
                int yCoordinate = value.getY();
                // We start with a negative offset
                for (int x = 0; x < pixelToUse; x++) {
                    for (int y = 0; y < pixelToUse; y++) {
                        int offsetX = (xCoordinate + (skipPixel * x));
                        int offsetY = (yCoordinate + (skipPixel * y));
                        int bufferOffset = (Math.min(offsetX, width))
                                + ((offsetY < height) ? (offsetY * width) : (height * width));
                        int rgb = rgbBuffer.get(Math.min(intBufferSize, bufferOffset));
                        r += rgb >> 16 & 0xFF;
                        g += rgb >> 8 & 0xFF;
                        b += rgb & 0xFF;
                        pickNumber++;
                    }
                }
                r = ImageProcessor.gammaCorrection(r / pickNumber);
                g = ImageProcessor.gammaCorrection(g / pickNumber);
                b = ImageProcessor.gammaCorrection(b / pickNumber);
                if (FireflyLuciferin.config.isEyeCare() && (r+g+b) < 10) r = g = b = 5;
                leds[key - 1] = new Color(r, g, b);
            });

            // Put the image in the queue
            FireflyLuciferin.sharedQueue.offer(leds);

            // Increase the FPS counter
            FireflyLuciferin.FPS_PRODUCER_COUNTER++;

        } finally {
            bufferLock.unlock();
        }
    }

Can I do some optimizations?

I can't believe that I have such a huge performance drop.

The strange things is that this performance drop does not happen on GSTreamer only.

Any idea on how to improve the situation? Is there some room for improvement?

Thanks
Davide

[Davide Perini]

Hi,
I answer my self.

Thanks to Seungha Yang and his wonderful merge request it works flawlessly now,
all I need to do to avoid the huge performance hit is to disable Windows 10 GPU Scheduling.

Don't know why Windows GPU Scheduling affect screen capture performance so badly but this is all I have done to solve the problem.

Thank you guys.
Davide

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gstreamer-jav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gstreamer-java/04ef5a27-3500-88d7-feaf-dbd00644b81c%40dpsoftware.org.

[Neil C Smith]

On Sat, 19 Dec 2020 at 23:53, Davide Perini
<perini...@dpsoftware.org> wrote:
> I hoped that a real DXGI implementation is GStreamer would have solved this problem and infact it solved it,
> GStreamer is super fast after that merge request but the problem is that GStreamer Java is super slow.

How about figuring out what the issue is before posting blanket, and
provably wrong, assertions?!

Aside from the GPU scheduling issue, which isn't that surprising, you
need to understand the difference between capture straight to the
standard GStreamer video sink (likely all on GPU), and your use of
AppSink (capture, download to CPU memory, then JVM memory, and bit
manipulation). Your code will be significantly more CPU intensive in
that case.

You could try investigating other GStreamer elements to see if you can
get the scaling down and/or colour correction done on the GStreamer
side, ideally on the GPU, so you have less to do inside the AppSink
callback.

Even swapping the for loops (y outside x) may give you a minimal improvement.

Best wishes,

Neil


--
Neil C Smith
Codelerity Ltd.
www.codelerity.com

Codelerity Ltd. is a company registered in England and Wales
Registered company number : 12063669
Registered office address : Office 4 219 Kensington High Street,
Kensington, London, England, W8 6BD

[Davide Perini]

You're right, I have many steps more than the standard GStreamer video sink.

I have tested GPU scheduling ON and OFF before with the old
dxgiscreencapsrc and nothing changed
from ON to OFF.
This is why I haven't tested it before posting.
Newer d3d11desktopdupsrc is heavily affected by GPU scheduling.

Sincerely I still don't understand why GPU Scheduling should be a
problem when screen capturing,
why you think that "isn't surprising"?

Apart this, thanks for the suggestion Neil and sorry if I said "that
GStreamer Java" is slow,
it was not my intention to say something bad to the project :)

Thank you
Davide

[Neil C Smith]

On Sun, 20 Dec 2020 at 18:21, Davide Perini
<perini...@dpsoftware.org> wrote:
> Sincerely I still don't understand why GPU Scheduling should be a
> problem when screen capturing,
> why you think that "isn't surprising"?

Actually, I'm mixing it up with another Windows GPU-related feature I
think - don't use Windows a lot. However, pages like this also point
towards similar problems with screen capture -
https://obsproject.com/wiki/How-to-disable-Windows-10-Hardware-GPU-Scheduler

[Davide Perini]

Thank you!!!

Best wishes,
Davide