With GStreamer 0.10.30, red and blue are swapped

[Patrick Balleux]

Hi,

I'm using the RGB Sink to capture the output of any pipeline for
GStreamer. With GStreamer 0.10.28 and previous, the red and blue
color were in the right order. I updated to the latest version using
the Ubuntu PPA, and now, the red and blue are swapped in the
RGBDataSink.

I am still trying to figure out if this is a GStreamer bug or a
gstreamer-java bug.

If I capture my webcam from the console, red and blue are not
swapped. But if I do it using gstreamer-java, with the RGBDatasink as
the sink, the red and blue are swapped.

I can manually re-swapped the red and blue, but I would prefer finding
out what is the issue and how to fix it.

Here is what I'm doing in my code:

@Override
public void rgbFrame(int w, int h, java.nio.IntBuffer buffer) {
captureWidth = w;
captureHeight = h;
if (!isRendering) {
isRendering = true;
int[] array = buffer.array();
tempimage =
graphicConfiguration.createCompatibleImage(captureWidth,
captureHeight, java.awt.image.BufferedImage.TRANSLUCENT);
if (activeEffect.equals("vertigotv") ||
activeEffect.equals("shagadelictv")) {
for (int i = 0;i<array.length;i++){
array[i] = array[i] | 0xFF000000;
}
}
tempimage.setRGB(0, 0, captureWidth, captureHeight, array,
0, captureWidth);
applyFaceDetection(tempimage);
detectActivity(tempimage);
applyEffects(tempimage);
applyShape(tempimage);
image = tempimage;
isRendering = false;
}

}

[Patrick Balleux]

This is the line that is interesting, where I get the array of bytes from the buffer ...
int[] array = buffer.array();
... and then provide a bufferedimage with the data...

tempimage.setRGB(0, 0, captureWidth, captureHeight, array, 0, captureWidth);

2010/8/5 Patrick Balleux <patrick...@gmail.com>

--
Patrick Balleux

[Paul Heyniger]

Patrick,

I reverted back to 10.28 to solve this problem, but I just thought of an idea.  Have you tried to change

VideoComponent.java

The getBufferedImage method.

currentImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);

Change to

currentImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);

That will change the color representation in the BufferedImage to the correct color order.  Not sure if this will affect rendering the buffered image on Linux or not.

I will have to give this a try when I upgrade to the latest gstreamer again.

Paul

--
You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
To post to this group, send email to gstream...@googlegroups.com.
To unsubscribe from this group, send email to gstreamer-jav...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gstreamer-java?hl=en.

[Andy Savage]

Interestingly I have also found this problem to happen... but actually it only happens on a quirky inbuilt HP web camera. Using other web-camera's and I can't replicate the problem...

I am considering to code in a setting to my app so that the user can switch the colours if things look funny.

[Raimo Järvi]

I had the same problem, I solved it by setting masks in RGBDataSink
caps:

--- src/org/gstreamer/elements/RGBDataSink.java (revision 414)
+++ src/org/gstreamer/elements/RGBDataSink.java (working copy)
@@ -60,7 +60,7 @@
//
Element conv = ElementFactory.make("ffmpegcolorspace",
"ColorConverter");
Element videofilter = ElementFactory.make("capsfilter",
"ColorFilter");
- videofilter.setCaps(new Caps("video/x-raw-rgb, bpp=32,
depth=24"));
+ videofilter.setCaps(new Caps("video/x-raw-rgb, bpp=32,
depth=24, red_mask=65280, green_mask=16711680, blue_mask=-16777216"));
addMany(conv, videofilter, videosink);
Element.linkMany(conv, videofilter, videosink);

On 7 loka, 21:19, Patrick Balleux <patrick.ball...@gmail.com> wrote:
> This is the line that is interesting, where I get the array of bytes from
> the buffer ...
> int[] array = buffer.array();
> ... and then provide a bufferedimage with the data...
> tempimage.setRGB(0, 0, captureWidth, captureHeight, array, 0, captureWidth);
>

> 2010/8/5 Patrick Balleux <patrick.ball...@gmail.com>

[Farkas Levente]

anybody understand the reason for this bug?
it seems to me an upstream bug but i can't see the reason and why nobody
recognize it...

--
Levente "Si vis pacem para bellum!"

[Andy Savage]

I find it only happens on certain buggy cameras, it's not a universal bug.

2010/10/14 Farkas Levente <lfa...@lfarkas.org>

anybody understand the reason for this bug?
it seems to me an upstream bug but i can't see the reason and why nobody
recognize it...

[Raimo Järvi]

I don't know the reason, but I compared caps of video buffers in
GStreamer 0.10.27 (that was working) and 0.10.30 (where colors were
swapped). In 0.10.27 mask values were same as in my patch, in 0.10.30
they were swapped for red and blue. My guess would be that you need to
specify masks in the caps to make sure colors are in the order that
Java expects.

[Patrick Balleux]

That's what I tought also. With the latest Ubuntu, the problem is
there also.

I'll try some stuff, but it is sad that something broke in the
GStreamer upgrade.

I'll let you know my findings

PS: Great job on gstreamer-java, WebcamStudio could not be if
gstreamer-java wasn't there. Thanks!
Pat

[Patrick Balleux]

Hi,

I applied the caps filter in my code, and your solution does work
well. But it seems as the RGBSink is considered as BRG instead of RGB
in the pipeline. I had to put the caps filter at the end to ensure
that the red and blue were in the correct order, just before the
RGBSink.

Tried by using a BufferedImage Type.BGR, and all variant, nothing
would work.

The the problem seems to be in the caps of the RGBsink and the new
GStreamer does not seem to know about it. One thing I saw is that the
output of the virtual webcam in WebcamStudio, which is in RGB format,
gstreamer will swap red and blue but Flash will not. If I put the
pixel format in UYVY, then the red and blue are not swapped by
gstreamer. Looks like the gstreamer folks are not doing something
right with RGB pixel format.

[Raimo Järvi]

Hi,

I guess it's BGR because JNA creates ByteBuffer using native byte
order and incoming video buffers are big endian. Which also means that
my patch probably wouldn't work on big endian. This might work (I
haven't tested on big endian), alternatively you could set ByteBuffer
order to big endian in VideoHandoffListener:

--- src/gstreamer-java/src/org/gstreamer/elements/RGBDataSink.java
(revision 424)
+++ src/gstreamer-java/src/org/gstreamer/elements/RGBDataSink.java
(working copy)
@@ -18,6 +18,7 @@

package org.gstreamer.elements;

+import java.nio.ByteOrder;
import java.nio.IntBuffer;

import org.gstreamer.Bin;
@@ -60,7 +61,16 @@

//
Element conv = ElementFactory.make("ffmpegcolorspace",
"ColorConverter");
Element videofilter = ElementFactory.make("capsfilter",
"ColorFilter");
- videofilter.setCaps(new Caps("video/x-raw-rgb, bpp=32,
depth=24"));

+ // JNA creates ByteBuffer using native byte order, set masks
according to that.
+ if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {

+ videofilter.setCaps(new Caps("video/x-raw-rgb, bpp=32,

depth=24, endianness=(int)4321," +
+ " red_mask=(int)0xFF00, green_mask=(int)0xFF0000,
blue_mask=(int)0xFF000000"));
+ }
+ else {

+ videofilter.setCaps(new Caps("video/x-raw-rgb, bpp=32,

depth=24, endianness=(int)4321," +
+ " red_mask=(int)0xFF0000, green_mask=(int)0xFF00,
blue_mask=(int)0xFF"));
+ }
+

addMany(conv, videofilter, videosink);
Element.linkMany(conv, videofilter, videosink);

[Farkas Levente]

are you sure it solve the problem and that's was the reason?
than i commit your patch.

[Raimo Järvi]

I know for sure that it solves the problem for me, but I've only
tested it with the Swing VideoComponent.

[Patrick Balleux]

I'm not that sure that Big Endian is the solution. Remember that you
also have the Alpha channel (ARGB) coming from the buffer. In my
testings, it was clear that what I get is ABGR. If Big Endian was the
issue then we would get (BGRA), and that's not the case. (Or on 16
bits level RABG)

Of course, there is the possibility that the Alpha channel is added by
the RGBSink on an RGB channels (Without the Alpha)

My 2 cents

[Andy Savage]

Hi all,

Here is something I found with this issue... 

It only happens on a laptop camera which seems a bit buggy. On my normal PC it works fine and indeed it seems to appear on a few other cameras.

But it's consistent, the camera either works fine, or it has a weird red & blue swapped problem.

Kind regards,

Andy Savage

[Patrick Balleux]

It can be because on other PC, you have different V4L(2) version. It
does the problem also with some videos and lately I made a pipeline to
read a stream of JPEG, and the red and blue were swapped anyway,

On 15 nov, 23:50, Andy Savage <a...@savage.hk> wrote:
> Hi all,
>
> Here is something I found with this issue...
>
> It only happens on a laptop camera which seems a bit buggy. On my normal PC
> it works fine and indeed it seems to appear on a few other cameras.
>
> But it's consistent, the camera either works fine, or it has a weird red &
> blue swapped problem.
>
> Kind regards,
> Andy Savage
>
> On Sat, Oct 23, 2010 at 9:51 AM, Patrick Balleux

> <patrick.ball...@gmail.com>wrote:

> > gstreamer-jav...@googlegroups.com<gstreamer-java%2Bunsu...@googlegroups.com>
> > .

[beemaster]

This way works for me for the following pipline (showing stream from IP-camera):

String pipelineDescription = String.format("rtspsrc location=%s latency=0 ! rtpjpegdepay ! jpegdec ! ffmpegcolorspace ! capsfilter name=filter", url);

But it looks rather as workaround than general solution. But is there solution?