Cast CVImageBuffer to CVPixelBuffer?

[Dzung Pham]

In Object-c it can easy to cast CVImageBufferRef to CVPixelBufferRef like this code:

CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);

But in java i use this code:

CVPixelBuffer cvimgRef = (CVPixelBuffer) sampleBuffer.getImageBuffer();

sampleBuffer from camera:

public void didOutputSampleBuffer(AVCaptureOutput captureOutput,
            CMSampleBuffer sampleBuffer, AVCaptureConnection connection) {}

And it failed to cast.

[Dzung Pham]

In swift someone face the same problem: http://stackoverflow.com/questions/24390190/swift-how-do-you-cast-a-cvimagebufferref-as-a-cvpixelbufferref.

They can solve it but i can not find a way to translate to java...

[blueri...@gmail.com]

I think you can try to convert the type with the .as(Class<?>) method.

For example:

CVPixelBuffer pixelBuffer =  imageBuffer.as(CVPixelBuffer.class);

[Dzung Pham]

I have try this before both with:

        CVPixelBuffer cvimgRef = imageBuffer.as(CVPixelBuffer.class);
        or CVPixelBuffer cvimgRef = imageBuffer.as(CVPixelBufferPtr.class).get();

But still error:

java.lang.Error: java.lang.reflect.InvocationTargetException
    at org.robovm.rt.bro.MarshalerLookup.toObject(MarshalerLookup.java:47)
    at org.robovm.rt.bro.NativeObject.as(NativeObject.java:53)
    at com.cameratest.MyViewController.didOutputSampleBuffer(MyViewController.java:145)
    at org.robovm.apple.avfoundation.AVCaptureVideoDataOutputSampleBufferDelegate$ObjCProxy.$cb$captureOutput$didOutputSampleBuffer$fromConnection$(Unknown Source)
Caused by: java.lang.reflect.InvocationTargetException
    at java.lang.reflect.Method.internalInvoke(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:591)
    at org.robovm.rt.bro.MarshalerLookup.toObject(MarshalerLookup.java:45)
    ... 3 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.robovm.apple.corevideo.CVPixelBuffer
    at org.robovm.rt.VM.allocateObject(Native Method)
    at org.robovm.rt.bro.NativeObject$Marshaler.toObject(NativeObject.java:88)
    at org.robovm.apple.corefoundation.CFType$Marshaler.toObject(CFType.java:62)
    at org.robovm.apple.corefoundation.CFType$Marshaler.toObject(CFType.java:51)
    ... 6 more
Unhandled exception (probably in a @Callback method called from native code): java/lang/Error

[Dzung Pham]

I will add more info so anyone can help me solve this.
In those code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection 
{

    CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);

    CVPixelBufferLockBaseAddress(cvimgRef, 0);
    size_t _width = CVPixelBufferGetWidth(cvimgRef);
    size_t _height = CVPixelBufferGetHeight(cvimgRef);
....
}

CMSampleBufferGetImageBuffer() returns a CVImageBuffer!. And CVPixelBufferLockBaseAddress() takes a CVPixelBuffer! as its argument. In C/ObjC, CVPixelBufferRef is typedef'd to CVImageBufferRef. But in robovm binding CVPixelBuffer extend  CVImageBuffer and there is no CVImageBufferRef and CVPixelBufferRef . 

[Niklas Therning]

Please provide us with a complete test case. Something we can debug. I suspect a problem with the static initializer of CVPixelBuffer...

--
You received this message because you are subscribed to the Google Groups "RoboVM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robovm+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dzung Pham]

Hi Niklas,

This is project you can check: https://www.dropbox.com/s/5ftn2e70z3c7uaj/testcamera.zip
Best option is debug on ios device with flash, app will turn on the camera and flash.
I also try write use static lib and call from java but without success.

Object-C:

+(CVPixelBufferRef) getPixelBuffer:(CMSampleBufferRef) value{
    CVPixelBufferRef cvimgRef = CMSampleBufferGetImageBuffer(value);
    return cvimgRef;
}

Binding:

    @Method(selector = "getPixelBuffer:")
    public static native CVPixelBuffer getPixelBuffer(CMSampleBuffer value);

[Kirill Prazdnikov]

You should directly bind CMSampleBufferGetImageBuffer as a "C" function

[Dzung Pham]

Yes, I do but without success, you can see above post

[Kirill Prazdnikov]

CVImageBuffer and CMSampleBuffer are different classes. 

You can not cast them. 

Which Object-C code are you trying to port to java ?

[Dzung Pham]

Java project, you can check it here:

https://www.dropbox.com/s/5ftn2e70z3c7uaj/testcamera.zip

Object-C code i need convert:
// process the frame of video


- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {


    // if we're paused don't do anything
    if(self.currentState==STATE_PAUSED) {
        // reset our frame counter
        self.validFrameCounter=0;
        return;
    }
    // this is the image buffer
    CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);
    //CVPixelBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);
    // Lock the image buffer
    CVPixelBufferLockBaseAddress(cvimgRef,0);
    // access the data
    size_t width=CVPixelBufferGetWidth(cvimgRef);
    size_t height=CVPixelBufferGetHeight(cvimgRef);
    // get the raw image bytes
    uint8_t *buf=(uint8_t *) CVPixelBufferGetBaseAddress(cvimgRef);
    size_t bprow=CVPixelBufferGetBytesPerRow(cvimgRef);
    // and pull out the average rgb value of the frame
    float r=0,g=0,b=0;
    for(int y=0; y<height; y++) {
        for(int x=0; x<width*4; x+=4) {
            b+=buf[x];
            g+=buf[x+1];
            r+=buf[x+2];
        }
        buf+=bprow;
    }
    r/=255*(float) (width*height);
    g/=255*(float) (width*height);
    b/=255*(float) (width*height);
//other code
}


[Kirill Prazdnikov]

I do not see any requirement to cast CVImageBufferRef to CVPixelBufferRef in this code. 

(Which is what you are asking in this thread, aren`t you? )

[Dzung Pham]

Because you don't read all the thread.
I add this here:

I will add more info so anyone can help me solve this.
In those code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection 
{

    CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);

    CVPixelBufferLockBaseAddress(cvimgRef, 0);
    size_t _width = CVPixelBufferGetWidth(cvimgRef);
    size_t _height = CVPixelBufferGetHeight(cvimgRef);
....
}

CMSampleBufferGetImageBuffer() returns a CVImageBuffer!. And CVPixelBufferLockBaseAddress() takes a CVPixelBuffer! as its argument. In C/ObjC, CVPixelBufferRef is typedef'd to CVImageBufferRef. But in robovm binding CVPixelBuffer extend  CVImageBuffer and there is no CVImageBufferRef and CVPixelBufferRef . 

[Kirill Prazdnikov]

Ok. I see, thanks. 

One of the option is to bind lockBaseAddress differently. With different argument.

[Kirill Prazdnikov]

You can bind one function as many times as you want. with different arguments types and with different return type. 

Something like

@Pointer long CVPixelBufferGetBaseAddress(CMSampleBuffer b);

@Pointer long CVPixelBufferGetBaseAddress(CVPixelBuffer b);

[Zhan Liang]

Hi,

Have you guys had any luck in converting the Objective C code to swift. I am having issues with this part of the code

  buf.append((CVPixelBufferGetBaseAddress(cvimgRef)?.load(as: UInt8.self))!)

        var bprow = CVPixelBufferGetBytesPerRow(cvimgRef)

        // and pull out the average rgb value of the frame

        var r = UInt8()

        var g = UInt8()

        var b = UInt8()

        for y in 0..<height {

            var x = 0

            while x < width * 4 {

                b += buf[x]

                g += buf[x + 1]

                r += buf[x + 2]

I am always getting index out of range and buf[0] = 0. Am I doing something wrong here?