Resizing an Image efficiently using ByteBuffer
Jay Nair
be resized. The inefficient approach I came up was like this.
public Buffer resize(ByteBuffer b, Dimension d) {
BufferedImage bi = createBufferedImage(b);
IplImage image = IplImage.createFrom(bi);
IplImage out=IplImage.create(d.width,d.height, image.depth(),
image.nChannels());
cvResize(image, out, CV_INTER_AREA);
BufferedImage outBI = out.getBufferedImage();
ByteArrayOutputStream boas = new ByteArrayOutputStream();
ImageIO.write(outBI, "png", boas);
byte[] bytesOut = boas.toByteArray();
image.release();
out.release();
return ByteBuffer.wrap(bytesOut);
}
The problem I see is that many copies of the image bytes exist in this
operation (b, bi, image, out, outBI, and the final Buffer). Is there a
way I can avoid using BufferedImage and pass the bytes from the
ByteBuffer and get the resized image in the same ByteBuffer. I
searched the archives and could not find the way to do this.
Thanks,
Samuel Audet
If your ByteBuffer.isDirect(), we can do things like
IplImage.imageData(new BytePointer(b)) on an object create with
IplImage.createHeader(). We would also get a direct buffer by calling
IplImage.getByteBuffer() on the output object.
And if you are asking whether cvResize() supports inplace resizing, the
answer is no. We need two buffers, but we can cache references somewhere
to avoid allocating them on every call. Sounds good?
Samuel
Jayakrishnan Nair
Samuel Audet
> Samuel,
>
> Thanks for pointing out the imageData method. Now I saved some memory by
> not creating the BufferedImage. But now this creates another problem. I
> get the input ByteBuffer over the network and don't know the width and
> height or depth of the image. In such a case what values do I pass for
> the createHeader()?
How can we resize an image whose size we do not know?
> Regarding caching, on the input side I can keep a pool of ByteBuffers so
> that they are reused and not created for each call.
ThreadLocal references would probably be more efficient...
Samuel
Jayakrishnan Nair
On 2011-11-11 15:39, Jayakrishnan Nair wrote:How can we resize an image whose size we do not know?
Samuel,
Thanks for pointing out the imageData method. Now I saved some memory by
not creating the BufferedImage. But now this creates another problem. I
get the input ByteBuffer over the network and don't know the width and
height or depth of the image. In such a case what values do I pass for
the createHeader()?
Samuel Audet
Hello,
Can you also make sure that IplImage.imageSize() == ByteBuffer.capacity()?
Samuel
Jayakrishnan Nair
Samuel Audet
> I can't explain this. I added some debug to see what is happening.
>
> The code has been simplified like this
>
> Capacity: "+b.capacity());
> IplImage originalImage = IplImage.createHeader(1024, 768, 8, 3);
> originalImage.imageData(new BytePointer(b));
> logger.info
>
> and this is the logger output
>
> INFO: Is Direct: true Buffer Capacity: 492777
> INFO: 2359296,492777
>
> The IplImage size is around 4.7 times the ByteBuffer.capacity. The
> image size on the disk is exactly the Buffer capacity.
Well then, it looks like 1024x768x3 is not the correct size of the
original image..
Samuel
Jayakrishnan Nair
IplImage image = cvLoadImage(f.getCanonicalPath());
sb.append(image.width());
sb.append(COMMA);
sb.append(image.height());
sb.append(COMMA);
sb.append(image.depth());
sb.append(COMMA);
sb.append(image.nChannels());
sb.append(COMMA);
sb.append(image.nSize());
sb.append("\n");
logger.info(sb.toString());
and the output is the following
INFO: 1.jpg,1024,768,8,3,112
So the size is 1024x768x3
I ran the code on a 32-bit Ubuntu machine and when using the Direct buffer method, I am getting the same error I saw in Windows 64 bit. In fact the buffer differences match exactly.
INFO: 2359296,492777 (the second value is the actual file size on disk).
Now I am able to resize the image, if I don't use direct memory and create the IplImage from a BufferedImage.
Is there anything else I can check to see why the direct memory method does not work.
Jayakrishnan Nair
private ByteBuffer readFileAsByteBuffer(String inputFile, boolean directMemory)
throws FileNotFoundException, IOException {
FileChannel fc = new FileInputStream(inputFile).getChannel();
long l = fc.size();
logger.info("File Size = "+l);
ByteBuffer bb = null;
if (directMemory) {
bb = ByteBuffer.allocateDirect((int)l);
logger.info("Direct Memory "+bb.capacity()+" File Size: "+l);
} else {
logger.info("JVM Memory");
bb = ByteBuffer.allocate((int) l);
}
int read = fc.read(bb);
fc.close();
logger.info("Read = "+read);
return bb;
}
Samuel Audet
from memory. Please refer to this thread for some info:
http://groups.google.com/group/javacv/browse_thread/thread/452eb41bbd8dc77b/7190187aa28f9256
Samuel
On 2011-11-14 09:07, Jayakrishnan Nair wrote:
> *Update*: it fails with the same error and the same size differences on
> a 64 bit Mac as well. Looks like I am missing something which is causing
> this. This is the code I use to read the file and create the direct
> buffer. I call it with directMemory set to true.
>
>
>
> private ByteBuffer readFileAsByteBuffer(String inputFile, boolean
> directMemory)
>
> throws FileNotFoundException, IOException {
>
> FileChannel fc = new FileInputStream(inputFile).getChannel();
>
> long l = fc.size();
>
> logger.info("File Size = "+l);
>
> ByteBuffer bb = null;
>
> if (directMemory) {
>
> bb = ByteBuffer.allocateDirect((int)l);
>
> logger.info("Direct Memory "+bb.capacity()+" File Size: "+l);
>
> } else {
>
> logger.info("JVM Memory");
>
> bb = ByteBuffer.allocate((int) l);
>
> }
>
> int read = fc.read(bb);
>
> fc.close();
>
> logger.info("Read = "+read);
>
> return bb;
>
> }
>
>
> On Sun, Nov 13, 2011 at 9:20 AM, Jayakrishnan Nair
> <jayakrish...@gmail.com <mailto:jayakrish...@gmail.com>>
> wrote:
>
> To verify this I did the following
>
> IplImage image = cvLoadImage(f.getCanonicalPath());
> sb.append(image.width());
> sb.append(COMMA);
> sb.append(image.height());
> sb.append(COMMA);
> sb.append(image.depth());
> sb.append(COMMA);
> sb.append(image.nChannels());
> sb.append(COMMA);
> sb.append(image.nSize());
> sb.append("\n");
> logger.info <http://logger.info>(sb.toString());
>
> and the output is the following
>
> INFO: 1.jpg,1024,768,8,3,112
>
> So the size is 1024x768x3
>
> I ran the code on a 32-bit Ubuntu machine and when using the Direct
> buffer method, I am getting the same error I saw in Windows 64 bit.
> In fact the buffer differences match exactly.
>
> INFO: 2359296,492777 (the second value is the actual file size on disk).
>
> Now I am able to resize the image, if I don't use direct memory and
> create the IplImage from a BufferedImage.
>
> Is there anything else I can check to see why the direct memory
> method does not work.
>
>
>
>
>
>
>
> On Sun, Nov 13, 2011 at 2:32 AM, Samuel Audet
> <samuel...@gmail.com <mailto:samuel...@gmail.com>> wrote:
>
> On 2011-11-13 02:49, Jayakrishnan Nair wrote:
>
> I can't explain this. I added some debug to see what is
> happening.
>
> The code has been simplified like this
>
> logger.info <http://logger.info> <http://logger.info>("Is
> Direct: "+b.isDirect()+" Buffer
>
> Capacity: "+b.capacity());
> IplImage originalImage = IplImage.createHeader(1024, 768, 8, 3);
> originalImage.imageData(new BytePointer(b));
> logger.info <http://logger.info>
> <http://logger.info>(""+__originalImage.imageSize()+","+__b.capacity());
Jayakrishnan Nair
Jayakrishnan Nair
Samuel Audet
Just use the type of the original image, i.e.: originalImage.depth() and originalImage.nChannels()
If originalImage == null then OpenCV does not support the format. Make sure b.position() == 0 though
Samuel
Jayakrishnan Nair
Samuel Audet
and b.capacity() corresponds to the number of elements, and you even get
get an IplImage object from cvDecodeImage(). So, what exactly is still
not working?
On 2011-11-15 11:04, Jayakrishnan Nair wrote:
> 1. b.rewind() did the trick. The resize is now happening. Whew!
>
> 2. How do I derive type information from depth and channels? The
> argument to cvMat is (int type). Could you give an example?
>
>
>
> On Mon, Nov 14, 2011 at 4:18 PM, Samuel Audet <samuel...@gmail.com
> <mailto:samuel...@gmail.com>> wrote:
>
> Just use the type of the original image, i.e.: originalImage.depth()
> and originalImage.nChannels()
>
> If originalImage == null then OpenCV does not support the format.
> Make sure b.position() == 0 though
>
> Samuel
>
> 2011/11/15 6:11 "Jayakrishnan Nair" <jayakrish...@gmail.com
> <mailto:jayakrish...@gmail.com>>:
>
> To add some more information
>
> private ByteBuffer resizeDirectMemory(ByteBuffer b, Dimension d)
> throws IOException {
> logger.info <http://logger.info>("Is Direct: "+b.isDirect()+"
> Buffer Capacity: "+b.capacity());
Jayakrishnan Nair
CV_8UC1 is fine, that corresponds to the element type of the ByteBuffer, and b.capacity() corresponds to the number of elements, and you even get get an IplImage object from cvDecodeImage(). So, what exactly is still not working?
On 2011-11-15 11:04, Jayakrishnan Nair wrote:
1. b.rewind() did the trick. The resize is now happening. Whew!
2. How do I derive type information from depth and channels? The
argument to cvMat is (int type). Could you give an example?
On Mon, Nov 14, 2011 at 4:18 PM, Samuel Audet <samuel...@gmail.com<mailto:samuel...@gmail.com>> wrote:
Just use the type of the original image, i.e.: originalImage.depth()
and originalImage.nChannels()
If originalImage == null then OpenCV does not support the format.
Make sure b.position() == 0 though
Samuel
2011/11/15 6:11 "Jayakrishnan Nair" <jayakrish...@gmail.com
<mailto:jayakrishnan.b.nair@gmail.com>>:
Samuel Audet
> I got the resize working for jpg images, but I wanted to clarify few
> things with you.
>
> 1. The input to my resize is a direct ByteBuffer and this is what I do
> for resize
>
> resize(ByteBuffer b, Dimension d) {
> CvMat mat = cvMat(1, b.capacity(), CV_8UC1, new
> BytePointer(b));
> IplImage originalImage = cvDecodeImage(mat);
> IplImage resizedImage = IplImage.create(d.width, d.height,
> originalImage.depth(),originalImage.nChannels());
> cvResize(originalImage, resizedImage, CV_INTER_AREA);
> }
>
> For now I hardcoded CV_8UC1 in cvMat() which I think is not right. So my
> question is what is the right way to get that value.
Yes, this is correct, that is the right way.
> 2. From the resizeImage, I am able to getBufferedImage() and render it.
> Works fine. But when I do the following the following, the resulting
> file size looks right, but none of the image viewers are able to display it.
>
> ByteBuffer rb = resizedImage.getByteBuffer();
> rb.rewind();
> byte[] a = new byte[rb.capacity()];
> rb.get(a);
> IOUtils.write(a, new FileOutputStream("c:\\tmp\\out\\1.jpg"));
>
> Do I have to do some processing on the bytes in 'rb' to convert it to an
> image?
Back to a JPG or something file? cvSaveImage()
Or back to a JPG-encoded memory buffer? There is also cvEncodeImage()
for that...
> 3. In (2) I noticed that 'rb' is a direct ByteBuffer and I am curious to
> know how memory allocated in C++ is passed back to Java. I looked for
> the JNI code for this and could not find it. Is the source for your JNI
> library opensourced?
It is passed back via JNI, whose code is automatically generated by
JavaCPP on build. Just run `ant -Djavacpp.options=-cpp` on JavaCV's
source code to have it leave the .cpp files for your inspection
Samuel
Jayakrishnan Nair
It is passed back via JNI, whose code is automatically generated by JavaCPP on build. Just run `ant -Djavacpp.options=-cpp` on JavaCV's source code to have it leave the .cpp files for your inspection
3. In (2) I noticed that 'rb' is a direct ByteBuffer and I am curious to
know how memory allocated in C++ is passed back to Java. I looked for
the JNI code for this and could not find it. Is the source for your JNI
library opensourced?
Samuel
Jayakrishnan Nair
Samuel Audet
> It is passed back via JNI, whose code is automatically generated by
> JavaCPP on build. Just run `ant -Djavacpp.options=-cpp` on JavaCV's
> source code to have it leave the .cpp files for your inspection
>
>
> But I could not see any *.cpp files in any of the folders.
Look inside the build directory, they should be there down the path..
> I have another question. Given the following code and the input is a NIO
> direct bytebuffer
>
> private ByteBuffer resizeDirectMemory(ByteBuffer b, Dimension d) throws
> IOException {
> b.rewind();
> long t = System.currentTimeMillis();
> CvMat mat = cvMat(1, b.capacity(), CV_8UC1, new
> BytePointer(b));
> IplImage originalImage = cvDecodeImage(mat);
> System.out.println(""+b.isDirect()+","+b.hasArray()+","+b.capacity()+","+ originalImage.imageData().capacity());
> IplImage resizedImage = IplImage.create(d.width, d.height,
> originalImage.depth(),originalImage.nChannels());
> cvResize(originalImage, resizedImage);
> System.out.println("Resize Time = "+(System.currentTimeMillis() - t));
> return resizedImage.getByteBuffer();
> }
>
> Since b is a Direct buffer, there is a shared memory allocated for it.
> Now after the originalImage is created from CvMat, is
> originalImage.imageData pointing to the same bytebuffer or is a new one
> allocated?
>
A new one, the original buffer is an encoded image, while
IplImage.imageData is the decoded pixel values, which are larger than
the encoded data, so it cannot be the same buffer.
Samuel
Samuel Audet
> I added the following lines to the code
>
> private ByteBuffer resizeDirectMemoryWithoutDebug(ByteBuffer b,
> Dimension d) throws IOException {
> b.rewind();
> long t = System.currentTimeMillis();
> BytePointer srcPointer = new BytePointer(b);
> CvMat mat = cvMat(1, b.capacity(), CV_8UC1, srcPointer);
> IplImage originalImage = cvDecodeImage(mat);
> System.out.println(""+b.isDirect()+","+b.hasArray()+","+b.capacity()+","+ originalImage.imageData().asBuffer().capacity());
> IplImage resizedImage = IplImage.create(d.width, d.height,
> originalImage.depth(),originalImage.nChannels());
> BytePointer bp = originalImage.imageData();
> System.out.println(srcPointer.toString()+","+bp.toString());
> cvResize(originalImage, resizedImage);
> System.out.println("Resize Time = "+(System.currentTimeMillis() - t));
> return resizedImage.getByteBuffer();
> }
>
> and this is the output I see
>
> com.googlecode.javacpp.BytePointer[address=0x8481000,position=0,capacity=26299,deallocator=null],com.googlecode.javacpp.BytePointer[address=0xa2233c0,position=0,capacity=0,deallocator=null]
>
> In this particular case, is imageData not being used since the capacity
> is 0.
capacity==0 usually means that the allocate size is unknown from
JavaCPP, that's normal
> All this investigation is to find out how many copies of the image bytes
> of the original image exist when the above method is run? Is the answer
> 1 (the bytes being present only in the direct allocated bytebuffer
> input)? Or does IplImage have a different structure after cvDecodeImage() ?
cvDecodeImage() allocates a new IplImage, so you should call
cvReleaseImage() on that, and you create another one there with
IplImage.create(), and that's it.
But I see that you return resizedImage.getByteBuffer() and you do not
keep a reference to resizedImage anywhere. This is a problem because the
memory pointed by resizedImage.getByteBuffer() will get deallocated by
OpenCV when resizedImage gets garbage collected...
Samuel
Jayakrishnan Nair
Jayakrishnan Nair
Samuel Audet
> My resizeDirectMemory() method returns a ByteBuffer from the
> resizedImage and I am trying to write it into a file.
Then call cvSaveImage(), not cvEncodeImage().
> I did the following
>
> ByteBuffer b2 = resizeDirectMemory(b, new Dimension(250, 250));
> byte[] arr = new byte[b2.capacity()];
> b2.get(arr);
>
> CvArr image = cvMat(1, b2.capacity(), CV_8UC1, new BytePointer(b2));
> CvMat encodedImg = cvEncodeImage(".jpg", image);
>
> Now if I want to display the image, what would I do?
You could use CanvasFrame.showImage() as shown in the README.txt file.
Samuel