Are Ther Any Effective Memory Usage Techniques When Using the Camera and Resizing Images
246 views
Skip to first unread message
SockThief
Aug 20, 2012, 2:31:19 AM8/20/12
to phon...@googlegroups.com
Hi again, everyone, hope it was a splendid weekend of hacking all round!
Im working on an extremely common idea in the app world - an image uploader - feels almost like a Camera Hello World project now, and everyone has their own take on it.
I was reading with great interest Simon's blog entry here: http://simonmacdonald.blogspot.com/2012/07/change-to-camera-code-in-phonegap-190.html regarding memory usage in images, and indeed, when testing with my HTC Desire HD (8MP Camera) I would get out-of-memory errors, though, from a user perspective, there was a small blip in the screen refresh and no real issues.
What I'm wondering is, can we get a file resource directly without loading the picture into memory? Quoting Simon: "In order to do any manipulation on an image like scaling or compression you need to load the entire image into memory." Which, alright, I can understand some of the design philosophy here (it does make implementations easier, after all). And from a phonegap perspective, to allow resizing and quality changes (with the end goal of a smaller image and less memory) you first have to load the entire image to memory, a catch-22.
Now, moving on, then, if I want to read the EXIF data, I need to load the image, with a quality of 100. Again, quoting Simon:
"The only way for the EXIF info not to be touched is to specify a quality of 100 and a destination of FILE_URI. Don't use any other parameters as that will cause the file to get re-written. I can only keep the following EXIF information as that is what the SDK will allow me to access:
https://developer.android.com/reference/android/media/ExifInterface.html
GPS and Datetime should stick around. They do on my Samsung phone."
So, taking my use-case, I want to load an image (from camera or disk), display it as a preview, then upload it, with EXIF data. So, I retrieve the file, as a URI, display it in a div, then use fileTransfer to upload it. All seems ok, except I get out of memory errors, and the file takes a massive amount of time to upload over a 3G connection.
From Simons blog: "The important part about the options is that you specify a quality of 100. This along with accepting the default width and height will skip the need to load the image in memory." - thus I can only conclude that I get the memory warnings when I use inline CSS to reduce the size of the image to fit the screen (i am yet to test this hypothesis).
So, it would seem that merely getting a URI is not enough, if you wish to display the image to the screen, an effective way of resizing is required (in addition to any resizing for upload). Using the targetWidth/targetHeight is not an option, we must use something native to android, though I fear this will lead us in a circle, as a native solution would still require loading the image to memory - it almost seems to beg the question, why does android allow such large MP count cameras, if it can't handle the processing of the images effectively?
I have looked into Raanan Webers resize plugin for PhoneGap (https://github.com/raananw/PhoneGap-Image-Resizer) though theres not much information from people have used it - might have to play and come back with a review.
So, in essence and a nutshell: What is the best way to handle images (esp. large images) inside Android / Phonegap (i suspect this is an Android issue, not just PhoneGap)? Is there a good process for displaying images on screen and also resizing for upload. If the image to upload is a diferent size than that shown on screen, a second resize could be required -> second memory issue!
After a weekend pondering, Im yet to come to a conclusion on this, though plenty of reports and similar queries around without an answer!
Whilst I leave this with the brainstrust, I will test if it is the CSS resize causing my memory issue, and see how Raanan Webers plugin plays
talk to you all soon!
SockThief
Im working on an extremely common idea in the app world - an image uploader - feels almost like a Camera Hello World project now, and everyone has their own take on it.
I was reading with great interest Simon's blog entry here: http://simonmacdonald.blogspot.com/2012/07/change-to-camera-code-in-phonegap-190.html regarding memory usage in images, and indeed, when testing with my HTC Desire HD (8MP Camera) I would get out-of-memory errors, though, from a user perspective, there was a small blip in the screen refresh and no real issues.
What I'm wondering is, can we get a file resource directly without loading the picture into memory? Quoting Simon: "In order to do any manipulation on an image like scaling or compression you need to load the entire image into memory." Which, alright, I can understand some of the design philosophy here (it does make implementations easier, after all). And from a phonegap perspective, to allow resizing and quality changes (with the end goal of a smaller image and less memory) you first have to load the entire image to memory, a catch-22.
Now, moving on, then, if I want to read the EXIF data, I need to load the image, with a quality of 100. Again, quoting Simon:
"The only way for the EXIF info not to be touched is to specify a quality of 100 and a destination of FILE_URI. Don't use any other parameters as that will cause the file to get re-written. I can only keep the following EXIF information as that is what the SDK will allow me to access:
https://developer.android.com/reference/android/media/ExifInterface.html
GPS and Datetime should stick around. They do on my Samsung phone."
So, taking my use-case, I want to load an image (from camera or disk), display it as a preview, then upload it, with EXIF data. So, I retrieve the file, as a URI, display it in a div, then use fileTransfer to upload it. All seems ok, except I get out of memory errors, and the file takes a massive amount of time to upload over a 3G connection.
From Simons blog: "The important part about the options is that you specify a quality of 100. This along with accepting the default width and height will skip the need to load the image in memory." - thus I can only conclude that I get the memory warnings when I use inline CSS to reduce the size of the image to fit the screen (i am yet to test this hypothesis).
So, it would seem that merely getting a URI is not enough, if you wish to display the image to the screen, an effective way of resizing is required (in addition to any resizing for upload). Using the targetWidth/targetHeight is not an option, we must use something native to android, though I fear this will lead us in a circle, as a native solution would still require loading the image to memory - it almost seems to beg the question, why does android allow such large MP count cameras, if it can't handle the processing of the images effectively?
I have looked into Raanan Webers resize plugin for PhoneGap (https://github.com/raananw/PhoneGap-Image-Resizer) though theres not much information from people have used it - might have to play and come back with a review.
So, in essence and a nutshell: What is the best way to handle images (esp. large images) inside Android / Phonegap (i suspect this is an Android issue, not just PhoneGap)? Is there a good process for displaying images on screen and also resizing for upload. If the image to upload is a diferent size than that shown on screen, a second resize could be required -> second memory issue!
After a weekend pondering, Im yet to come to a conclusion on this, though plenty of reports and similar queries around without an answer!
Whilst I leave this with the brainstrust, I will test if it is the CSS resize causing my memory issue, and see how Raanan Webers plugin plays
talk to you all soon!
SockThief
Simon MacDonald
Aug 21, 2012, 12:57:39 PM8/21/12
to phon...@googlegroups.com
Hey Robert,
Since I'm so liberally linked to I figured I should respond. My
comments are inline.
Simon Mac Donald
http://hi.im/simonmacdonald
To select and image from the Camera/Gallery without loading it into
memory you would want to do this:
var options = {
quality: 100,
destinationType : navigator.camera.DestinationType.FILE_URI,
sourceType: navigator.camera.PictureSourceType.CAMERA,
encodingType: navigator.camera.EncodingType.JPEG,
}
navigator.camera.getPicture(win, fail, options);
Your "win" method will be called with a URI that points to the image
in question. The image is never loaded into memory and the EXIF
information is never touched. An 8MP camera using JPEG encoding with
100% quality will result in an image of about 2 megabytes. Taking into
account that many 3G service providers cap their upload speeds at 150
kbps you could get a theoretical quickest upload time of about one
minute and 46 seconds.
> So, it would seem that merely getting a URI is not enough, if you wish to
> display the image to the screen, an effective way of resizing is required
> (in addition to any resizing for upload). Using the targetWidth/targetHeight
> is not an option, we must use something native to android, though I fear
> this will lead us in a circle, as a native solution would still require
> loading the image to memory - it almost seems to beg the question, why does
> android allow such large MP count cameras, if it can't handle the processing
> of the images effectively?
If you want to display the image on screen you can use the width and
height parameters of the image tag to specify a size that'll fit on
your screen. That is asking the browser to do a lot of extra work for
you but it can handle it. Alternatively you can use the
targetWidth/targetHeight parameters. I made changes in PG Android that
take advantage of a scaling algorithm that Android provides so we
don't load the entire 8 MP image into memory if we can help it.
For instance if you have a 2000x2000 image and you want a 400x400
image the algorithm will actually load a quarter sized image into
memory at 500x500 and then reduce it down to 400x400. The algorithm
only works in powers of 2 so it will get a half, quarter, eighth,
sixteenth, etc. sized image.
So memory usage is much better now when you are asking for thumbnailed images.
> I have looked into Raanan Webers resize plugin for PhoneGap
> (https://github.com/raananw/PhoneGap-Image-Resizer) though theres not much
> information from people have used it - might have to play and come back with
> a review.
I've looked over this plugin and it suffers from the fact that it
tries to load the entire image into memory at line 193. It could be
improved to use the same algorithm I described to load a smaller
amount of data in memory. However, the way it looks now is that your 8
MP image would probably cause the plugin to throw an out of memory
error as well.
> So, in essence and a nutshell: What is the best way to handle images (esp.
> large images) inside Android / Phonegap (i suspect this is an Android issue,
> not just PhoneGap)? Is there a good process for displaying images on screen
> and also resizing for upload. If the image to upload is a diferent size than
> that shown on screen, a second resize could be required -> second memory
> issue!
Honestly, at this point I'd just go with keeping the large image at
100% quality and letting the browser do the resizing. Please let me
know how that goes.
> After a weekend pondering, Im yet to come to a conclusion on this, though
> plenty of reports and similar queries around without an answer!
>
> Whilst I leave this with the brainstrust, I will test if it is the CSS
> resize causing my memory issue, and see how Raanan Webers plugin plays
>
> talk to you all soon!
> SockThief
Where did my socks go? They're gone but my sneakers are still on.
Since I'm so liberally linked to I figured I should respond. My
comments are inline.
Simon Mac Donald
http://hi.im/simonmacdonald
memory you would want to do this:
var options = {
quality: 100,
destinationType : navigator.camera.DestinationType.FILE_URI,
sourceType: navigator.camera.PictureSourceType.CAMERA,
encodingType: navigator.camera.EncodingType.JPEG,
}
navigator.camera.getPicture(win, fail, options);
Your "win" method will be called with a URI that points to the image
in question. The image is never loaded into memory and the EXIF
information is never touched. An 8MP camera using JPEG encoding with
100% quality will result in an image of about 2 megabytes. Taking into
account that many 3G service providers cap their upload speeds at 150
kbps you could get a theoretical quickest upload time of about one
minute and 46 seconds.
> So, it would seem that merely getting a URI is not enough, if you wish to
> display the image to the screen, an effective way of resizing is required
> (in addition to any resizing for upload). Using the targetWidth/targetHeight
> is not an option, we must use something native to android, though I fear
> this will lead us in a circle, as a native solution would still require
> loading the image to memory - it almost seems to beg the question, why does
> android allow such large MP count cameras, if it can't handle the processing
> of the images effectively?
height parameters of the image tag to specify a size that'll fit on
your screen. That is asking the browser to do a lot of extra work for
you but it can handle it. Alternatively you can use the
targetWidth/targetHeight parameters. I made changes in PG Android that
take advantage of a scaling algorithm that Android provides so we
don't load the entire 8 MP image into memory if we can help it.
For instance if you have a 2000x2000 image and you want a 400x400
image the algorithm will actually load a quarter sized image into
memory at 500x500 and then reduce it down to 400x400. The algorithm
only works in powers of 2 so it will get a half, quarter, eighth,
sixteenth, etc. sized image.
So memory usage is much better now when you are asking for thumbnailed images.
> I have looked into Raanan Webers resize plugin for PhoneGap
> (https://github.com/raananw/PhoneGap-Image-Resizer) though theres not much
> information from people have used it - might have to play and come back with
> a review.
tries to load the entire image into memory at line 193. It could be
improved to use the same algorithm I described to load a smaller
amount of data in memory. However, the way it looks now is that your 8
MP image would probably cause the plugin to throw an out of memory
error as well.
> So, in essence and a nutshell: What is the best way to handle images (esp.
> large images) inside Android / Phonegap (i suspect this is an Android issue,
> not just PhoneGap)? Is there a good process for displaying images on screen
> and also resizing for upload. If the image to upload is a diferent size than
> that shown on screen, a second resize could be required -> second memory
> issue!
100% quality and letting the browser do the resizing. Please let me
know how that goes.
> After a weekend pondering, Im yet to come to a conclusion on this, though
> plenty of reports and similar queries around without an answer!
>
> Whilst I leave this with the brainstrust, I will test if it is the CSS
> resize causing my memory issue, and see how Raanan Webers plugin plays
>
> talk to you all soon!
> SockThief
SockThief
Aug 22, 2012, 5:40:12 AM8/22/12
to phon...@googlegroups.com
Simon,
it was such a great blog post, and illustrated the issues with dealing with images - especially as resolutions climb ever higher - so well that it was the best reference for the post! That, and I highly recommend all PhoneGap users to read it
With respect to your replies, indeed you are correct, I have basically the code you use, and I don't receive any memory errors (= good!) what I do get though is:
nativeDrawContent took too long: 437ms
when the image is drawn to screen. This would be the extra processing being placed onto the browser, and does cause havoc with the the onResume event. For instance, in my app, I am ooking for the onResume and re-initialising a few bits a pieces around the place, this was as I wanted to reload some data when the app became visible again, as JS wont get processed when the app is in the background - however, because the redraw takes so long, the onResume event is triggered... Ouch!
I've since worked around this - but a trap for young players indeed!
With respect to image resizing for upload, I am still of the opinion that this is a necessary feature, as you point out, 1min 46 is about the upload time, in a best case scenario. Unfortunately I work in many situations where the 3G connection is not going to be this good - even in "developed" nations, for instance areas of Australia would be lucky to have EDGE. In these cases, the upload time is going to be well over the 2 minute mark. I had a case yesterday where an upload on 3G (in Sweden!) took over 10 minutes.... this was due to a combination of network congestion, international carriers and server issues, however in a perfect-storm scenario, the need for smaller images still exists. Even at 1min 30, there exists an issue, in that, as best as I can see (correct me if Im wrong), there exists no official support in PhoneGap for file upload progress - though there a couple of plugins I believe - this could leave the user in the situation of looking at a spinning-gif for an unknown amount of time. Without any user-experience stats at hand, im betting that after 30seconds or 1 minute at most, users would be cancelling the upload beleiving it to have hung - for social network purposes, an upload that takes more than 30 seconds or so is probably not user friendly (unless they chose to use a full res upload and were aware that it takes time)
its a tricky situation to handle, I feel - and certainly not made easier by Androids image handling routines - I have written an iOS port of this app (native objective-c) and it doesn't seem to exhibit any such issues, even on my iPhone 4S - would be interesting to see how it handled a 20MP image from my DSLR though...
Is it possible - as a workaround - to pull the URI as we do now at 100% quality, extract the metadata (EXIF) as a string, then resize the image, knowing the URI in a memory efficient way - the EXIF would then be uploaded seperately and recombined server side (how you have to handle resized images + exif in the iOS world)?
Step 1. Steal Socks
Step 2. ?
Step 3. Profit.... thanks for the donation!
SockThief
it was such a great blog post, and illustrated the issues with dealing with images - especially as resolutions climb ever higher - so well that it was the best reference for the post! That, and I highly recommend all PhoneGap users to read it
With respect to your replies, indeed you are correct, I have basically the code you use, and I don't receive any memory errors (= good!) what I do get though is:
nativeDrawContent took too long: 437ms
when the image is drawn to screen. This would be the extra processing being placed onto the browser, and does cause havoc with the the onResume event. For instance, in my app, I am ooking for the onResume and re-initialising a few bits a pieces around the place, this was as I wanted to reload some data when the app became visible again, as JS wont get processed when the app is in the background - however, because the redraw takes so long, the onResume event is triggered... Ouch!
I've since worked around this - but a trap for young players indeed!
With respect to image resizing for upload, I am still of the opinion that this is a necessary feature, as you point out, 1min 46 is about the upload time, in a best case scenario. Unfortunately I work in many situations where the 3G connection is not going to be this good - even in "developed" nations, for instance areas of Australia would be lucky to have EDGE. In these cases, the upload time is going to be well over the 2 minute mark. I had a case yesterday where an upload on 3G (in Sweden!) took over 10 minutes.... this was due to a combination of network congestion, international carriers and server issues, however in a perfect-storm scenario, the need for smaller images still exists. Even at 1min 30, there exists an issue, in that, as best as I can see (correct me if Im wrong), there exists no official support in PhoneGap for file upload progress - though there a couple of plugins I believe - this could leave the user in the situation of looking at a spinning-gif for an unknown amount of time. Without any user-experience stats at hand, im betting that after 30seconds or 1 minute at most, users would be cancelling the upload beleiving it to have hung - for social network purposes, an upload that takes more than 30 seconds or so is probably not user friendly (unless they chose to use a full res upload and were aware that it takes time)
its a tricky situation to handle, I feel - and certainly not made easier by Androids image handling routines - I have written an iOS port of this app (native objective-c) and it doesn't seem to exhibit any such issues, even on my iPhone 4S - would be interesting to see how it handled a 20MP image from my DSLR though...
Is it possible - as a workaround - to pull the URI as we do now at 100% quality, extract the metadata (EXIF) as a string, then resize the image, knowing the URI in a memory efficient way - the EXIF would then be uploaded seperately and recombined server side (how you have to handle resized images + exif in the iOS world)?
Step 1. Steal Socks
Step 2. ?
Step 3. Profit.... thanks for the donation!
SockThief
Simon MacDonald
Aug 28, 2012, 1:52:03 PM8/28/12
to phon...@googlegroups.com
On Wed, Aug 22, 2012 at 5:40 AM, SockThief <robert....@gmail.com> wrote:
>
> With respect to your replies, indeed you are correct, I have basically the
> code you use, and I don't receive any memory errors (= good!) what I do get
> though is:
>
> nativeDrawContent took too long: 437ms
>
> when the image is drawn to screen. This would be the extra processing being
> placed onto the browser, and does cause havoc with the the onResume event.
> For instance, in my app, I am ooking for the onResume and re-initialising a
> few bits a pieces around the place, this was as I wanted to reload some data
> when the app became visible again, as JS wont get processed when the app is
> in the background - however, because the redraw takes so long, the onResume
> event is triggered... Ouch!
>
> I've since worked around this - but a trap for young players indeed!
Yeah, it is not necessarily the best workaround as it does put all the
>
> With respect to your replies, indeed you are correct, I have basically the
> code you use, and I don't receive any memory errors (= good!) what I do get
> though is:
>
> nativeDrawContent took too long: 437ms
>
> when the image is drawn to screen. This would be the extra processing being
> placed onto the browser, and does cause havoc with the the onResume event.
> For instance, in my app, I am ooking for the onResume and re-initialising a
> few bits a pieces around the place, this was as I wanted to reload some data
> when the app became visible again, as JS wont get processed when the app is
> in the background - however, because the redraw takes so long, the onResume
> event is triggered... Ouch!
>
> I've since worked around this - but a trap for young players indeed!
work in the content view. Can you share how you've worked around the
issue so other can benefit?
> With respect to image resizing for upload, I am still of the opinion that
> this is a necessary feature, as you point out, 1min 46 is about the upload
> time, in a best case scenario. Unfortunately I work in many situations where
> have hung - for social network purposes, an upload that takes more than 30
> seconds or so is probably not user friendly (unless they chose to use a full
> res upload and were aware that it takes time)
Yeah, I don't disagree that uploading can take quite a lot of time.
> seconds or so is probably not user friendly (unless they chose to use a full
> res upload and were aware that it takes time)
Even with 2 minutes being the best case scenario is still not great.
There is work currently going on to add progress indications to the
FileTransfer code. Hopefully it will be ready for the 2.2 release in
October.
> Is it possible - as a workaround - to pull the URI as we do now at 100%
> quality, extract the metadata (EXIF) as a string, then resize the image,
> knowing the URI in a memory efficient way - the EXIF would then be uploaded
> seperately and recombined server side (how you have to handle resized images
> + exif in the iOS world)?
Sadly Android does not give you full access to all EXIF headers. Take
a look at:
https://developer.android.com/reference/android/media/ExifInterface.html
to see what is supported.
Reply all
Reply to author
Forward
0 new messages