PouchDB retrieve binary blob (image/png) attachment

[Yechezkal Gutfreund]

I really like PouchDB. Learned it fast, good doc, and it works well.

Issue: I am having trouble retrieving a binary blob that I stored as an attachment. (type: image/png). I think it is a simple issue in that I don't understand the response item;. I have used createObjecURL() on data stored directly in IndexedDB, but I think
it is a small issue of how to do this from PouchDB.

the blob was fetched by using the xhr lib: https://github.com/p-m-p/xhr2-lib

with the function:

function getBlobs(fileList) {
    console.log("starting to fetch blobs");
    $.each(fileList, function (i, val) {
        var path = baseURL + val.path;
        $xhr.ajax({
            url: path,
            dataType: "blob",
            success: function (data) { saveBlob(data, val.size, val.id); }
        });
    });
}

The saving of the 

function putBlob(blob, id) { //don't worry is always a unique string
    var cnt;
    var type = blob.type; //for now the type is always 'image/png'
    DB.putAttachment(id, blob, type, function (err, response) {
        if (err) {
            console.error("Could store blob: error: " + err.error + " reason: " + err.reason + " status: " + err.status);
        } else {
            console.log("saved: ", response.id + " rev: " + response.rev);
            showID = response.id;
            cnt = vm.get("blobCount");
            vm.set("blobCount", ++cnt);
            if (cnt == manifest.files.length) {
                setTimeout(storeComplete, 0);
            }
        }
    });
}

The part that blows up is when I try to grab the binary blob (json objects stored in the pouchDB work fine).

Question: Should I be wrapping my binary blob in a JSON object when I store it?


Here is the routine that fails. it does on the createObjectURL (both in FireFox and Chrome).
Oh yes, I just grabbed the latest nightly build of PouchDB.

function setImage() {
    var id = showID;
    var blob;
    DB.getAttachment(id, function (err, response) {
        if (err) {
            console.error("Could find blob: error: " + err.error + " reason: " + err.reason + " status: " + err.status);
        } else {
            blob = response._attachments;
            var imgURL = URL.createObjectURL(blob);
            // Set img src to ObjectURL
            $("#imgTile").attr("src", imgURL);
            // Revoking ObjectURL
            URL.revokeObjectURL(imgURL);
        }
    });
}

 

[Yechezkal Gutfreund]

Oh yes, I looked at issue #730, and I tried doing createObjectURL() for the entire response (though I don't see how that would work). And it did not work for me:

https://github.com/daleharvey/pouchdb/issues/730

[Danny Ching]

have you considered using encoding your blobs to base62 encoding so that you just need to save strings and not blobs instead?

On Wed, May 22, 2013 at 3:05 AM, Yechezkal Gutfreund <ygutf...@gmail.com> wrote:

Oh yes, I looked at issue #730, and I tried doing createObjectURL() for the entire response (though I don't see how that would work). And it did not work for me:

https://github.com/daleharvey/pouchdb/issues/730

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/bda21bd4-0f38-48f2-b092-cd193f094a25%40googlegroups.com?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Yechezkal Gutfreund]

Hello Danny,

Base64 encoding is a last resort for my application. I am storing 0.5GB to 1GB of raster map tiles (PNG 256x256) in the IndexedDB for offline mapping. Base64
would not only increase the size by 33%, but also:

• Add the retrieval time (decode from base64 to blob to CreateURL
• Might make smooth zoom and pan stop working (too slow) for the map application that is fetching the data from PouchDB (Leaflet).
  

Besides, a few months ago I created a native IndexedDB for binary blob storage. Works great in IE10, FF, and Chrome (well,for Chrome I used the FileSystem as a polyfill). This works great. Now I am told that PouchDB supports binary blob attachments. I understand Dale is interested in seeing this work. I am also.

I suspect it is just something in the PouchDB API I do not understand. But if it takes more than that, I am willing to spend time, and create CodePen.io demos help figure this one out. (i.e. I am motivated, and funded to make help make PouchDB work for binary blobs).

[Danny Ching]

Retrieval should be fine without decoding if you use data url's. but I do see why the 33% additional storage will be bad.

--

You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/87d19b6b-447c-4266-9f9a-8eecfe36aac0%40googlegroups.com?hl=en.

[Yechezkal Gutfreund]

The storage can be an issue on mobile devices (obviously, I am thinking more about tablets for a map application, than a phone).

But the performance issues are the real one of concern. Especially on mobile. An offline map has to grab the tiles really fast for pan and zoom. A DB is great, since the retrieval can be optimized.

BTW: Do you know how to convert a image/png blob (fetched by XHR2) into a base64? and then how I would convert it back from Base64 to blob?

It looks like a fragmented set of tools. I would rather work with Dale and get the binary blob storage working. I hope to have a CodePen up today for people to see what is going on. Given that the GitHub site says this works, lets see it always work).


http://stackoverflow.com/questions/246801/how-can-you-encode-to-base64-using-javascript

[Johannes J. Schmidt]

Hi Yechezkal,

this might be a problem with the id. Does it follow the form '<document
id>/<attachment name>'?
The documentation should mention that an attachment id must contain a slash.

I think in your case a document is created with an attachment with the
name "" (empty string).
Then later, when you request the attachment, the document is returned
instead of the attachment.
Both should not be possible and PouchDB should throw an error.

If my assumptions are right, could you open an issue at Github, please?

g jo
On 05/21/2013 09:03 PM, Yechezkal Gutfreund wrote:
>
> <https://lh4.googleusercontent.com/-ucEssB17TGE/UZvE2m3exkI/AAAAAAAAAOg/LmtKg0r-92U/s1600/CropperCapture%5B1%5D.jpg>

> --
> You received this message because you are subscribed to the Google
> Groups "PouchDB" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pouchdb+u...@googlegroups.com.
> To post to this group, send email to pou...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pouchdb/26cce745-0ac5-4da8-9747-42db2da4ef8c%40googlegroups.com?hl=en.

[Danny Ching]

Hope this can help. I don't decode fom base64 any more. I just use data URL's. don't know if it cuts down on CPU usage though.

<head>

<script src="jquery.js"></script>

</head>

<body>

<input type="file" id="fileupload" onchange="process(this, function(reply){$('#results').html('<img src=\''+reply+'\' />');});" style="display:none;"/>

<div onclick="$('#fileupload').click();" style=" font-size:32px;">load image</div>

<div id="results"></div>

<script >

function process(fobj, callback) {

var reader = new FileReader(); 

reader.onload = function (theevent) {

callback(theevent.target.result);

}

var file =$(fobj).get(0).files[0]; 

reader.readAsDataURL(file); 

}

</script>

</body>

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/519E23B3.2010605%40netzmerk.com?hl=en.

[Yechezkal Gutfreund]

Guten Tag Johannes,

It is now working in Chrome. Thank you. I suspected that since I have not done any CouchDB or PouchDB before, it was the documentation that was tripping me up. That was correct.
I did not know that one needed the ID syntax to be "<doc>/<attachment>". I did not want to put an issue into GitHub until I knew it whether this was an bug, or just the fact that I did not understand the API.

Dale, sent me mail a few days ago, saying how I could avoid an extra transaction (doc put, then attach put) and store the blob directly and the document would be generated:

-----

You can create attachments without previously creating the document (the empty doc will be auto created)

https://github.com/daleharvey/pouchdb/blob/master/tests/test.attachments.js#L170

---------

I still have issues with FireFox and IE10, So I am going to put a "jsFiddle" so that others can see what is going on. IE10 has a fatal issue with lack of FileReader.readAsBinaryString() (GitHub issue filed, and I see that you have already commented on the existing md5sum issue). https://github.com/daleharvey/pouchdb/issues/771


FireFox is not displaying a full size image. The JSFiddle should help.

 

[Yechezkal Gutfreund]

I have created a "JsFiddle" (actually I prefer CodePen these days), as a playground for showing how to use PouchDB to cache off-line raster map tiles.

http://codepen.io/DrYSG/pen/hpqoD

The Algorithm it uses is as follows:

1. Test for presence of XHR2, IndexedDB, and Chrome (which does not have binary blobs, but Base64). and show this status info
2. Fetch a JSON manifest of PNG tiles from GoogleDrive (I have 171 PNG tiles, each 256x256 in size). The manifest lists their names and sizes.
   
3. Store the JSON manifest in the DB
4. MVVM and UI controls are from KendoUI (This time I did not use their superb grid control, since I wanted to explore CSS3 Grid Styling).
5. XHR2 is from: https://github.com/p-m-p/xhr2-lib/blob/master/test/index.html
6. I am using the nightly build of PouchDB
7. All files PNG file are stored on Google Drive (NASA Blue Marble. I created the tile pyramid with Safe FME 2013 Desktop. http://www.safe.com/fme/fme-technology/fme-desktop/overview

Before Pressing the button "Download Tiles" Check to see that the manifest has been stored in the DB, and that 171 tiles are present. If you already ran the test  then your PouchDB is going to already have tiles in the DB, and you will get errors. In that case, Press Delete DB, and then reload the page.

When you press "Download Tiles" The following steps occur:

1. The Manifest is fetched from the DB
2. A XHR2 Fetch loop grabs the PNG blobs from GoogleDrive. 
3. As loop runs, it starts saving the Blobs into PouchDB.
4. Note: Fetching and Saving are on overlapped threads (think co-routines), since those (fetch & store) operations are running async on separate threads.
   
5. When the Fetch loop is done it reports the elapsed time.
6. Note: This time is not pure Fetch work, since PouchDB putAttachments() are running in parallel.
7. When all the Tiles are saved, it will report full statistics, and display a tile fetched from PouchDB.
8. The Blob-Rate the total fetch and store time per each png tile
   

Right now Chrome is running fine. Firefox is very slow. I found this out a few months when I did a native IndexedDB API. So I don't think this is a PouchDB issue. Probably more due to FireFox using SQLlite which is a relational approach to a no-SQL DB.

IE10 is not working. This is sad, since my prior tests with IE10 shows it has a fantastically fast IndexedDB solution: http://stackoverflow.com/questions/14113278/storing-image-data-for-offline-web-application-client-side-storage-database

• A Must read article on "How the browsers store IndexedDB data"
  • http://www.aaron-powell.com/web/indexeddb-storage
• Note: FireFox uses SQLlite for the NOSQL IndexedDB. That might be the reason for the slow performance. (blobs stored separately)
• Note: Microsoft IE uses the extensible storage engine:
  • http://en.wikipedia.org/wiki/Extensible_Storage_Engine
• Note: Chrome uses LevelDB http://code.google.com/p/leveldb/

[Yechezkal Gutfreund]

BTW:

How is PouchDB doing it's bulk sync? Why do I ask?

I tried increasing the number of raster tiles fetched from 171 to 7,000. FireFox works fine. But Chrome starts getting XHR2 errors at about 5,500. No change in the code. Detailed studies of the logs shows that Chrome putting all the XHR2 requests on a background thread.

How do I know this? When the fetches work (e.g. 1000 or so), then I see that all the XHR2 requests go through first, then the PouchDB saves. Well in the case of failure, the XHR2s start failing (response code 0). It seems that Chrome does not do any throttling of the async XHR2 requests, and so too many are piling up. Did you discover this with Chrome? [It will take me a while to create a demo].

[Yechezkal Gutfreund]

I have added a second CodePen demo to show display of map tiles from the PouchDB in Leaflet.

http://codepen.io/DrYSG/pen/oEItn

Make sure you run the first demo first, and populate your local PouchDB (IndexedDB). 

You can turn off the PouchDB overlay using the controls. Zoom only goes from levels 0-4 of the NASA Blue Marble Raster Tiles.

The key idea it getting this to work is to use Leaflet, and the Leaflet Functional Tile Layer plug-in with JQuery DEFER for getting the raster tiles from the PouchDB.

[Dale Harvey]

It looks like you may be using an old version of pouchdb in your demo
https://github.com/daleharvey/pouchdb/commit/3060592d195aad38a5f1b5bb552795fb8b96e118
should have fixed any issues with parallel xhr requests, its possible there are bugs however as it is hard to test, its very possible that the throttling doesnt play well when the requests fail

(I also filed https://github.com/daleharvey/pouchdb/issues/782 so that its easier to figure out)

As for the blob issue, apparently arraybuffers work fine inside indexedDB in chrome, I will be looking into it shortly

These demos are great and the feedback is very useful, thanks a lot, I will try to catch up on the issues you have brought up so far.

Cheers

Dale

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/60c1d160-81f8-469a-ad2f-a395a0d091d0%40googlegroups.com?hl=en.

[Yechezkal Gutfreund]

I just grabbed the latest nightly build of PouchDB. But I don't think that the parallel XHR2 fetches have anything to do with IDB or PouchDB. I think it is an Chrome implementation of XHR2 issue. I created a chromium  bug, and have a better write-up of that issue here:
http://stackoverflow.com/questions/16819676/chrome-chokes-on-too-many-xhr2-requests (as well as demos to see the problem)  (I.e. I was getting the "parallel xhr2 error" when I was doing raw IDB, before I tried PouchDB).

I do have an issue open in PouchDB about IE10 support. I have not heard if I can do any work around: https://github.com/daleharvey/pouchdb/issues/771 basically, I know that IE10 has great binary blob support in IDB. The trick seems to be to get the md5sum for PouchDB (or some other work around).

As for Chrome, it works for me for now. It will be nice to see the chromium folks add true binary blob support. I don't understand what the need is for ArrayBuffers is. If you see the latest post from the chrome development team, it looks like they are going to have excellent binary blob support sometime soon: http://code.google.com/p/chromium/issues/detail?id=108012
-------

@piranna: IDB should support anything covered by the Structured Clone algorithm.  So once I get this feature in, it will support Blobs, Files, and FileLists.  It will not support FileEntry or DirEntry; it's not clear what exactly that would mean w.r.t. permissions, file lifetimes, etc., and the FileSystem spec isn't likely to see a lot of further development in the near term.

[Yechezkal Gutfreund]

Any suggestions for me to get IE 10 Working?

For example, should I convert the blob to an ArrayBuffer, and then store the ArrayBuffer in PouchDB?
Or do I have to convert the blob to a Base64 string and then store it in PouchDB?

In either case, I will have to convert it back to a blob after I fetch it from PouchDB