Emscripten Asynchronous File System API questions

[Robert Goulet]

Hi all,

I have a few questions about asynchronous file system API:

1. Can we use emscripten_async_wget or other async methods before we enter the emscripten main loop?
2. The documentation says "In addition to fetching the URL from the network, the contents are prepared so that the data is usable in IMG_Load and so forth..." Is this still true in 1.34.12? I recall there was a change to disable content preparation by default; does this apply to these functions or are they still preparing content?
3. I don't see any API to verify if a file exist before we async wget it, would it be possible to add an API that only tell us if the file exist on the server?

In our engine, the way it currently is we load resources asynchronously, but it must be completed before we start the update/render loop. Would that be an architecture issue?

Thanks!

[Alon Zakai]

1. Yes, the main loop and async_wget etc. APIs are not related. Although, you might want to pause the main loop while waiting, but whether that makes sense for the app depends on the app.

2. Yes, still true.

3. I'm not sure how to check if a file exists remotely aside from trying to load it and seeing if it succeeds. In other words, you can just see if you get the async_wget error callback?

No architecture issue. But given that, you might want to pause the main loop until all the data you need for the main loop arrives.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Goulet]

1. We are not in the main loop yet at that moment, so I guess I have to keep blocking the main thread with a semaphore waiting for all resource threads to finish, just like we do on other platforms.
2. Ok is there a way to disable it?
3. What about a HEAD request? That would be much more efficient than GET, since it doesn't download the content.

One more question:

     4. When we download files using async_wget (the ones that return a filepath rather than memory location), are those files cached by the browser? are they persisted in MEMFS or anywhere else? or do we have to manually handle this use case with a different file system? In fact I wonder if its possible to actually write to disk when running in a browser?

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Alon Zakai]

2. Not currently. Is this a concern - are you downloading a large number of images or audios? If necessary, we could add an api like emscripten_async_wget_without_preload_plugins()

3. Makes sense, although it's also pretty simple to do with an EM_ASM block with an XHR. But I can see that a C API might be nicer to use, no objection.

4. wget etc. methods that write to a path in the filesystem will store the file in the emscripten filesystem, which by default is MEMFS. So it is stored in memory, as part of the JS of the page. You can't directly write to disk in a browser, but you can asynchronously use IndexedDB for persistent data. Emscripten has IDBFS and emscripten_idb* methods here.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Floh]

As for point 3: there's isn't really a useful way in HTTP to check if a file exists, apart from firing a request and checking the returned HTTP status (the emscripten_async_wget function would call the onerror callback in that case).

What I do usually is to create a table-of-content (TOC) file in an offline-build process, this contains all existing files on the web servers, and sometimes additional information like file size and MD5 hash. The TOC file is downloaded first and stored in a map/dictionary in RAM. When the actual asset files are loaded, an exists-check is done first through the TOC, thus if a file doesn't exist no server round-trip is necessary. You can also check for the existence of directories, and iterate over their contents, since all this information is in the TOC-file (and HTTP doesn't provide this functionality).

Cheers,
-Floh.

[Robert Goulet]

@Alon #2 is not a concern, but I don't see why this is forced, perhaps just a simple boolean to tell it if we need it would be sufficient.

@Floh As I mentioned above, a HEAD request would be much more efficient than GET for this kind of operation.

[Alon Zakai]

#2: A boolean argument is nicer in some ways, I agree, but it also means we would need to break the existing API and force existing code using it to make a change. But maybe it's worth it.

--

[Robert Goulet]

Regarding the emscripten_idb_async_store function, can we safely free the memory passed to this function or does it need to exist until either store or error callbacks are called?

And if I need to get the content I wrote using this IndexedDB system, is there a way to get it outside the web browser? For instance, say I'm creating a screenshot in PNG format, and want to write it on disk, and use it elsewhere, is it possible with IndexedDB?

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Alon Zakai]

It's safe to do so, the method creates a copy,

https://github.com/kripken/emscripten/blob/master/src/library_idbstore.js#L24

I think the answer to the second question is no. Unless someone wrote a tool to read a specific browser's IndexedDB storage. But in general, there is no spec for how that is stored, there is only a spec for how using IndexedDB inside the browser. I suppose a tool could be written though that opens a browser, reads the file from IndexedDB, and saves it somewhere, all automatically, as a way to read from IndexedDB.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

The problem I was trying to solve was to write on disk the content of a binary buffer that was in memory when running the JS generated from emscripten. I found a clever way to do that:

I encode the content of the binary buffer to base64, then I create an html data download link on the web page using emscripten_run_script(). Then when the page is executed, I get that link I can click which will allow me to save the file on disk.

So I didn't really need the IndexedDB for this, but at least I can see how the IndexedDB will be useful later for storing user settings or other kind of data.

Cheers!

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.