WasmFS for jsdos

[Александр Гурьянов]

I know that WasmFS is in active development, and I want to switch jsdos to WasmFS.
So, I want to share my use case and if needed help with development.

jsdos is based on dosbox and it uses POSIX syscalls to access the file system.

dosbox has the following scenarios accessing files:
* directly accessing files in FS (read/write)
* mounting iso/cue images (CDROM support) (only read)

jsdos layer has additional features:
* storing FS changes in cloud or idbs
* FS compression (currently I use zip)

jsdos works with zip archives to populate the filesystem. The emulated program has 2 zip archives:
**first** - is a initial game file system
**second** - contains changed files.

Before dosbox actually starts, these two "FS archives" are merged into actual FS.

Merge implementation is very simple:
1. Extract first archive **and store modification time of files in hashtable**.
2. Extract second archive and overwrite files

When jsdos is exited (closed by user) it iterates over FS and checks if some file have newer
modification time. If so, this file is added to brand new **second** archive.

The **second** archive can be stored in idbfs, or in cloud.

This solution has practical problems:
1. zip is slow when extracting files. Not dramatically, but it will be much better to use native decoder of the browser (gzip/brotli).
2. I can say zip compression rate **is good enough** for most games. But you can't extract big games on mobiles. It results in OOM. To extract you need to download the whole archive (1st copy), put it into WASM memory (2st copy) and keep it while it is extracting.
3. At the end of extraction you will consume the same amount of memory as the original game size (e.g. 600Mb for CDROM games). **Unacceptable for mobiles**.
4. There is no option in emscripten FS to notify jsdos when some file is changed, because of that storing changes is triggered manually when the user closes the jsdos window. BUT, most users just **closes the browser tab and lost all progress.**
5. The **second** archive can't handle deletion of files. If some file is deleted it will be there on the next run. It's because there is no marker in the zip to signal that some file needs to be deleted. However, for jsdos this happens very rarely and can be ignored.

WasmFS ideas:
1. WasmFS will notify jsdos layer that some file is changed (write/delete etc), so I can schedule update of second archive in cloud.
2. I can switch from zip archives to any "bundle/asset" file format introduced by WasmFS. Initially I can download some metafile that contain information about the whole FS. When the dosbox tries to access some file, I will:

* stop execution with asyncify
* lookup in meta which FS block need to be downloaded
* download it (with progress indication)
* mount it
* resume execution

I imagine that we can use gzip/brotli browser decoder here.
I think this flow is more or less generic and can be implemented on the WasmFS side, so from jsdos side I only need to build an asyncify white list.

3. Replacement for iso/cue. I can fix dosbox to understand WasmFS file format form (2) to download only needed blocks.


With switching to WasmFS I want to achieve three key features:
* Minimize download size of full-CD games/programs (like RedAlert, Full Throttle, Win95)
* Minimize memory consumption to work on mobiles
* Effectively store changes in cloud

If you have any question, or need my help, feel free to ask.

 

 

[Thomas Lively]

Thanks for the great writeup!

>  WasmFS will notify jsdos layer that some file is changed (write/delete etc), so I can schedule update of second archive in cloud.

Once the core WasmFS system is stable and documented, I hope it will be straightforward to create a custom virtual backend (i.e. a backend that forwards work to another backend) to fire notifications on file modifications.

> At the end of extraction you will consume the same amount of memory as the original game size (e.g. 600Mb for CDROM games). **Unacceptable for mobiles**.

This should be possible to fix with WasmFS by streaming files into WasmFS persistent backends like the OPFS backend or a future IndexedDB backend. These backends will never store entire files in memory, unlike IDBFS in the current system. That streaming should be able to use the browser's built-in decompression schemes as well.

> I can switch from zip archives to any "bundle/asset" file format introduced by WasmFS

I don't think we have plans to introduce a new data bundle format native to WasmFS, but you can implement your own however you'd like.

>  Initially I can download some metafile that contain information about the whole FS. When the dosbox tries to access some file, I will...

This sounds like it could be implemented as a read-only backend that fetches data from the network combined with a layering virtual backend that reads missing data from the network backend and writes modifications to a second backend.

Overall your planned changes sound like a good match for WasmFS. The core WasmFS backend API is still in flux, but once it is stabilized, we could certainly use help documenting it, providing utilities to make it easy to write new backends from C, C++, or JS, and writing general-purpose virtual backends for caching, layering, notifying on changes, etc.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAKOm%3DVHwC_W%2BkScuLm0U0m4UF4vCJ6ZUzF7D1F4AJZosrKx1kQ%40mail.gmail.com.

[Александр Гурьянов]

Clear. I remembered another feature that great to have. Will be great if FS could make a snapshot of opened files, and restore descriptors on next load.

I experimented with snapshotting jsdos. When the user clicks on snapshot, then on next asyncify pause jsdos do  dump of wasm memory to file. On next page load on same asyncify pause jsdos replaces wasm memory with memory from snapshot and resumes.

This work fine until the game need to use fs layer, because opened file descriptors became invalid. So, it will be great if new fs layer can restore them somehow.

Чт, 28 июля 2022 г. в 03:45, 'Thomas Lively' via emscripten-discuss <emscripte...@googlegroups.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAJZD_EVm7x4v%3Dch%3Dzeg_7YOeEWNBQ8UX9HJ-0JeAm_sePuMmMA%40mail.gmail.com.

[Thomas Lively]

Since WasmFS keeps all of its core data structures (including file descriptors) in Wasm memory, snapshotting should work for it without modifications. The only caveat would be that all the backends that use Web APIs keep state in JS and that state would not be captured by the snapshot, so you would be constrained to use the in-memory backend only.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAKOm%3DVFBBPnSs8n2AG3VMcwhA8HPSnCwetCEqvOmYsCEqzW4Sw%40mail.gmail.com.