WebAssembly and Filesystem access
437 views
Skip to first unread message
Kevin Chowski
Nov 4, 2022, 5:08:09 PM11/4/22
to golang-nuts
Hey all,
I couldn't find a prior thread or other information on the web after a bit of searching, but if this is answered elsewhere I'd appreciate a link to follow.
I am on a project which primarily ships a Go command line interface (CLI), but we have aspirations of using the wasm compilation mode to also distribute a simple webapp version of it, while sharing most of the code.
Currently simple things work perfectly. (To everyone involved with this: great work!) The next big hurdle is the fact that we cache things on disk for later reuse, so things are failing out-of-the-box any time we attempt to touch the filesystem. Luckily, in our case, the fact that we are touching the filesystem is a bit incidental (its used as a cache for making consecutive CLI invocations faster), and as long as the system was consistent for the lifetime of the main Go process things will work just fine.
We /could/ pass a filesystem object across the codebase, and maybe we will one day we will anyway for some other reasons, but I'd rather not have to do that just to get further with this webapp prototype: it's a lot of tedious plumbing, and there is a nontrivial amount of code to touch.
So instead I decided to try some global-mutable-at-startup variables for things like os.OpenFile, os.Remove, etc, that should be easy to cleanup if we decide not to move forward with the prototype; but even then, there are plenty of random things I have to do to get this to work. I've spent about 2 hours on this direction so far and I keep hitting unexpected roadblocks - thus this email seeking out a new strategy.
Are there any plans to automatically support filesystems on wasm-compiled Go applications? Even an ephemeral, in-memory filesystem would basically solve all of my problems without having to change any code on my end, which would be nice.
In lieu of that, does anyone have any tips about how I could go about doing this in a better way?
-Kevin
Brian Candler
Nov 4, 2022, 6:25:40 PM11/4/22
to golang-nuts
Not sure this is what you're looking for, but Go 1.16 introduced an abstract filesystem interface:
"the new testing/fstest package provides a TestFS function that checks for and reports common mistakes. It also provides a simple in-memory file system implementation, MapFS, which can be useful for testing code that accepts fs.FS implementations."
Konstantin Khomoutov
Nov 5, 2022, 7:59:08 AM11/5/22
to golang-nuts
On Fri, Nov 04, 2022 at 02:08:08PM -0700, 'Kevin Chowski' via golang-nuts wrote:
[...]
these two facts:
- The os package was implemented as a sort-of grab bag of many features
one expects on a typical OS (hence the name). The fact the filesystem
operations were implemented in that package directly and not elsewhere
is the direct manifestation of that approach.
An environment in which WASM runs in a browser is too different from that
provided by a typical OS, so I cannot see how one could sensibly implement
in GOOS=js.
- Even if some sort of FS emulation were to be implemented for GOOS=js,
the question is: which one exactly? Keeping stuff in memory in just one
way of doing things. While I'm not a web developer, I know of at least
session storage and local storage which provide for two levels of
persistence beyond keeping stuff in memory. Which one to pick for
implementing stuff from the os package?
> In lieu of that, does anyone have any tips about how I could go about doing
> this in a better way?
If you need a hard-and-fast solution, I'd propose you're trying to abstract
your stuff away on a wrong level. I think it would be cleaner to factor out
your persistence code into a set of functions "store this stuff" and "load
that stuff"; the default implementations would call out to the os package, and
the wasm implementation would use either approach you select: keeping things
in memory, using local storage etc.
I mean, trying to implement and pass around through the whole codebase
something like a "filesystem accessor" switchable at runtime is only worth it
if your code makes heavy use of the filesystem in nontrivial way. I'd say that
it's better to abstract away high-level store/load operations.
[...]
> I am on a project which primarily ships a Go command line interface (CLI),
> but we have aspirations of using the wasm compilation mode to also
> distribute a simple webapp version of it, while sharing most of the code.
[...]
> but we have aspirations of using the wasm compilation mode to also
> distribute a simple webapp version of it, while sharing most of the code.
> The next big hurdle is the fact that we cache things on disk
> for later reuse, so things are failing out-of-the-box any time we attempt
> to touch the filesystem. Luckily, in our case, the fact that we are
> touching the filesystem is a bit incidental (its used as a cache for making
> consecutive CLI invocations faster), and as long as the system was
> consistent for the lifetime of the main Go process things will work just
> fine.
>
> We /could/ pass a filesystem object across the codebase, and maybe we will
> one day we will anyway for some other reasons, but I'd rather not have to
> do that just to get further with this webapp prototype: it's a lot of
> tedious plumbing, and there is a nontrivial amount of code to touch.
>
> So instead I decided to try some global-mutable-at-startup variables for
> things like os.OpenFile, os.Remove, etc, that should be easy to cleanup if
> we decide not to move forward with the prototype; but even then, there are
> plenty of random things I have to do to get this to work. I've spent about
> 2 hours on this direction so far and I keep hitting unexpected roadblocks -
> thus this email seeking out a new strategy.
>
> Are there any plans to automatically support filesystems on wasm-compiled
> Go applications? Even an ephemeral, in-memory filesystem would basically
> solve all of my problems without having to change any code on my end, which
> would be nice.
I'm not a Go developer, but I'd say it is unlikely due to the combination of
> for later reuse, so things are failing out-of-the-box any time we attempt
> to touch the filesystem. Luckily, in our case, the fact that we are
> touching the filesystem is a bit incidental (its used as a cache for making
> consecutive CLI invocations faster), and as long as the system was
> consistent for the lifetime of the main Go process things will work just
> fine.
>
> We /could/ pass a filesystem object across the codebase, and maybe we will
> one day we will anyway for some other reasons, but I'd rather not have to
> do that just to get further with this webapp prototype: it's a lot of
> tedious plumbing, and there is a nontrivial amount of code to touch.
>
> So instead I decided to try some global-mutable-at-startup variables for
> things like os.OpenFile, os.Remove, etc, that should be easy to cleanup if
> we decide not to move forward with the prototype; but even then, there are
> plenty of random things I have to do to get this to work. I've spent about
> 2 hours on this direction so far and I keep hitting unexpected roadblocks -
> thus this email seeking out a new strategy.
>
> Are there any plans to automatically support filesystems on wasm-compiled
> Go applications? Even an ephemeral, in-memory filesystem would basically
> solve all of my problems without having to change any code on my end, which
> would be nice.
these two facts:
- The os package was implemented as a sort-of grab bag of many features
one expects on a typical OS (hence the name). The fact the filesystem
operations were implemented in that package directly and not elsewhere
is the direct manifestation of that approach.
An environment in which WASM runs in a browser is too different from that
provided by a typical OS, so I cannot see how one could sensibly implement
in GOOS=js.
- Even if some sort of FS emulation were to be implemented for GOOS=js,
the question is: which one exactly? Keeping stuff in memory in just one
way of doing things. While I'm not a web developer, I know of at least
session storage and local storage which provide for two levels of
persistence beyond keeping stuff in memory. Which one to pick for
implementing stuff from the os package?
> In lieu of that, does anyone have any tips about how I could go about doing
> this in a better way?
your stuff away on a wrong level. I think it would be cleaner to factor out
your persistence code into a set of functions "store this stuff" and "load
that stuff"; the default implementations would call out to the os package, and
the wasm implementation would use either approach you select: keeping things
in memory, using local storage etc.
I mean, trying to implement and pass around through the whole codebase
something like a "filesystem accessor" switchable at runtime is only worth it
if your code makes heavy use of the filesystem in nontrivial way. I'd say that
it's better to abstract away high-level store/load operations.
Kevin Chowski
Dec 16, 2022, 5:10:14 PM12/16/22
to golang-nuts
I recently learned that WASI (https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md) supports filesystem abstractions directly for WASM code.
Is there an plan to integrate this into Go?
Raffaele Sena
Dec 16, 2022, 6:01:40 PM12/16/22
to Kevin Chowski, golang-nuts
tinygo now has a "wasi" target that you can try.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dd8306ab-75bf-4bb6-a9ac-3c9f778665a7n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages