Q: how to make a "filesystem within a file" inside a golang executable file

[Jason E. Aten]

Given: an existing C language based system (that includes many on-disk files). 

Given: I'm using CGO to call that C code that expects those files on disk.

Goal: I'd like to copy-in/compile-in all the files that would normally live on disk, putting them into the go executable file. 

Rationale: for easy (single executable binary, in classic Go style) distribution of all the required files and directories that attend the CGO called code. 

Constraint: Ideally the C code in the libraries wouldn't have to be modified in order to redirect its access to the filesystem embedded inside the binary.

Q: Is there an existing "filesystem within a file" library for Go?   I'd like OSX and linux support.

I imagine I could use LD_PRELOAD / DYLD_INSERT_LIBRARIES to hook open/read/write calls and redirect them to the internal data. But then I still need to have a "filesystem within a file" directory structure logic implemented that lets me embed that filesystem in the file somewhere. The filesystem can be read-only.

Are there any existing libraries or parts for such a job?

Caveat: I'd prefer to avoid the complexity and installation requirements of a container/Docker/docker data volume.

I'm currently thinking I may need to resort to a single external database file. Perhaps this file is managed by boltdb or bazil, then use LD_PRELOAD -> hook filesystem open(),, read(), write() filesystem calls, and redirect the filesystem operations that are to the "internal paths" to the single-file database.

Seems like a dead-end: I could do some kind of loopback device/mount scheme, but I don't see how to make a loopback device point to just part (say the last half) of a file.

Thanks for your thoughts!

[Tamás Gulácsi]

Easy: create a zip file, append it to the executable, the you only have to locate the executable at runtime.

On linux, you can use systemd-nspawn (see https://chimeracoder.github.io/docker-without-docker/#35). I don't know osx.

Why can'z just extract the fs from the exexutable zip, and start in a chroot?

[Jason E. Aten]

Tamás: thank you for those suggestions.

Additional constraint: A run of the final executable should avoid writing to disk. No unpacking of a filesystem hierarchy to disk should be required.

[Viktor Kojouharov]

write the files' bytes in a go file and use the files as variables.

[Micky]

"encoding/base64"

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ziffusion]

This may be useful.

https://github.com/gchaincl/gotic

It allows you to embed resources within a go executable, which are then accessible as fs objects via the ioutil.ReadFile()interface.

[Jason E. Aten]

Thanks guys for the suggestions.  Here's what I came up with as the closest thing to work with:

https://github.com/bazil/zipfs  

The zipfs demo for Tommi Virtanen's bazil.org/fuse is the closest thing I've found to what I had in mind. I may have to adapt it a little, but it is written in Go and allows legacy C code to read a zipfile that is mounted with fuse, and uses the Go standard library archive/zip to do the reading. With a little adjustment, I think this can be made to work for my purpose.

[Dmitri Shuralyov]

> Q: Is there an existing "filesystem within a file" library for Go?   I'd like OSX and linux support.

I'm not sure if it'll work for your needs/constraints, but consider vfsgen:

https://github.com/shurcooL/vfsgen

It generates Go code that statically embeds the input virtual filesystem (which can be a folder on disk via http.Dir adapter). Content is gzip compressed (only if it makes the given file smaller), but otherwise it's a part of the executable code; it's not something appended to the built binary.

[Tamás Gulácsi]

Nice, but as far as I understand, the main problem is providing access to the embodied tree to the legacy C programs, as an os filesystem.

If he can use FUSE, than he can choose from a multitude of options. If not, only extracting to a tmpfs is the way.

[Jason E. Aten]

Hi Dmitri, 

Yes, vfsgen got me thinking in this direction, it's a very innovative approach. I consider it inspiration.

Tamás has it right though -- I need to supply files to non-go code as well. 

I put together a working system to solve this problem, it lets one create a single binary that can embed C based libraries (like R https://www.r-project.org/) that want to read their sub-libraries from a filesystem. See

https://github.com/glycerine/libzipfs

for the code. You can also use it quite readily to serve web media. There's an included utility 'libzipfs-combiner' which combines your go executable and a given zip archive and gives you back a file that will serve a fuse endpoint from the internalized zipfile.

Enjoy!