Attach files to compiled go program

[Konstantin Kanellopoulos]

Hi,

I spend some evenings to this idea and would appriciate your feedback.

I'm a fairly fresh gopher, so please me tell if I get something fundamentally wrong here.

The Idea

I really like having a single executable after compiling my go code.
This stops with multiple non-go files which your app relies on.
i.e. Templates, Images, Javascript, CSS, etc ...

I want to append all files as tar(.gz?) to my executable and make them
available for the application at runtime.

I know there's allready an other solution which converts files into go code
but I'd like to be able to change the appended data without recompiling and
maybe also compress them.


The API

Currently I try to make the following work:

package main

import(
    "fmt"
    "io/ioutil"
   
    // I currently called it bpack which is a abbrevation for backpack
    "bpack"
)

func main() {
    // The backpack returns io.Reader for the requested content
    r, err := bpack.Get("/public/js/jquer.min.js")
    if err != nil {
        // handle
    }
   
    content, err:= ioutil.ReadAll(r)
    if err != nil {
        // handle
    }
   
    fmt.Println("%s\n", content)
}

If someone want's to join me, I will put the code on github.

Thanks,
Konsi

[Rodrigo Kochenburger]

There is a couple of other implementations of this already. For example: https://github.com/GeertJohan/go.ricec

- RK

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hotei]

I think the Rodrigo's post should have ended with "go.rice" 

[Ondekoza]

This already exists:

https://github.com/jteeuwen/go-bindata

On Tuesday, April 1, 2014 7:08:23 PM UTC-4, Konstantin Kanellopoulos wrote:

[Konstantin Kanellopoulos]

I know there's allready an other solution which converts files into go code
but I'd like to be able to change the appended data without recompiling and
maybe also compress them.

 

Thank you. I allready knew about that ;-)

[Konstantin Kanellopoulos]

Hi,

thank you for your link. I looked into it, but still feel that is should roll
my own implementation. (at least for sake of learninig go)

Rice uses "Boxes" to organize the files. It feels a little bit overengineered to me.
I*d prever something more simple. Retrieve a file by a path. If its not in the backpack, load it from filesystem.

[Tamás Gulácsi]

It isn't that complicated: mentally replace "Box" with "SubDir".
Please try box, and help develop it! We already have at least 5 such half-finished projects!

[Dobrosław Żybort]

> I*d prever something more simple. Retrieve a file by a path. If its not in the backpack, load it from filesystem.

That's exactly what go.rice is doing:
https://github.com/GeertJohan/go.rice#gorice

[David Skinner]

I am using http://godoc.org/gopkg.in/cookieo9/resources-go.v2 and have had no problems so far.

[Miki Tebeka]

There's also nrsc ;)

[Konstantin Kanellopoulos]

Am Donnerstag, 3. April 2014 03:06:20 UTC+2 schrieb David Skinner:

I am using http://godoc.org/gopkg.in/cookieo9/resources-go.v2 and have had no problems so far.

That's exactly what I wanted. Thank you ;-)

[Geert-Johan Riemer]

I like your idea of tar.gz, I don't know why I didn't think of that..
The current append feature in rice requires zip to be installed, which causes problems on windows.
I wonder if tar.gz can do the same as zip, without the need for an external command.

There's a TODO to add "Single's" (single file), beside Boxes (folders).
The complete file virtualization can stay the same, but it needs implementation in the tool and go.rice package to load the 'Single' run-time.

Currently, rice allows for "embedding" (generating go source containing the resources) and "appending" (adding zip to exec).
I'm thinking about adding feature to generate object files that the linker will add to the executable.
The whole idea of go.rice is that resources can be added to an executable in several ways, without having to change the code depending on those resources.

/GeertJohan

[Tamás Gulácsi]

Zip: the file listing is at the file's end, thus you can append a zip to anything and after fixing the offsets, the file can be read from the beginning as usual, or as a zip (from the end).

Tar is a streaming format, thus you need to read the whole to know what is in there. But if you invent your own zip format, then it is possible.

[Jens Alfke]

On Wednesday, April 2, 2014 6:06:20 PM UTC-7, David Skinner wrote:

I am using http://godoc.org/gopkg.in/cookieo9/resources-go.v2 and have had no problems so far.

This looks useful, but the docs are somewhat vague — it says resources can be loaded from "a zip file embedded in the executable", which is exactly what I'd want, but I couldn't find anything explaining how you embed a zip file in a Go executable. Can someone explain?

—Jens

[Tamás Gulácsi]

Go build && cat exe zip >exe2 && zip -A exe2

[Jens Alfke]

On Apr 3, 2014, at 8:30 AM, Tamás Gulácsi <tgula...@gmail.com> wrote:

Go build && cat exe zip >exe2 && zip -A exe2

OK, this appends the Zip archive onto the binary. Does this work on all architectures? I.e. do the loaders for Mach-O, ELF, Windows, etc. all ignore extra data past the end of the binary?

—Jens

[Tamás Gulácsi]

AFAIK all starts reading from the beginning and no thing points after the executables end.

[David Skinner]

The readme file on github gave explicit insturctions. https://github.com/cookieo9/resources-go/tree/v2

I am currently only using Debian sid AMD64. Cannot vouch for other cases.

On Thu, Apr 3, 2014 at 11:20 AM, Tamás Gulácsi <tgula...@gmail.com> wrote:

AFAIK all starts reading from the beginning and no thing points after the executables end.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/_LszV5IqBWg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Gyepi SAM]

I am most familiar with the ELF format, but those are all structured files and
have a length value encoded in there somewhere so yes, the loaders should ignore
the extra data.

In any case, that should be easy enough to verify with a small program and a
test zip.

Also, note that the Go stdlibs contain readers for all three formats if you
need to manipulate the files or just read the docs.

-Gyepi