Go offline development recommendations

[snmed]

Hi all

Our customer demands an offline development environment with no internet connection, is there any best practices to handle package download and project setup for such an use case? And how would the new go modules fit in in such an environment?

Any advise will be most appreciated.

Cheers Sandro

[Burak Serdar]

On Wed, Dec 12, 2018 at 1:00 PM snmed <sandro....@gmail.com> wrote:
>
> Hi all
>
> Our customer demands an offline development environment with no internet connection, is there any best practices to handle package download and project setup for such an use case? And how would the new go modules fit in in such an environment?

Somebody just mentioned this today. It looks like it is doing what
you're asking for:

https://github.com/gomods/athens

>
> Any advise will be most appreciated.
>
> Cheers Sandro
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[snmed]

Thank you for you reply,

yes i have already read about that project, but as far as I see, there is no offline loading implemented.
But I'm sure it would be doable with some customisation. I wondering if there is another approach for an offline scenario.

Some other ideas or suggestions?

Thanks in advance

Sandro

[Tom Mitchell]

Such a requirement can only be addressed with an air gap bridge.
Are they working with Linux or another platform?

On linux  it is easy to "touch /var/tmp/now"  and "find / -newer /var/tmp/now"
and collect anything that is pulled in locally.  then burn selected  bits to a CDROM.

Users inside the offline isolation booth have to make requests for library and module
source or binaries and communicate possibly with paper for the bits they need.
Dummy, stub programs will make it easy to identify and trigger the downloads of objects. 
No USB sticks...

The CDROM set is an audit trail and allows virus scanning.   

The same is true for any and all system services libraries and more inside the development environment.
That off line development  environment likely needs exactly the same or stricter audit and reproducible setup.
At each release cycle and checkpoint for debugging ... so leverage that same audit and management
systematic process.   

Their ask is bigger than golang.

--

   T o m    M i t c h e l l

[thepud...@gmail.com]

Hi Sandro,

Vendoring is another approach that can work here.

In a pre-modules world, vendoring is fairly well known.

In a modules world, vendoring is still an option. For example, you can see this FAQ here that touches on using 'go mod vendor' with modules:

    https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away

An alternative approach in a modules world is to use the module cache. It can end up with similar benefits as traditional vendoring (and in some ways ends up with a higher fidelity copy), but uses a different technique. This approach is explained as a "Go Modules by Example" walkthrough here:

    https://github.com/go-modules-by-example/index/blob/master/012_modvendor/README.md

Best,

thepudds

[snmed]

Hi thepudds

Thanks for you Reply. Indeed vendoring is an Option, but I'm not sure how long that will be supported. I think i've read about a blog post which says vendoring will be remove from the go tools, but i'm not sure if this still on the Roadmap of the go Team.
I will have a look into the walkthrough you've posted and see if i get some new ideas out of it.

Cheers

[snmed]

Hi Tom

Thanks for your input. Generally speaking, the overall process how to handle development in an air gapped environment is clear and also how i can identify the required files. My question was more about best practices to handle such situation for go project.

At the moment i favour athens as proxy and try to preload all required packages some how.

[Wojciech S. Czarnecki]

On Wed, 12 Dec 2018 11:59:59 -0800 (PST)
snmed <sandro....@gmail.com> wrote:

> Hi all
>
> Our customer demands an offline development environment with no internet
> connection, is there any best practices to handle package download and
> project setup for such an use case?

Such setups are supported by the Go from its onset.

1. DMZ box where `go get` can reach external repositories
2. IDP internal distribution point - with vetted go tools and vetted libs

External downloads are done via the DMZ box then inspected.
After security team OK's something it is moved to the IDP
host from where devs ro mount:

Std tools and libs - to the fixed GOROOT location, say /usr/local/go

Vetted internal libs - to the fixed main GOPATH location, say /usr/local/gours [1]

Vetted third party libs - to the second GOPATH location, say /usr/local/go3p.

Local development takes place in the third GOPATH location,
say /home/${USER}/project/go

On the dev box:

# GOROOT=; not set - it is present in the tools via GOROOT_FINAL
# GOPATH=/internal:/thirdparty:/local [2]
export GOPATH=/usr/local/gours:/usr/local/go3p:/home/${USER}/project/go

[1] Authors of internal libs usually work on shadowmount within
this path, then commit to the reviewers repo from where their vetted
lib goes to the IDP.

[2] Accidental `go get` of some privileged (or in sec rules violation) person
will try to write to the first "/internal" path where either it errs because its read only
or at most it will write to the local disk - then shadowed by the remote mount.

read `go help gopath` for details.

> And how would the new go modules fit in
> in such an environment?

It does not AFAICS. I can not elaborate on this due to the list's CoC.
I just hope that Go team ultimately will not pull GOPATH from under our feet.

> Any advise will be most appreciated.

Hope this helps,


--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Wojciech S. Czarnecki]

On Wed, 12 Dec 2018 13:35:36 -0800 (PST)
thepud...@gmail.com wrote:

> Hi Sandro,
>
> Vendoring is another approach that can work here.

IMO a big No! Vendoring is a footgun (talking secure corporate environment
settings). There are too many ways to get Bob and Anna developers and builder
box to get out of sync. Esp if all use git. Even with fossil-scm central repo and
its autosync caps its still possible.

Note that IMO, though ;)

[Justin Israel]

On Thu, Dec 13, 2018 at 11:19 AM snmed <sandro....@gmail.com> wrote:

Hi thepudds

Thanks for you Reply. Indeed vendoring is an Option, but I'm not sure how long that will be supported. I think i've read about a blog post which says vendoring will be remove from the go tools, but i'm not sure if this still on the Roadmap of the go Team.
I will have a look into the walkthrough you've posted and see if i get some new ideas out of it.

Even if vendoring somehow goes away, it seems like you could achieve the same goal by manually populating the module cache location as you deem fit, and pointing GOPROXY=file:///...   (instead of Athens)

[snmed]

Thank you very much for your reply. It seems to be a possible way to do it, what do you think about the athens way? In my point of view it would be the easiest way as far i can preload the athens cache with all the required packages,
And then the only thing a developer has to do, is to set the GOPROXY to the athens instance.

Cheers

[Tom Mitchell]

On Wed, Dec 12, 2018 at 10:15 PM snmed <sandro....@gmail.com> wrote:

Thank you very much for your reply. It seems to be a possible way to do it, what do you think about the athens way? In my point of view it would be the easiest way as far i can preload the athens cache with all the required packages,
And then the only thing a developer has to do, is to set the GOPROXY to the athens instance.

 

Such setups are supported by the Go from its onset.

1. DMZ box where `go get` can reach external repositories
2. IDP internal distribution point - with vetted go tools and vetted libs

 

The methods in athens seem to be a good starting point.   The challenge is the vetting when a big 
resource like this has a lot of changes often quickly.

Be sure to anchor yourself in a strategy that can scale.
I can see success spawning multiple projects that cannot see each other multiplying 
permutations.   A DMZ box is not an air gap but might be necessary as a first  step in
protect the transfer and stage the process before anything crosses the air gap.

It sounds like a project ... it does sound as if you understand the challenges.
You will depend on Athens so that needs to be tracked as well.
Sounds like fun.
 

[Wojciech S. Czarnecki]

On Wed, 12 Dec 2018 22:15:23 -0800 (PST)
snmed <sandro....@gmail.com> wrote:

> Thank you very much for your reply. It seems to be a possible way to do it,
> what do you think about the athens way?

From the secop pov it'll be a hells gate. Also it does not allow for
vetted binary arifacts as current unix/Go ways do.

> what do you think about the athens way?

1) Athens is in flux. 2) It is yet another complicated piece of software
to analyze and monitor. 3) It again brings all compiling to the local
machine while GOPATH way allows all devs to use binary artifacts built
on the hardened builder machine.

> In my point of view it would be the easiest way as far i can preload the

> athens cache with all the required packages.

So the security team will need to produce an internal vetted package instead
of signing a tag within the IDP 3rd party package repo.

(IMO whole idea of "zipped packages" is the bad J-flu infection... Ah - CoC)

> And then the only thing a developer has to do, is to set the GOPROXY to the
> athens instance.

It fits loose distributed settings. Not controlled ones. And I -- from
"offline"/"airgap" constraint -- assumed that your client is concerned about
security, not about connectivity.

[snmed]

I'm not sure if i fully understand your point on "vetted binaries", but if every source code is vetted and then transferred to the isolated environment, there should not be a problem with security issues. All the developer machine living already in the same isolated environment and also i would place athens there, so all builds will be made with vetted source code. 

It's easily possible that i miss some import point in this scenario, but anyway i will verify your idea and take it into account for our go development strategy.

[akshita babel]

Hey, can anyone guide me on how to take octet stream as input in API and/or how to convert octet stream to byte array using golang

[Daniel Theophanes]

Hi,

Vendoring will be around a long time. Support for vendoinrg + modules is getting better with the 1.12 release even.

If you need to develop offline, that is the way to go. Also, vendoring is not a footgun. You check in your vendor directory and that ensures all developers and CI tools are on the exact same page.

If this situation, you either need to do per project vendoring or per environment vendoring like athens. Both just take similar project structure.

-Daniel

[Ian Lance Taylor]

On Thu, Dec 13, 2018 at 4:00 AM akshita babel
<akshitab...@gmail.com> wrote:
>
> Hey, can anyone guide me on how to take octet stream as input in API and/or how to convert octet stream to byte array using golang

If you want to ask an unrelated question, please send a new message,
rather than replying to an existing thread. Thanks.

Ian

[Jeff Kowalczyk]

On Thursday, December 13, 2018 at 11:27:40 AM UTC-8, Daniel Theophanes wrote:

Vendoring will be around a long time. Support for vendoinrg + modules is getting better with the 1.12 release even.

If you need to develop offline, that is the way to go. Also, vendoring is not a footgun. You check in your vendor directory and that ensures all developers and CI tools are on the exact same page.

That is excellent to hear. The ability to build on a fully-offline single host is expected in some environments.

Can you describe or link to the upcoming 1.12 changes to module vendoring?

Jeff

[Nathan Fisher]

We use dep at work and commit the vendor folder. The main benefit we see is that it ensures consistent builds across machines, tends to be faster, and allows offline development. assuming you don’t have to use a third party security or infrastructure team to download the dependencies. If you do then it can be a bit of a nuisance because they need the tooling on their machine. Committing the vendor folder is a lot less effort than alternative solutions from my experience in similarly restrictive environments.

--

- sent from my mobile

[snmed]

Indeed vendoring is a possible way we will consider that option but with go mod instead with dep. I strongly believe that the module way go is heading is the right way, but i'd like to have more documentation about using go modules in an air gapped network environment.

Anyway thank you for your input