Haskell Stack/Stackage Local Server

[David Sicilia]

Hi there,

I am behind a firewall with strict rules about downloading software from the internet,

so I would like to know if it is possible to download an entire stackage LTS snapshot

in one go and then serve it on a local server, to which we could then point the stack tool.

By "snapshot" I would be referring to the source code for all packages because

we'd need to draw from that local server to build on both Linux and Windows.

For example, perhaps we'd be able to go to a server within the firewall and run

"stack new", then edit the yaml file to point it to a local stackage server with given

resolver number, then it would just behave normally from then on, except always

downloading packages from the local server.

What about compilers? I know that stack also downloads the compilers, so I guess

those would have to be included in the downloaded snapshot-package for various

platforms?

Any guidance would be appreciated, because we will not be able to make use of

stack/stackage without something like this!

Thank you

David

[Neil Mitchell]

Hi David,

Yes, this is possible. It didn't used to work fully, but now it does,
and is merely a little wonky in the corners.

The magic all lies behind stack setup --stack-setup-yaml with which
you can point at a config.yaml which says where to get each element.
If you download the elements in advance and serve them up from a fresh
(local) http server it will work. In addition I've found that copying
the config.yaml into the stack root makes things work better (reasons
not really clear...).

There are certainly ways this could be easier - for example if the
stack.yaml could point at the setup-yaml. Currently if you do stack
build before stack setup with the right setup flag then everything
gets a bit wonky and deleting stack root seems advisable.

See https://docs.haskellstack.org/en/stable/yaml_configuration/#non-project-specific-config
for some docs. A how-to guide for your situation would have been very
useful when I attempted it, as I imagine inside some organisations
it's quite a common thing to do.

Note that there's no automatic way of mirroring just the packages you
need, yet...

Thanks, Neil

> --
> You received this message because you are subscribed to the Google Groups
> "haskell-stack" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to haskell-stac...@googlegroups.com.
> To post to this group, send email to haskel...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/haskell-stack/0629a320-d923-47ae-9b8e-8d1828e5b66b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Sicilia]

Hi Neil,

Thanks very much for the reply, I'll take a look at what you have suggested below!

David

[Neil Mitchell]

Since I had such difficulty with this option, I tried on a fresh
virtual machine, trying to go the full route to a working stack
installation using only a localhost mirroring server with the network
disabled. I didn't get very far, and managed to totally confuse
myself. There are two essential files, config.yaml (stored in
STACK_ROOT) and setup-info.yaml (passed on the command line or in the
stack.yaml file). I think it would be a reasonable design decision to
merge them, and I've raised a ticket for that at
https://github.com/commercialhaskell/stack/issues/2982.

I would suggest the restriction you mention in large commercial
organisations, and Stack could certainly be better set up for it.

Thanks, Neil

> --
> You received this message because you are subscribed to the Google Groups
> "haskell-stack" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to haskell-stac...@googlegroups.com.
> To post to this group, send email to haskel...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/haskell-stack/cf1530fe-ff7b-4a5a-ae6a-0e3e5d2251b1%40googlegroups.com.

[David Sicilia]

Hi Neil,

Thank you for raising the ticket, I just gave it a +1, if that helps :-)

So would you recommend that I go ahead and try it right now,

or is it complicated enough that I should just wait for that change

to be implemented?  I guess what I mean is, after you realized

which config files to change, was it straightforward from there?

David

On Saturday, January 21, 2017 at 11:27:22 AM UTC-5, David Sicilia wrote:

[Neil Mitchell]

Hi David,

Its not that bad, now the steps are known, so go ahead and email me if you get stuck. Steps are:

* create stack root

* copy config.yaml to the root

* run stack setup with the setup yaml flag

Figuring our these steps took weeks :)

Thanks, Neil 

--

You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/35616520-204d-41f6-b1c9-e37ee1570473%40googlegroups.com.

[Alexander T]

Please, can you describe all operations in detail. Not sure if google groups is a good place for this though.

The whole idea of using haskell with stack/cabal offline is crucial for work at many companies.

First of all it's not clear how to obtain all the data needed for stack and cabal to serve locally.

[Neil Mitchell]

> Please, can you describe all operations in detail.

Describing is fragile. I've created a github project that describes
it. See the README at
https://github.com/ndmitchell/offline-stack#readme, plus the code
which executes it at
https://github.com/ndmitchell/offline-stack/blob/master/appveyor.yml.
I test those steps on Appveyor, so it's guaranteed to work/

> The whole idea of using haskell with stack/cabal offline is crucial for work
> at many companies.
> First of all it's not clear how to obtain all the data needed for stack and
> cabal to serve locally.

Agreed! Hopefully everything is now 100% clear.

> https://groups.google.com/d/msgid/haskell-stack/094da0e4-6eab-4c10-8842-4d3c37b3ad78%40googlegroups.com.

[Alexander T]

Thank you. I tried this, http-redirect server builds fine and runs. This is what I figured out.

Once I did "stack setup", "stack build" through the proxy (with corresponding config.yaml), I can reproduce it on the same system without internet connection.

But if I take a fresh system with empty global stack root (e.g. ~/.stack on GNU/Linux) except solely config.yaml, it immediately fails. http-redirect throws this:

HttpExceptionRequest Request {

  host                 = "raw.githubusercontent.com"

  port                 = 443

  secure               = True

  requestHeaders       = []

  path                 = "/fpco/lts-haskell/master//lts-8.0.yaml"

  queryString          = ""

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

Interestingly, the file actually exists in the mirror directory: "mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml". 

"stack new" fails in both cases with:

HttpExceptionRequest Request {

  host                 = "raw.githubusercontent.com"

  port                 = 443

  secure               = True

  requestHeaders       = [("If-None-Match","\"8ef96691e15d68bb890626821e8624b1b19e0b6c\"")]

  path                 = "/commercialhaskell/stack-templates/master/new-template.hsfiles"

  queryString          = ""

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

Here I can see that this request is not processed by the http-redirect, it keeps silent when stack throws this exception.

Maybe I missed something?

[Neil Mitchell]

> Thank you. I tried this, http-redirect server builds fine and runs. This is
> what I figured out.

> Once I did "stack setup", "stack build" through the proxy (with
> corresponding config.yaml), I can reproduce it on the same system without
> internet connection.
> But if I take a fresh system with empty global stack root (e.g. ~/.stack on
> GNU/Linux) except solely config.yaml, it immediately fails. http-redirect
> throws this:

Did you copy the config.yaml from the offline-stack repo into
STACK_ROOT? Is your STACK_ROOT actually where you think it is? (use
"stack path" to see).

> "stack new" fails in both cases with:

> Here I can see that this request is not processed by the http-redirect, it
> keeps silent when stack throws this exception.
>
> Maybe I missed something?

stack new isn't supported - I imagine you have to have another
redirect in config.yaml. Pull requests welcome! I've also raised a
ticket at https://github.com/ndmitchell/offline-stack/issues/1.

Thanks, Neil

> https://groups.google.com/d/msgid/haskell-stack/674862b8-2d46-4c6e-ac93-0b1cdfde0ec5%40googlegroups.com.

[Simon Jakobi]

Hi!

2017-02-17 3:21 GMT+01:00 Alexander T <alexo...@gmail.com>:

"stack new" fails in both cases with:

With the usual `stack new` syntax of `stack new proj` or `stack new proj hspec`, stack attempts to download the template from GitHub. However you can also specify a filepath or URL as the template source, e.g. `stack new proj my-template.hsfiles` or `stack new proj http://example.com/template.hsfiles`.

Cheers,

Simon

[Alexander T]

Yes I did.

Server side

$ env | grep STACK_ROOT

STACK_ROOT=/data/srv/stack_root

$ pwd

/data/srv

$ ls

http-redirect  mirror  stack_root

$ stack path | grep config

config-location: /data/srv/stack_root/global-project/stack.yaml

$ cat /data/srv/stack_root/config.yaml

urls:

  latest-snapshot: http://localhost:3000/https/www.stackage.org/download/snapshots.json

  lts-build-plans: http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/

  nightly-build-plans: http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/

package-indices:

- name: Hackage

  download-prefix: http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/

  http: http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz

Client side

$ env | grep STACK_ROOT

$ cat ~/.stack/config.yaml

urls:

  latest-snapshot: http://localhost:3000/https/www.stackage.org/download/snapshots.json

  lts-build-plans: http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/

  nightly-build-plans: http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/

package-indices:

- name: Hackage

  download-prefix: http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/

  http: http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz

When I try to run "stack path" on client side without internet connection on both sides

$ stack path

Downloading nightly-2017-02-02 build plan ...RedownloadFailed Request {

  host                 = "localhost"

  port                 = 3000

  secure               = False

  requestHeaders       = []

  path                 = "/https/raw.githubusercontent.com/fpco/stackage-nightly/master//nightly-2017-02-02.yaml"

  queryString          = ""

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

 "/home/alex/.stack/build-plan/nightly-2017-02-02.yaml" (Response {responseStatus = Status {statusCode = 500, statusMessage = "Internal Server Error"}, responseVersion = HTTP/1.0, responseHeaders = [("Date","Sun, 19 Feb 2017 01:23:48 GMT"),("Server","Warp/3.2.11"),("Content-Type","text/plain; charset=utf-8")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose})

At the same time http-redirect server throws:

HttpExceptionRequest Request {

  host                 = "raw.githubusercontent.com"

  port                 = 443

  secure               = True

  requestHeaders       = []

  path                 = "/fpco/stackage-nightly/master//nightly-2017-02-02.yaml"

  queryString          = ""

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

[Alexander T]

Ah, also forgot to clarify that the nightly-2017-02-02.yaml file was already previously retrieved by the http-redirect server and actually exists in it's mirror directory (with filename renamed according to server's conventions).

Server side

$ stat mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml

  File: mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml

  Size: 5377585         Blocks: 10504      IO Block: 4096   regular file

Device: 2dh/45d Inode: 5819        Links: 1

Access: (0644/-rw-r--r--)  Uid: ( 1004/    alex)   Gid: (  100/   users)

Access: 2017-02-17 05:02:18.946927356 +0300

Modify: 2017-02-17 05:02:18.936927354 +0300

Change: 2017-02-19 03:30:53.923441888 +0300

 Birth: -

[Neil Mitchell]

The fact that you can't do "stack new" is known, and raised at
https://github.com/commercialhaskell/stack/issues/2804.

Note that the http-redirect tool I wrote _always_ goes to the
internet, it never serves up mirrored content. If you want to run on a
server which doesn't have internet access you'll need to modify that
code, or spin up another type of server to serve up the content. The
purpose of http-redirect is to robustly test my configuration steps -
I didn't write it with the idea of production use in mind.

Thanks, Neil

> https://groups.google.com/d/msgid/haskell-stack/17cc8023-090c-4cf8-982e-c8e743df65ed%40googlegroups.com.

[Alexander T]

Yeah, I understood that "stack new" is not supported atm.

The problem is with "stack build", even with "stack path", etc.

The name of github repository "offline-stack" is a bit misleading. And repository descriptions "Install Stack without internet access", "demo of using Haskell Stack without internet access" confuse further.

I thought that the http-redirect server is able to run offline, i.e. without internet access at all once all the required files were downloaded by passing requests through proxy.

Thank you for all your work in this direction.

So, do you think it is possible to turn this idea into an offline server? Do you see any pitfalls building up such system for production? Like, does the theory of idea need major adjustments in your opinion?

[Neil Mitchell]

> The name of github repository "offline-stack" is a bit misleading. And
> repository descriptions "Install Stack without internet access", "demo of
> using Haskell Stack without internet access" confuse further.

Sorry, I'll tweak the text. It's meant to be a demo of the steps you
need to install stack without internet access - not a ready rolled
solution. In particular, it isn't the latter because:

1) If your machine has full internet access then this solution isn't
required. You have to have some way to initially get the files onto
the machine, and that will vary by location, so there's no chance of
doing something standard.

2) Most places have existing file serving servers, and in my case I
just reused one of them. No point firing up a new Haskell server just
to serve up a few files, when putting it on an Apache/IIS server is
probably easier.

> I thought that the http-redirect server is able to run offline, i.e. without
> internet access at all once all the required files were downloaded by
> passing requests through proxy.

You could easily do that - in fact I've just pushed a 1 line change to
do that - https://github.com/ndmitchell/offline-stack/commit/86c6ff60dc7df9e20363c1e4f4cbcd79a6543d2f
(entirely untested).

> Thank you for all your work in this direction.
> So, do you think it is possible to turn this idea into an offline server? Do
> you see any pitfalls building up such system for production? Like, does the
> theory of idea need major adjustments in your opinion?

In my case, step 1 doesn't work, and step 2 is more hassle than
reusing existing solutions. However, if you can get around step 1
(which it seems like you can), and step 2 is no big deal, then it
should work.

Thanks, Neil

> https://groups.google.com/d/msgid/haskell-stack/c2b35b88-e1f7-4a87-adf1-0ac423a4d53a%40googlegroups.com.

[Alexander T]

Hi again!

I tried your commit that skips downloading if file is present and it works.

However, imho this is still a suboptimal solution.

This is what I came up with. A script that downloads every url from stack setup

yaml and all packages from hackage index. Here is the repository:

https://github.com/AleXoundOS/haskell-dev-mirror-script . In other words, you

get everything you need to develop software in Haskell except remote docs.

After everything is downloaded it uses 20GiB of space and takes about 15

minutes to verify files integrity. Not that much. Tbh, I was afraid it will take

more space and time.

Downloaded files are placed into proper subdirectories in "mirror" directory

that is ready to be served by http server. Clients need to get generated

config.yaml with all the paths adjusted to the structure of "mirror" directory.

And finally, to bypass the well-known stack bug ignoring "setup-info" field,

clients need the generated stack setup yaml for "stack setup" operations.

[Alexander T]

Renamed repository:

https://github.com/AleXoundOS/haskell-stack-mirror-script

[Neil Mitchell]

Thanks, I've linked to that from my offline-stack repo.

> https://groups.google.com/d/msgid/haskell-stack/34ac3637-269f-4193-b8e6-1c003e757427%40googlegroups.com.

[Neil Mitchell]

Out of curiosity, how much is the disk space required for all the
packages alone, without the compilers? Are 1 or 2 packages a
significant portion of that?

[Alexander T]

$ du -hs mirror/*

23M     mirror/00-index.tar.gz

3.3G    mirror/build-plans

7.8G    mirror/packages

4.0K    mirror/snapshots.json

8.1G    mirror/stack

32K     mirror/stack-setup-2.yaml

28K     mirror/stack-setup-mirror.yaml

> Are 1 or 2 packages a significant portion of that? 

No. But it also depends on the package. The biggest package I've seen is 44

MiB. There are 77k+ packages. Dunno if it tells something useful, the median

value of sorted sizes list is 13006 bytes. As expected, source packages do not

use much disk space.

Unexpectedly, build plans use quite a lot of space. But build plans yaml files

are transferred over Git in compressed form, so use less traffic than on-disk

size. Also they greatly benefit from using a filesystem with transparent

compression like Btrfs (approximately saves 60% of disk space in my case).