nix-prefetch-github

[Sebastian Jordan]

Hi,

I wrote a small script to "prefetch" github git repositories and also to automatically calculate the appropriate arguments for nixpkgs.fetchFromGitHub.  I thought I'll share my script with the community.  The source code can be found on PyPI and GitHub and thus can be installed via pip.

If you might be interested in using this, feel free to do so.  Feedback is welcome and appreciated.

Cheers,

Sebastian

[Matthias Beyer]

Hi,

On 19-04-2018 13:50:27, 'Sebastian Jordan' via nix-devel wrote:
> "prefetch" github git repositories and also to > automatically
> calculate the appropriate arguments for
> nixpkgs.fetchFromGitHub.

awesome!

> can be installed via pip.

Care to put it into a nixpkg? :-) I'm not aware of any tool like this
in nixpkgs yet, so I'd welcome if we had one!

--
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Consider switching to free software.
It adds value to your life.
https://www.gnu.org/

[Silvan Mosberger]

Hi Sebastian

I don't mean any harm, but what's the difference between this and using `nix-prefetch-git --unpack github.com/<owner>/<repo>/archive/<rev>.tar.gz` to get the hash and prefetch it? Maybe I just don't see the full picture yet, but I feel like a simple bash script could do that in a couple lines with this.

Silvan

[Sebastian Jordan]

Hi Silvan,

You would get not only the "hash" but also the latest commit sha aka "rev".  This script uses the github API to find out the latest commit "rev" and falls back to "nix-prefetch-git" if the github API rate limit is reached which can be nice if you use it on a travis-ci server or something similar.  And yes, this all can be done with a bash script, but I don't see why everybody should write their own bash script every time they need to do this kind of task. Thank you for your feedback :) much appreciated :)

Cheers,

Sebastian

--
You received this message because you are subscribed to the Google Groups "nix-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nix-devel+...@googlegroups.com.
To post to this group, send email to nix-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/60A74B10-A7E2-4061-867D-19EAAF76FBAA%40icloud.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Simons]

Hi Sebastian,

> You would get not only the "hash" but also the latest commit sha aka "rev".

I might be missing something, but this information is readily available
in many ways, for example:

$ git ls-remote git://github.com/NixOS/cabal2nix.git master
7882850b0182d64765b242e19f42642249fcab00 refs/heads/master

Best regards,
Peter

[Danylo Hlynskyi]

I've tried to implement "simple" Bash script based on comments above, but I think hash computation for fetchFromGitHub isn't straightforward

https://gist.github.com/danbst/1b7d6b97401ebdc34df0a03b8f3a1424

There are 2 ways of hash computing:
1) hash=$(nix-prefetch-url "https://github.com/$project/archive/$rev.tar.gz")
2) hash=$(nix-prefetch-git "github.com/$project.git" --rev "$rev" --quiet | jq -r .sha256)

The output is:

$ ./prefetch-github.sh "NixOS/hydra" b7bc4384b7b471d1ddf892cb03f16189a66d5a0d
{
    "owner": "NixOS",
    "repo": "hydra",
    "rev": "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d",
    "sha256": "1ri2fcpqh6cfgcbcl32ghsr711iicr2y2m09rr84i4hb538dmhf7"
}
{
    "owner": "NixOS",
    "repo": "hydra",
    "rev": "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d",
    "sha256": "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
}

But should be

{
    owner = "NixOS";
    repo = "hydra";
    rev = "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d";
    sha256 = "05g37z3ilazzqa5rqj5zljndwxjbvpc18xibh6jlwjwpvg3kpbbh";
}

(as per https://github.com/NixOS/nixpkgs/blob/192221ae3f2493f1e47a77726059dc4a65a9800d/pkgs/development/tools/misc/hydra/default.nix#L69-L74)

To be honest, I can't run the OPs nix-prefetch-github inside virtualenv:

$ nix-prefetch-github NixOS hydra
Traceback (most recent call last):
  File "/home/danbst/venv-nix/bin/nix-prefetch-github", line 7, in <module>
    from nix_prefetch_github import _main
  File "/home/danbst/venv-nix/lib/python2.7/site-packages/nix_prefetch_github/__init__.py", line 110
    return commit_info['sha']
SyntaxError: 'return' with argument inside generator

--
You received this message because you are subscribed to the Google Groups "nix-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nix-devel+unsubscribe@googlegroups.com.

To post to this group, send email to nix-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/87vacm6y74.fsf%40write-only.cryp.to.

[Tuomas Tynkkynen]

Hi,

On Friday, April 20, 2018 at 1:00:27 PM UTC+3, Danylo Hlynskyi wrote:

I've tried to implement "simple" Bash script based on comments above, but I think hash computation for fetchFromGitHub isn't straightforward

https://gist.github.com/danbst/1b7d6b97401ebdc34df0a03b8f3a1424

There are 2 ways of hash computing:
1) hash=$(nix-prefetch-url "https://github.com/$project/archive/$rev.tar.gz")
2) hash=$(nix-prefetch-git "github.com/$project.git" --rev "$rev" --quiet | jq -r .sha256)

The output is:

<snip 2 wrong outputs> 

But should be

{
    owner = "NixOS";
    repo = "hydra";
    rev = "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d";
    sha256 = "05g37z3ilazzqa5rqj5zljndwxjbvpc18xibh6jlwjwpvg3kpbbh";
}

(as per https://github.com/NixOS/nixpkgs/blob/192221ae3f2493f1e47a77726059dc4a65a9800d/pkgs/development/tools/misc/hydra/default.nix#L69-L74)

The 100% fool-proof way is to generate the same Nix expression you'd be generating but with a bogus hash, and then parse it from the error message:

$ nix-build -E 'with import <nixpkgs> {}; fetchFromGitHub { owner = "NixOS"; repo = "hydra"; rev = "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d"; sha256 = "0000000000000000000000000000000000000000000000000000"; }'

....

unpacking source archive /tmp/nix-build-source.drv-0/b7bc4384b7b471d1ddf892cb03f16189a66d5a0d.tar.gz

fixed-output derivation produced path '/nix/store/52dmfk71mamdh3hsklqdvawksg61an8y-source' with sha256 hash '05g37z3ilazzqa5rqj5zljndwxjbvpc18xibh6jlwjwpvg3kpbbh' instead of the expected hash '0000000000000000000000000000000000000000000000000000'

error: build of '/nix/store/gxvpc4fgs2nvpycifk5a0vab30qrvy8d-source.drv' failed

This is also the only fully general way, for example if you want your script to support the keepDotGit or fetchSubmodules.

[Sebastian Jordan]

Hi,

@Danylo: You have to run nix-prefetch-github with python 3.  Then it *should* work.

On 20.04.2018 12:55, Tuomas Tynkkynen wrote:

Hi,

On Friday, April 20, 2018 at 1:00:27 PM UTC+3, Danylo Hlynskyi wrote:

I've tried to implement "simple" Bash script based on comments above, but I think hash computation for fetchFromGitHub isn't straightforward

https://gist.github.com/danbst/1b7d6b97401ebdc34df0a03b8f3a1424

There are 2 ways of hash computing:
1) hash=$(nix-prefetch-url "https://github.com/$project/archive/$rev.tar.gz")
2) hash=$(nix-prefetch-git "github.com/$project.git" --rev "$rev" --quiet | jq -r .sha256)

The output is:

<snip 2 wrong outputs> 

But should be

{
    owner = "NixOS";
    repo = "hydra";
    rev = "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d";
    sha256 = "05g37z3ilazzqa5rqj5zljndwxjbvpc18xibh6jlwjwpvg3kpbbh";
}

(as per https://github.com/NixOS/nixpkgs/blob/192221ae3f2493f1e47a77726059dc4a65a9800d/pkgs/development/tools/misc/hydra/default.nix#L69-L74)

The 100% fool-proof way is to generate the same Nix expression you'd be generating but with a bogus hash, and then parse it from the error message:

This is exactly how nix-prefetch-github is implemented.

$ nix-build -E 'with import <nixpkgs> {}; fetchFromGitHub { owner = "NixOS"; repo = "hydra"; rev = "b7bc4384b7b471d1ddf892cb03f16189a66d5a0d"; sha256 = "0000000000000000000000000000000000000000000000000000"; }'

....

unpacking source archive /tmp/nix-build-source.drv-0/b7bc4384b7b471d1ddf892cb03f16189a66d5a0d.tar.gz

fixed-output derivation produced path '/nix/store/52dmfk71mamdh3hsklqdvawksg61an8y-source' with sha256 hash '05g37z3ilazzqa5rqj5zljndwxjbvpc18xibh6jlwjwpvg3kpbbh' instead of the expected hash '0000000000000000000000000000000000000000000000000000'

error: build of '/nix/store/gxvpc4fgs2nvpycifk5a0vab30qrvy8d-source.drv' failed

This is also the only fully general way, for example if you want your script to support the keepDotGit or fetchSubmodules.

--
You received this message because you are subscribed to the Google Groups "nix-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nix-devel+...@googlegroups.com.

To post to this group, send email to nix-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/c229229c-e0a9-4199-8849-9e6dec91a8a7%40googlegroups.com.