Using private github repositories as package sources?

[Alexis King]

I have been looking for a way to use Racket at work, and we’ve found
a couple places where it might be useful for documentation or
tooling. As part of this, it would be very nice to keep our source
code private, but it would still be helpful to make use of the
package manager to handle dependency resolution. We explored creating
a custom catalog that would contain our packages, which has worked
reasonably well, but the Racket package system does not appear to
be capable of fetching packages backed by private repositories.

Glancing over the git protocol documentation and interacting with
a private repository via HTTP client, implementing this on the
technical side doesn’t look too difficult. GitHub uses the “smart”
HTTP protocol as documented here[1], and authorization is done using
HTTP Basic Authentication. Implementing this without using libgit
or the git CLI might be a little difficult, but distributing libgit
would not be hard if it ended up being a problem.

The trickier issue is the social side, as well as the user interface.
How would git credentials be provided to the package manager so
that it could actually access these packages? More importantly, is
it actually okay for the success of package installation to be
dependent on some configuration that lives on a user’s local machine?
If these sorts of packages were uploaded to the main package catalog,
what would be the policy for handling them?

Ultimately, I think it’s important for the package manager to support
private package distribution mechanisms for me to be able to
comfortably adopt Racket in a corporate setting, so I think having
some solution to this problem that does not involve out-of-band
trickery would be nice. I’m just not sure what that solution might
look like or how it would mesh with the current, open-source centric
package management strategies. I’m curious about how other people
feel about these issues.

Thanks,
Alexis

[1]: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

[Matthew Flatt]

Accessing private GitHub repos via HTTPS URLs is a matter of adding a

Authorization: Basic <user:password, but base64-encoded>

header to the request sent by `net/git-checkout`, where "password" is a
GitHub personal access token. (I temporarily added a hardwired line in
`http-request-headers` in `net/git-checkout` to check that it works
with my GitHub account.)

I'm not sure where to add a username+password configuration for `raco
pkg`. Maybe `net/git-checkout` should look in a particular file. Maybe
`git-checkout` should take a username and password as arguments, and
`raco pkg` should get them from somewhere. If this seems like the right
direction to you, I'd be happy to see a pull request that implements
whatever configuration approach you think is appropriate.

You'd need to have some local configuration to access a private repo
through HTTPS, but that doesn't seem like a problem to me (i.e.,
similar to local DNS or package-catalog configuration). Nothing would
prevent you from registering an HTTPS package source with the main
package catalog, but it would show up as "install fails" at best; using
a private catalog seems clearly better for private packages.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/367855D6-5D7E-4772-8240-1699B2AF13
> 42%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Alexis King]

Alright, I’ve opened a pull request that adds support for authentication
to net/git-checkout, which appears to be able to successfully clone
private GitHub repositories. Presumably, adjusting the package
system to use this information is not terribly technical difficult,
but I’m finding myself fairly unsatisfied with the different places
these credentials could be stored.

They could be stored within `raco pkg config`, which seems like a
reasonable place for them. Something like `raco pkg config set
git-username ...` and a `git-password` counterpart would be reasonable
enough, but this completely breaks down if someone ever has the
need to pull packages from multiple private sources that need
different credentials. It’d also be possible to somehow store a
list of username/password pairs, then try each one in succession,
but this is both error-prone and complicated to design a useful
user interface for.

I think that the problem of package installation of being dependent
on potentially-transient local machine configuration is probably
unavoidable, so I am going to try and ignore that problem for now.
However, there’s still the annoying issue of having private packages
being potentially uploaded to the main package catalog, which is
probably not what we want.

Would it make sense to associate an optional username/password pair
with each catalog, rather than globally? This would allow having
multiple private package sources, and I think it would accommodate
most practical use-cases without getting to complicated to manage
by users. It would also contain the effects of authentication to
one place to avoid ending up with too much confusing global
configuration.

Thoughts?