Pub: Hosted dependencies with Basic HTTP Auth

[Danny Kirchmeier]

At the core, I am asking for a pub repository where I can control read access to the packages I make. I need this for my team because we have a handful of private libraries which we cannot/will not open source.

We currently use git dependencies, and while these are adequate, I find the hosted dependency version resolution to be superior.

Current Options

After some playing around, I see that this presently works:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://username:pass...@myserver.com

The downside to this is the requirement of storing my username and password in my pubspec file, in source control. Plus, it means everyone is going to share the same username and password. If I am going to control access to packages, I really need to have each user be responsible for their own username and password.

Moving on, I considered other alternatives. While digging through the pub source, I found that I can use the “PUB_HOSTED_URL” environment variable to set the pub url. This allows each member of my team to set their credentials in the environment variable and have requests go through my server.

But this has the downside of including passwords in environment variables, and requiring all of my pub traffic to be routed through my pub repo. Either my repo needs to be intelligent enough to mirror the core pub repo, or it will need to act as a proxy and pass calls through when it doesn’t have a packaged.

So those are all of the options I currently see at my disposal. Neither is really great, leading to this discussion.

Ideal Solution

If I could have my way entirely, I would like to be able to use ssh as the transport mechanism for my private pub store. This might require maintaining static json files which return the data pub get needs, but I have no problem with that.

Proposed Solution

Assuming using SSH is too complicated, I think Basic HTTP auth is nice middle ground to compromise on. To make it work easily, I think we would need to add a ‘requireAuth’ key in the pubpec:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://myserver.com
      requireAuth: true

Then, in some other file, perhaps ~/.pub/authentication.yaml, you would find this:

- url: https://myserver.com
  username: johnsmith
  password: goteamgo

I figured the authentication lookup would match authentication based upon the url, ensuring credentials only go to the url specfied in the auth file.

Does anyone else have any thoughts or ideas on the matter?

—Danny

​

[Bob Nystrom]

On Thu, Jan 29, 2015 at 9:55 PM, Danny Kirchmeier <da...@kirchmeier.us> wrote:

At the core, I am asking for a pub repository where I can control read access to the packages I make. I need this for my team because we have a handful of private libraries which we cannot/will not open source.

Understandable. A number of users have asked to be able to run their own pub repository. From pub's perspective, you're more than welcome to do so. We just haven't implemented a reusable pub server that doesn't run on AppEngine. But, as long as you speak the same dead simple HTTP API that the pub client expects, it's pretty straightforward to write one yourself.

We (the pub team) never have because it's not something we can prioritize at Google but I've always hoped some fine upstanding community members would take it upon themselves to write a reusable one and share it with the rest of the world. Hint hint. ;)

We currently use git dependencies, and while these are adequate, I find the hosted dependency version resolution to be superior.

Totally.

Current Options

After some playing around, I see that this presently works:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://username:pass...@myserver.com

The downside to this is the requirement of storing my username and password in my pubspec file, in source control. Plus, it means everyone is going to share the same username and password. If I am going to control access to packages, I really need to have each user be responsible for their own username and password.

Worse, in my opinion, is that you have to add that URL to every dependency in all of your transitive dependencies' pubspecs. 

Moving on, I considered other alternatives. While digging through the pub source, I found that I can use the “PUB_HOSTED_URL” environment variable to set the pub url. This allows each member of my team to set their credentials in the environment variable and have requests go through my server.

I could have sworn we documented PUB_HOSTED_URL somewhere, but now I can't find it. Yes, this is the better way to point pub at a different server. Sorry you had to dig through code to find it.

But this has the downside of including passwords in environment variables, and requiring all of my pub traffic to be routed through my pub repo.

Depending on your perspective, either or both of these are features. Some users definitely do want the ability to control the visible set of packages.

Either my repo needs to be intelligent enough to mirror the core pub repo, or it will need to act as a proxy and pass calls through when it doesn’t have a packaged.

Yup! A clever server could proxy as needed.

So those are all of the options I currently see at my disposal. Neither is really great, leading to this discussion.

Ideal Solution

If I could have my way entirely, I would like to be able to use ssh as the transport mechanism for my private pub store. This might require maintaining static json files which return the data pub get needs, but I have no problem with that.

Proposed Solution

Assuming using SSH is too complicated, I think Basic HTTP auth is nice middle ground to compromise on. To make it work easily, I think we would need to add a ‘requireAuth’ key in the pubpec:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://myserver.com
      requireAuth: true

If you're hosting your own package server for your organization to use, wouldn't you want all package lookups to go through it? Or are you OK with hitting public pub for some of your dependencies? I ask because it seems like it might be a pain to have to add requireAuth: true to a bunch of pubspec instead of just making it a global ambient thing. 

Then, in some other file, perhaps ~/.pub/authentication.yaml, you would find this:

- url: https://myserver.com
  username: johnsmith
  password: goteamgo

I figured the authentication lookup would match authentication based upon the url, ensuring credentials only go to the url specfied in the auth file.

Does anyone else have any thoughts or ideas on the matter?

Is this significantly better than adding something to your .bashrc that sets a PUB_HOSTED_URL environment variable with username and password in it?

Cheers!

- bob

[Kevin Moore]

On Monday, February 2, 2015 at 1:53:01 PM UTC-8, Bob wrote:

On Thu, Jan 29, 2015 at 9:55 PM, Danny Kirchmeier <da...@kirchmeier.us> wrote:

At the core, I am asking for a pub repository where I can control read access to the packages I make. I need this for my team because we have a handful of private libraries which we cannot/will not open source.

Understandable. A number of users have asked to be able to run their own pub repository. From pub's perspective, you're more than welcome to do so. We just haven't implemented a reusable pub server that doesn't run on AppEngine. But, as long as you speak the same dead simple HTTP API that the pub client expects, it's pretty straightforward to write one yourself.

There is work to rebuild our pub site on Dart and the implementation contains a separate package with support for the pub HTTP protocol. You'll have to wire in your own storage, but you'll be using the same server code that the pub site uses.

There isn't a timeline on this work, but know it's coming.

 

We (the pub team) never have because it's not something we can prioritize at Google but I've always hoped some fine upstanding community members would take it upon themselves to write a reusable one and share it with the rest of the world. Hint hint. ;)

We currently use git dependencies, and while these are adequate, I find the hosted dependency version resolution to be superior.

Totally.

Current Options

After some playing around, I see that this presently works:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://username:password@myserver.com

The downside to this is the requirement of storing my username and password in my pubspec file, in source control. Plus, it means everyone is going to share the same username and password. If I am going to control access to packages, I really need to have each user be responsible for their own username and password.

Worse, in my opinion, is that you have to add that URL to every dependency in all of your transitive dependencies' pubspecs. 

Moving on, I considered other alternatives. While digging through the pub source, I found that I can use the “PUB_HOSTED_URL” environment variable to set the pub url. This allows each member of my team to set their credentials in the environment variable and have requests go through my server.

I could have sworn we documented PUB_HOSTED_URL somewhere, but now I can't find it. Yes, this is the better way to point pub at a different server. Sorry you had to dig through code to find it.

It is documented - https://www.dartlang.org/tools/pub/installing.html#optional-environment-variables

Perhaps it could be documented better. 

[Rob Bishop]

There is work to rebuild our pub site on Dart and the implementation contains a separate package with support for the pub HTTP protocol. You'll have to wire in your own storage, but you'll be using the same server code that the pub site uses.

There isn't a timeline on this work, but know it's coming.

This is fantastic!  We need this capability in the worst way.  It'll simplify our build processes and should lead to an explosion of use at our organization due to increased discoverability and lowered barriers.

Is there any way that we can help speed this along?

[Bob Nystrom]

On Wed, Feb 4, 2015 at 7:00 AM, Rob Bishop <robbi...@gmail.com> wrote:

Is there any way that we can help speed this along?

Adding Martin Kustermann who is working on this. :)

- bob

[Martin Kustermann]

Hi Rob,

unfortunately I cannot give you the best answer but here are a few notes from me:

a) Using pub with private package repository

As you have found out, there are a number of built-in assumptions, which are reflected in the (not yet officially documented) API used for communication between the pub client and the pub server. Namely the pub client:

- requires you to basically have a gmail account and requires you to grant it oauth2 access (to get your email address)

- the uploading protocol is a 3 step process which was required to upload to google cloud storage directly from the client

- it communicates with pub.dartlang.org (possibly adding the Authorization header which can be used to retrieve the uploader's email address)

- it uses HTTP as protocol

This has the implication that neither authorization nor transport protocol is plugable.

=> This is a limitation is in the pub client and there might be several ways of making these two things plugable. AFAIK we haven't seen demand for this so far, but it wouldn't hurt filing a feature request for the pub client (feel free to create a bug at http://dartbug.com/new with Area=Pub).

b) The rewrite of pub.dartlang.org in dart & a plugable server

We are in the process of rewriting the pub.dartlang.org application in dart (for a number of reasons). When doing so we tried to abstract the authentication (i.e. google oauth2) and transport protocol (i.e. http) away from an implementation of a package repository (as kevin already mentioned).

This will allow you to implement a pub server e.g. based on a directory on the filesystem, without dealing with the specifics of the HTTP API (e.g. like the 3-step upload process).

=> I'll split this into two git repositories during the next days and will ping this thread so you can use this :)

c) What YOU can do with current system.

Without modifications to the current pub client, here is what you can do:

Every developer has a small local pub proxy running (which adds developer credentials and talks to another server) & has the PUB_HOSTED_URL environment variable point to it.

c1) Forward: Just let the local pub proxy forward to the real server with modified credentials (i.e. with credentials of the developer who started the local pub proxy) but the same API.

c2) Custom Protocol: Translate to a different API (e.g. use SSH or something else to upload packages, download packages and metadata queries).

=> For read-only this works fine, but for uploads this will still require developers to have a gmail address.

To prevent that, you can use another tool (i.e. not the pub client) for publishing packages [e.g. plain SSH, in which case you obviously don't need oauth2/gmail account] but still use the pub client + local pub proxy for fetching packages.

As mentioned, I'll create two github repositories, one for pub.dartlang.org, and one for the common code during the next days and will let you know when it's there :)

> and should lead to an explosion of use at our organization due to increased discoverability and lowered barriers.

That would be fantastic :)

> Is there any way that we can help speed this along?

Once the github repos are in place, you're more than welcome to hack on it :)

Best Regards,

Martin

[Rob Bishop]

Looking forward to this capability.

Things on my wish list:

• Deployment flexibility -- easy to run on a corporate network (on a VM) or as a cloud app
• Windows service for automatic restart on VM reboot
• File system storage just works (but other storage options welcome)
• Shelf-based with configurable pub-related and auth2 layers
• Completely configurable front-end, but one that looks and works like the pub.dartlang.org as default
• Completely configurable security
• For example in our organization we would probably want smart card based tiered-access (view, get, publish, admin), plus perhaps also password restrictions for some tiers
• SSL support
• Automatic generation of API docs on publish with link to view
• Publish and usage stats
• HTML-based publish support so developers can select, vet and publish packages right from the browser
• Searchable (including API docs)
• Audit log
• Fast and scalable

I realize that there are things on that list that are likely out of scope for what you're trying to do.  I'm sure the community (me included) can work to fill in the gaps, but having a core set of pieces to work around would be a big boost.  I'm sure the community also has a wider variety of wish list items than I've listed here -- maybe post them below so we can get an idea of what a configurable private pub solution that satisfies most needs would look like.

[Bob Nystrom]

On Thu, Feb 5, 2015 at 6:50 AM, Martin Kustermann <kuste...@google.com> wrote:

a) Using pub with private package repository

As you have found out, there are a number of built-in assumptions, which are reflected in the (not yet officially documented) API used for communication between the pub client and the pub server. Namely the pub client:

- requires you to basically have a gmail account and requires you to grant it oauth2 access (to get your email address)

- the uploading protocol is a 3 step process which was required to upload to google cloud storage directly from the client

- it communicates with pub.dartlang.org (possibly adding the Authorization header which can be used to retrieve the uploader's email address)

- it uses HTTP as protocol

This has the implication that neither authorization nor transport protocol is plugable.

=> This is a limitation is in the pub client and there might be several ways of making these two things plugable. AFAIK we haven't seen demand for this so far, but it wouldn't hurt filing a feature request for the pub client (feel free to create a bug at http://dartbug.com/new with Area=Pub).

Note that this is only a problem if you want to use the pub client to publish packages to your hand-rolled package server. If you just want to pull packages from it—which is the most common use case by far—the pub HTTP API requires no Google account, oauth2 or any authentication at all.

For corporate users that want to run their own package server, I expect most would just define their own process to pushing new packages to the server without using the pub client, which may be as simple as "dump them into this directory on the Intranet".

The pub client doesn't add a lot of value as a publishing mechanism. Ultimately, all it does is stick a .tar.gz somewhere. What you really want to reuse is the version resolution and installation behavior. For that, your custom server doesn't need to support any sort of authentication.

Cheers!

- bob

[Rob Bishop]

\o/

https://github.com/dart-lang/pub_server

On Friday, January 30, 2015 at 12:55:32 AM UTC-5, Danny Kirchmeier wrote:

At the core, I am asking for a pub repository where I can control read access to the packages I make. I need this for my team because we have a handful of private libraries which we cannot/will not open source.

We currently use git dependencies, and while these are adequate, I find the hosted dependency version resolution to be superior.

Current Options

After some playing around, I see that this presently works:

dependencies:
  my_lib:
    hosted:
      name: my_lib
      url: https://username:password@myserver.com

[Martin Kustermann]

@Rob:

You found it a bit earlier than I anticipated. I wanted to land this change first, before I respond on this thread. You can try cloning the pub_server repository and applying this patch. Then you should be able to just run a local pub server with "~/pub_server $ pub run serve" :)

We've opened the repository containing an (very much in-progress) re-write of pub.dartlang.org in dart as well dart-lang/pub-dartlang-dart. At the beginning it will look and behave exactly as the old pub.dartlang.org app -- which will make it very easy to transparently switch gradually to the dart version. Once the switch is complete we can iterate on the Dart version (e.g. adding audit logs, different search service, package stats, etc.).

=> But remember that this is experimental code ATM.

About the long list of features you would like to have (and I'm sure I could add one or two things myself as well :) ):

I do agree that many of the things you mentioned would be super nice to have, but I'm also of the opinion that the more specific you get, the less we'll be able to find a one-size-fits-all solution. 

A few examples:

- Can we assume that the server is only a single instance or can it be scaled to several instances?

- Is the search service an external service (e.g. custom search API as we use currently for pub.dartlang.org) or a custom one?

- Is the dartdocs service an external service (e.g. dartdocs.org is a completely different site, managed by the community ATM) or a custom one?

- Do we cache things for performance reasons? Using memcached / ...?

- Do you use a batch-job for generating package stats? What schedules the batch jobs (e.g. cron or something else)?

- Do you keep download counts or deduce them from log files? What format do the log files have?

- Does the UI need the README or CHANGELOG files to render the web page? Does it need other things, like the list of libraries in the package? ...

As you can see, it highly depends on the requirements. Though I'm sure we can come up with a number of re-usable sub-parts (as e.g. the HTTP pub API).

If you want to help accelerate this, or want to guide it in a certain direction just let us know or send us pull requests :)

@Bob:

I agree - as I've already mentioned a custom way of uploading is certainly one possibility.

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Martin Kustermann]

Here's actually the correct link to the change and the diff.

[Rob Bishop]

I've been working on standing up an internal pub server at our firm and I have one question and some observations:

Q:  How am I supposed to add an uploader to a package?  The pub uploader command ignores the --server option and if left off the package is not found (cause it's looking in pub.dartlang and not my local pub server).

Observations:  The pub_server package handles the basic package protocol functionality very well (THANK YOU), but in terms of replicating the pub web site there are many holes.  Holes which I've had to fill by extending the ShelfPubServer and creating my own mini protocol for things like getting a list of the latest versions of each package or getting the upload date for a package... not to mention building out a client app to display everything.  Also, the dartdoc situation is frustrating.  It looks like I'm going to have to customize dartdoc-viewer and further extend my custom server on the back end to automatically generate and serve the dartdoc JSON, which is a bit daunting,  Just some feedback to let you know where the pain points are -- at least for my org.

   

[Rob Bishop]

I see a "dartdoc" placeholder was just added to pub.  Any chance we can get a 0.1.0 release soon so that I can start to integrate it?   Looks like just what I need.

[Kevin Moore]

You can grab the source here: https://github.com/dart-lang/dartdoc

FYI: it's still considered alpha.

On Fri, Apr 3, 2015 at 9:00 AM, Rob Bishop <robbi...@gmail.com> wrote:

I see a "dartdoc" placeholder was just added to pub.  Any chance we can get a 0.1.0 release soon so that I can start to integrate it?   Looks like just what I need.

--

[Rob Bishop]

Right, I know. Apparently the IDE (eclipse) or pub doesn't like to handle packages with version 0,0.0 though because it doesn't show up.  I don't know if that's a bug or a feature. 

I can work around the issue by using a git dependency instead of a pub.dartlang dependency.

  dartdoc: #any

    git: https://github.com/dart-lang/dartdoc.git