Running a private pub instance?

[Danny Tuppeny]

If we decide to start using Dart for our projects, we're going to end up with a lot of internal packages that we need to be able to reference from other projects without publishing them publicly on pub.dartlang.org. We already do this with .NET for our NuGet packages using the official NuGet Server package.

I found that the pub.dartlang.org code is on GitHub; but I have two questions:

1. Is this easily runnable without AppEngine? We would want to host it within our office on a Windows box, and the AppEngine SDK seems like a weird way to do it (since it runs interactively in a logged on session).

2. How do we define in pubspec.yaml which packages come from which source? (StackOverflow)

[Günter Zöchbauer]

The readme in the pub-dartlang repo says that it can't be run locally because of the app-engine dependency but that the code can be used as reference when creating your custom implementation.

The protocol is not complicated (JSON-based)

[Alexandre Maurel]

checkout 

https://pub.dartlang.org/packages/pub_proxy_server

i will perhaphs meet your needs

[Danny Tuppeny]

On 31 October 2014 15:49, Günter Zöchbauer <gzo...@gmail.com> wrote:

The readme in the pub-dartlang repo says that it can't be run locally because of the app-engine dependency but that the code can be used as reference when creating your custom implementation.

The protocol is not complicated (JSON-based)

If it's not complicated; seems kinda silly Google don't provide one. Doesn't make sense for every company wanted a private feed to have to implement it themselves; it's just another stumbling block for companies :(

[Danny Tuppeny]

On 31 October 2014 17:03, Alexandre Maurel <alexandr...@gmail.com> wrote:

checkout 

https://pub.dartlang.org/packages/pub_proxy_server

i will perhaphs meet your needs

Thanks; I'll take a look next week! 

[Natalie Weizenbaum]

On Fri, Oct 31, 2014 at 8:35 AM, Danny Tuppeny <da...@tuppeny.com> wrote:

If we decide to start using Dart for our projects, we're going to end up with a lot of internal packages that we need to be able to reference from other projects without publishing them publicly on pub.dartlang.org. We already do this with .NET for our NuGet packages using the official NuGet Server package.

I found that the pub.dartlang.org code is on GitHub; but I have two questions:

1. Is this easily runnable without AppEngine? We would want to host it within our office on a Windows box, and the AppEngine SDK seems like a weird way to do it (since it runs interactively in a logged on session).

As other people have pointed out, the dependency on AppEngine is pretty tight. That said, other than package publishing (which you could handle differently) the protocol is pretty straightforward; it shouldn't be too hard to write your own server that implements them.

 

2. How do we define in pubspec.yaml which packages come from which source? (StackOverflow)

The full syntax of dependency specifications is explained in the documentation, but the short answer is:

dependencies:
  transmogrify:
    hosted:
      name: transmogrify
      url: http://your-package-server.com
    version: '>=0.4.0 <1.0.0'

If it's not complicated; seems kinda silly Google don't provide one. Doesn't make sense for every company wanted a private feed to have to implement it themselves; it's just another stumbling block for companies :(

It's really a question of time. Our main mission is to do things that benefit all pub users, and we already have our own server that works. Even if we were to write  a new one in, say, Dart, we'd want to write one that we'd use—which would probably mean also being hosted on AppEngine.

- Natalie 

[Danny Tuppeny]

On 31 October 2014 18:24, 'Natalie Weizenbaum' via Dart Misc <mi...@dartlang.org> wrote:

If it's not complicated; seems kinda silly Google don't provide one. Doesn't make sense for every company wanted a private feed to have to implement it themselves; it's just another stumbling block for companies :(

It's really a question of time. Our main mission is to do things that benefit all pub users, and we already have our own server that works. Even if we were to write  a new one in, say, Dart, we'd want to write one that we'd use—which would probably mean also being hosted on AppEngine.

Well; I must say that sounds kinda silly. If you were going to go to the effort of rewriting it in, say, Dart; you should really think about how beneficial it would be for companies if it was also hostable locally too.

The less friction in adopting Dart; the more people will do it. All of these little things we have to do ourselves add up; and eat away at the advertised benefits of using Dart :O(

I'm working really hard to make Dart seem like a viable option for some huge projects where I work; but when we hit little bumps like this (Serialisation was another) the response from the Dart Team seems to be "meh". This doesn't help convince teams of experienced technical people and managers that Google have our back if we go all-in on a language we can't convert back to JavaScript from.

[Bob Nystrom]

On Fri, Oct 31, 2014 at 11:41 AM, Danny Tuppeny <da...@tuppeny.com> wrote:

Well; I must say that sounds kinda silly. If you were going to go to the effort of rewriting it in, say, Dart; you should really think about how beneficial it would be for companies if it was also hostable locally too.

A pub server implementation has to store packages somewhere, right? We could, I suppose, use raw dart:io and write directly to the file system, but once you start thinking about querying, searching, performance, logging, etc., you really want something better than just bare file IO.

That means you have to pick some kind of back-end datastore. And that means, yes, you are dependent on that datastore platform. We could pick MongoDB, or MySQL, or whatever, and users would just as rightfully object to being locked into one of those. The current implementation uses AppEngine because AppEngine was a pretty natural answer for a managed, cloud hosted application written by a bunch of Googlers.

It would be possible to write some kind of back-end agnostic server with a pluggable back-end layer, but that's a complex undertaking. The server itself is extremely simple. It's a tiny vanilla CRUD app. It's probably easier to just rewrite it (maybe reusing some of the front-end code) for each back-end than to try to make a generic one.

- bob

[Anders Holmgren]

I haven't looked at the code but assuming that the persistence is abstracted behind a DAO then the community could contribute different implementations (e.g. mongo). Probably more likely to happen if it was in Dart I guess.

Then assuming the AE dependencies were well quarantined, spinning up a standalone server against mongo or other should be fairly easy. Docker might help wrap it up nicely too.

A

[Hans - Jürgen Alps]

You should really think about, if it is worth while implementing a private pub instance.  If you are in a big company it does not make sence to force all teams to use your central components,  because they have different requirements and it would slow down their productivity if they would have to use components which do not fit their needs 100 percent.
If you are in a small company you could choose different approaches like organizing shared libraries in a common structure.

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Danny Tuppeny]

On 31 October 2014 19:47, 'Bob Nystrom' via Dart Misc <mi...@dartlang.org> wrote:

A pub server implementation has to store packages somewhere, right? We could, I suppose, use raw dart:io and write directly to the file system, but once you start thinking about querying, searching, performance, logging, etc., you really want something better than just bare file IO.

That means you have to pick some kind of back-end datastore. And that means, yes, you are dependent on that datastore platform. We could pick MongoDB, or MySQL, or whatever, and users would just as rightfully object to being locked into one of those. The current implementation uses AppEngine because AppEngine was a pretty natural answer for a managed, cloud hosted application written by a bunch of Googlers.

It would be possible to write some kind of back-end agnostic server with a pluggable back-end layer, but that's a complex undertaking. The server itself is extremely simple. It's a tiny vanilla CRUD app. It's probably easier to just rewrite it (maybe reusing some of the front-end code) for each back-end than to try to make a generic one.

It wasn't me that said it was a simple task ;)

But that's not the point; it's one of those things that so many people might want that it seems to make sense to be done "officially" than hundreds of developers do the same work themselves (or have to trust/review/understand an open source one). The NuGet Team did this; because they understood that having private feeds would be fundamental need for any reasonable sized company.

If Google are too busy to find time to do it; then probably so are all of the devs out there considering adopting Dart. IMO, Google should be tackling all friction points for adoption of Dart before anything else; and that includes thing like this and serialisation. Otherwise, the list of "mundane things we need to do if we choose to adopt Dart that we don't have to do if we don't" is going to get bigger.

Don't get me wrong; I'm a big Dart fan; but I need to convince a bunch of people we should use it. All of my complaints are just pre-empting what's going to come up as we start to nail down our decisions; there are very tech-savy people and we're deliberately ripping every tech being evaluated to shreds to understand all possible pain points and drawbacks. We probably won't ever have such a clean slate to start with tech-wise; so we have to be absolutely sure our decisions are going to make things easier and faster.

[Danny Tuppeny]

On 31 October 2014 21:35, Hans - Jürgen Alps <hansjuer...@googlemail.com> wrote:

You should really think about, if it is worth while implementing a private pub instance.  If you are in a big company it does not make sence to force all teams to use your central components,  because they have different requirements and it would slow down their productivity if they would have to use components which do not fit their needs 100 percent.
If you are in a small company you could choose different approaches like organizing shared libraries in a common structure.

I don't really understand what you mean. Any reasonable sized company will want to share code between projects; and pub is by far the best way to do it. We went for years without a private NuGet, and used to update references to other teams components (and even some of our own) by copy/pasting DLLs into a folder committed in our repos. Since setting up the internal feed (which was absolutely trivial; I don't know why we put it off so long), we've extracted more and more reusable libraries and decreased reimplementation across both teams and projects within teams.

It doesn't need much thinking about; for the work we do sharing code easily across teams and projects is an absolute requirement. Pub is without a doubt, the best way to do this. As much as I'd love to start OSSing some of our stuff; it just wouldn't work with our business, so using pub.dartlang.org install is out of the question.

[Adam Stark]

You can host the libraries in a self-hosted git repository too if you need to. Plus you get versioning and access control (almost) for free. I already use a git repo for libraries that I don't want to put on pub. The only thing missing is robust versioning queries.

--

[Danny Tuppeny]

On 1 November 2014 19:44, Adam Stark <llama...@gmail.com> wrote:

You can host the libraries in a self-hosted git repository too if you need to. Plus you get versioning and access control (almost) for free. I already use a git repo for libraries that I don't want to put on pub. The only thing missing is robust versioning queries.

This is a clumsy workaround; and publishing to Git from a buildserver is not as straight forward as pushing to a Pub server.

I'm not saying it's not possible; but it's just another annoyance. If Google want companies to adopt Dart, there shouldn't be all these little holes that we need to plug ourselves. What happened to Dart being "batteries included"? More time should be spent making Dart easier to adopt/sell :(

[Bob Nystrom]

On Sat, Nov 1, 2014 at 1:06 PM, Danny Tuppeny <da...@tuppeny.com> wrote:

What happened to Dart being "batteries included"? More time should be spent making Dart easier to adopt/sell :(

We are trying to include the batteries, but there are a lot of potential batteries to include. We're a pretty large team, but even so we have to decide what makes the most sense for us to implement and what is better handled by the ecosystem.

Most of the time, we tend to prioritize things that require deep understanding of the Dart platform over stuff that's less tightly coupled. Since we have a lot of Dart expertise, we can usually move faster there than most people. A reusable private pub server would be nice, but it doesn't really play to our strengths. It's basically just a CRUD app, and doesn't require much Dart expertise. What it does require is server-side web application experience—stuff like persistence, replication, authentication, and query optimization—and the Dart ecosystem probably knows that better than many of us on the Dart team do.

Cheers,

- bob

[Danny Tuppeny]

I understand (and agree with) the priorities; but you should underestimate the expectation people have of a platform. For people that work on large apps and are constantly bitten by "open source" code that gets abandoned, breaking changes, commercialised, etc.; any significant hole in the platform is something they'll need to plug themselves.

Dart appears to be designed for large apps (that's where I think you'll get the biggest benefits), yet lack of things like serialisation and a private pub server make it feel like you're aiming for OSS/hobbyists that would love to plug these holes for fun.

I apologise if this sounds like a rant; I'm just trying to give feedback on things that I think are stumbling blocks to adopting; and I think adopting is the one thing that Dart is really behind on right now and needs prioritising. Look at this nonsense... People talking about replacing JavaScript because it's not fit for purpose. I get the impression they've never even heard of Dart.

[Anders Holmgren]

Worth remembering that in Java there is nothing like pub that is provided by the Java team. The closest thing would be something like maven, which is entirely ecosystem including maven central (analogous to pub.dartlang.org).

[Bob Nystrom]

On Mon, Nov 3, 2014 at 11:49 AM, Anders Holmgren <andersm...@gmail.com> wrote:

Worth remembering that in Java there is nothing like pub that is provided by the Java team.

See also: Go, Ruby, Python, PHP, C# (until Win 10, I think?), C, C++, Lua, etc.

- bob

[Danny Tuppeny]

On 3 November 2014 20:21, 'Bob Nystrom' via Dart Misc <mi...@dartlang.org> wrote:

Worth remembering that in Java there is nothing like pub that is provided by the Java team.

See also: Go, Ruby, Python, PHP, C# (until Win 10, I think?), C, C++, Lua, etc.

Most of them are "ancient"; and C# has has NuGet for years. But in any case, expectations change/increase. If you want someone to invest heavily in your platform; the advantages need to outweigh the disadvantages, and having to do a bunch of up-front infrastructure work eats aware at the possible savings.

I'm only trying to help adoption by highlighting things that appear to be missing to me and my colleagues (and possibly others businesses considering adopting Dart). It's your language; the decisions are yours. I won't get upset if our ideas are not priorities; though I'll obviously be a little sad if we end up not picking Dart as a result of the perceived benefits not outweighing the risks :O(