Using dart with bazel

[Hochhaus, Andy]

Hello,

My organization is a long time bazel user (for Go, Closure style JS, C++, Python). We are now considering using dart for some servers and a web application. We have significant internal experience with bazel (authoring BUILD files) and no experience working pub (authoring pubspec.yaml) files.

The dazel repo says that it is "highly experimental" while the underlying rules are stable. However, the underlying rules_dart repo says that they are "not production ready" and should "only be used with automatically generated BUILD files".

Given these seemingly conflicting statements, can anyone clarify the stability (or lack thereof) for these repositories? Given our preference for the consistency of using BUILD files across our entire monorepo, do any reasons exist to prefer pubspec.yaml?

Best,
Andy

[Ryan Gonzalez]

Dazel basically takes in a pubspec and outputs Bazel rules. The underlying rules don't need to be stable, because they're autogenerated.

I'd highly advise getting used to pub anyway; although its future as a build tool is somewhat hazy (IIRC the build package is a replacement for pub transformers), it's still the current dependency management tool.

--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/

[Hochhaus, Andy]

Thanks Ryan.

On Sun, Nov 26, 2017 at 10:29 AM, Ryan Gonzalez <rym...@gmail.com> wrote:
> Dazel basically takes in a pubspec and outputs Bazel rules.

Makes sense. This matches my understanding.

> The underlying rules don't need to be stable, because they're autogenerated.

Assuming one is using dazel, this also makes sense. However, if one
plans to hand write BUILD files the inconsistency is potentially still
worth clarifying.

> I'd highly advise getting used to pub anyway

I respectfully disagree here. One of the major wins of a tool like
bazel is that it supports "multiple languages with one tool". Having a
company-wide monorepo which uses a single tool for all compiles,
tests, deployments, etc regardless of language allows developers to
move about the code base while minimizing cognitive load.

Based on the wording in the README, I would guess that Google
internally prefers the use of bazel over pub.

> although its future as a build tool is somewhat hazy (IIRC the build package is a replacement for pub transformers), it's still the current dependency management tool.

Thanks! Do you happen to have a reference regarding the build package
replacing pub (so that I can learn more about the road map for the
tooling)?

Best,
Andy

[Matan Lurey]

Hi Ryan and Andy!

disclaimer: I work on the Dart team and AngularDart, which is a consumer of both pub, bazel, and other build systems we use. I can try and answer any other questions you have or redirect them to folks that would know more.

tl;dr, at one point we had no build system. Dart 1.0 was predicated around having a JIT (just-in-time) source-code VM, so similar to the JS ecosystem in ~2011/2012, there wasn't an apparent need to have a platform-level build system (remember, Dart->JS was just going to be for legacy browsers, and you did most browser development in Dartium).

Once pub, as a package manager, was built and shipped, there were various opinions on whether it or another tool should play a build/asset system role, and the current implementation of pub does via "pub serve", "pub build", and asset transformers (commonly called "transformers").

This worked well enough for small/medium-ish size projects, but broke down quite a bit once your project(s) got larger - designed to be an in-memory monolithic transformation system, not a scaleable, modular build system.

Internally, because of how Google is setup, we used/were required to use Blaze (now called "Bazel"), and new developments here helped a lot - for example we could quickly re-compile code to JavaScript using DDC.

Externally, we considered offering full Bazel support, but shied away:

* Bazel at the time barely supported Windows, which is important to us.

* Even today, lots of our Bazel integration is Unix-specific. It would take a lot of time, resources, and debugging to get this to work great out-of-the-box to Windows users, and even then Bazel is pretty heavyweight for some projects.

* Dart users liked the ability to author new transformers/builders, and we already had some success with source_gen/build_runner, which generated files on disk instead of through pub. The infamous built_value used this system.

So, we took build_runner, and tried building on it. And it turned out to work quite well. Basically, there is one user-level set of interfaces, package:build, with a set of contracts that if you follow work for pub transformers, bazel/blaze, and build_runner. 

So, where are we in EO 2017?

1. We're about to soft deprecate "pub transformers" entirely.

2. We've suspended work on shipping Bazel support externally.

3. And we're working to enhance build_runner into a longer-term solution.

If you follow our milestones for build_runner, you'll see we are making quite a bit of progress, but with still more to do. Please let us know if you have any other questions.

I've answered your questions inline below, as well.

Cheers!

~ Matan

On Sun, Nov 26, 2017 at 11:04 AM Hochhaus, Andy <ahoc...@samegoal.com> wrote:

Thanks Ryan.

On Sun, Nov 26, 2017 at 10:29 AM, Ryan Gonzalez <rym...@gmail.com> wrote:
> Dazel basically takes in a pubspec and outputs Bazel rules.

Makes sense. This matches my understanding.

Dazel was based on the idea that we could infer a build system configuration based on your package configuration and a bit of extra metadata. We've actually kept this model as "build_config" for build_runner. The idea here is that users that consume generators would have to do little or no work.

https://github.com/dart-lang/build/tree/master/build_config 

 

> The underlying rules don't need to be stable, because they're autogenerated.

Assuming one is using dazel, this also makes sense. However, if one
plans to hand write BUILD files the inconsistency is potentially still
worth clarifying.

Unfortunately we're not in a place where it's feasible to do this with guaranteed success - though we welcome folks contributing if this is something that is important to you. We might revisit this story in 2018, but not before we've shipped something that lets people move off pub serve/build.

 

> I'd highly advise getting used to pub anyway

I respectfully disagree here. One of the major wins of a tool like
bazel is that it supports "multiple languages with one tool". Having a
company-wide monorepo which uses a single tool for all compiles,
tests, deployments, etc regardless of language allows developers to
move about the code base while minimizing cognitive load.

Definitely agree with ... both of you :)

Pub as a package manager is likely to stay indefinitely (similar to NPM for Node). As far as a build system goes, we'll be focusing on build_runner. We're also still working out what the long-term entry-points for compilation, testing, deployment will be.

For example some people prefer a single monolothic tool, others would prefer specialized packages. If you have opinions please speak up :)

 

Based on the wording in the README, I would guess that Google
internally prefers the use of bazel over pub.

Because we use a single mono-repo internally without branches, there is no reason to have dependency management, so a lot (read: all) of the benefits of pub sort of go away anyway.

 

> although its future as a build tool is somewhat hazy (IIRC the build package is a replacement for pub transformers), it's still the current dependency management tool.

Thanks! Do you happen to have a reference regarding the build package
replacing pub (so that I can learn more about the road map for the
tooling)?

See above :)

 

Best,
Andy

--
For other discussions, see https://groups.google.com/a/dartlang.org/

For HOWTO questions, visit http://stackoverflow.com/tags/dart

To file a bug report or feature request, go to http://www.dartbug.com/new
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Hochhaus, Andy]

Thanks for the detailed reply and added context Matan.

I agree that bazel shouldn't be the only way to build dart code. Potentially, the build tooling could consider two use cases:

1. Smaller repositories (possibly only containing dart code) which prefer minimal setup & configuration. These users will likely prefer a dart specific tool set (such as pub or build_runner).
2. Larger existing projects containing code written in a wide variety of languages. These users will likely be willing to undertake additional configuration steps.
   

On Sun, Nov 26, 2017 at 11:31 AM, Matan Lurey <ma...@lurey.org> wrote:
> Externally, we considered offering full Bazel support, but shied away:
>
> * Bazel at the time barely supported Windows, which is important to us.
> * Even today, lots of our Bazel integration is Unix-specific. It would take
> a lot of time, resources, and debugging to get this to work great
> out-of-the-box to Windows users, and even then Bazel is pretty heavyweight
> for some projects.

In the "two build systems" model suggested above, windows support need not be a requirement here. It would be acceptable to have windows only supported by the dart tooling (pub, build_runner) while macOS and linux were also supported by bazel.

Further, as bazel windows support has significantly improved, should adequate interest appear in using dart with bazel on windows it would be possible to fix those issues.

> So, we took build_runner, and tried building on it. And it turned out to
> work quite well. Basically, there is one user-level set of interfaces,
> package:build, with a set of contracts that if you follow work for pub
> transformers, bazel/blaze, and build_runner.

Can you say more here? Specifically, does any documentation exist on using these with bazel?

>> On Sun, Nov 26, 2017 at 10:29 AM, Ryan Gonzalez <rym...@gmail.com> wrote:
>> > Dazel basically takes in a pubspec and outputs Bazel rules.
>>
>> Makes sense. This matches my understanding.
>
> Dazel was based on the idea that we could infer a build system configuration
> based on your package configuration and a bit of extra metadata. We've
> actually kept this model as "build_config" for build_runner. The idea here
> is that users that consume generators would have to do little or no work.
>
> https://github.com/dart-lang/build/tree/master/build_config

Interesting. Are you sating that build_config can be used to generate bazel BUILD files?

The rules_go repo has a tool called gazelle which functions in a similar manner to dazel (walks a directory tree, parses *.go files, generates BUILD.bazel files). My dislike for dazel (and maybe build_config?) is that it depends on pubspec.yaml files which are "pub specific" files (vs simply dart source code). In the gazelle case only go source files are required.

Is it possible to have dazel (or build_config?) avoid a strict dependency on pubspec.yaml (by falling back to parsing *.dart files and assuming the "any" version for all dependencies when pubspec isn't present)? In the gazelle case, if the user wished to pin an external library to a specific version they simply modify the WORKSPACE file.

>> > The underlying rules don't need to be stable, because they're
>> > autogenerated.
>>
>> Assuming one is using dazel, this also makes sense. However, if one
>> plans to hand write BUILD files the inconsistency is potentially still
>> worth clarifying.
>
> Unfortunately we're not in a place where it's feasible to do this with
> guaranteed success - though we welcome folks contributing if this is
> something that is important to you. We might revisit this story in 2018, but
> not before we've shipped something that lets people move off pub
> serve/build.

That sounds very reasonable to me.

> Pub as a package manager is likely to stay indefinitely (similar to NPM for
> Node). As far as a build system goes, we'll be focusing on build_runner.
> We're also still working out what the long-term entry-points for
> compilation, testing, deployment will be.

Thanks for the clarification here.

>> Based on the wording in the README, I would guess that Google
>> internally prefers the use of bazel over pub.
>
> Because we use a single mono-repo internally without branches, there is no
> reason to have dependency management, so a lot (read: all) of the benefits
> of pub sort of go away anyway.

Makes sense. The bazel WORKSPACE file also allows for granular dependency management (including external repositories). Therefore, if bazel had dart support, those users likely would also not get much value from pub.

Best,

Andy

[Matan Lurey]

Replies inline.

On Sun, Nov 26, 2017 at 1:12 PM Hochhaus, Andy <ahoc...@samegoal.com> wrote:

Thanks for the detailed reply and added context Matan.

I agree that bazel shouldn't be the only way to build dart code. Potentially, the build tooling could consider two use cases:

1. Smaller repositories (possibly only containing dart code) which prefer minimal setup & configuration. These users will likely prefer a dart specific tool set (such as pub or build_runner).
2. Larger existing projects containing code written in a wide variety of languages. These users will likely be willing to undertake additional configuration steps.
   

That's a sort of hypothetical long-term vision. I say hypothetical because we're trying to double down on the small-medium size case to completion first, and let the other cases be more use-case driven.

If you/your company would find immediate value from better Bazel support in Dart, please contact us and I'd be happy to schedule some sort of higher-level chat about your needs - you can email me at work (mat...@google.com).

 

On Sun, Nov 26, 2017 at 11:31 AM, Matan Lurey <ma...@lurey.org> wrote:
> Externally, we considered offering full Bazel support, but shied away:
>
> * Bazel at the time barely supported Windows, which is important to us.
> * Even today, lots of our Bazel integration is Unix-specific. It would take
> a lot of time, resources, and debugging to get this to work great
> out-of-the-box to Windows users, and even then Bazel is pretty heavyweight
> for some projects.

In the "two build systems" model suggested above, windows support need not be a requirement here. It would be acceptable to have windows only supported by the dart tooling (pub, build_runner) while macOS and linux were also supported by bazel.

Sure. Just until we have one build system e2e complete we've been trying not to focus on another that won't capture all of our users (in this case, many/most use Windows today).

 

Further, as bazel windows support has significantly improved, should adequate interest appear in using dart with bazel on windows it would be possible to fix those issues.

> So, we took build_runner, and tried building on it. And it turned out to
> work quite well. Basically, there is one user-level set of interfaces,
> package:build, with a set of contracts that if you follow work for pub
> transformers, bazel/blaze, and build_runner.

Can you say more here? Specifically, does any documentation exist on using these with bazel?

Not really, no, again just due to prioritization.

If you're really curious, you can look yourself at the bazel_codegen package. It is a Bazel-based implementation of the build package, though you will notice it rolls its own implementation for most of the compilers.

 

>> On Sun, Nov 26, 2017 at 10:29 AM, Ryan Gonzalez <rym...@gmail.com> wrote:
>> > Dazel basically takes in a pubspec and outputs Bazel rules.
>>
>> Makes sense. This matches my understanding.
>
> Dazel was based on the idea that we could infer a build system configuration
> based on your package configuration and a bit of extra metadata. We've
> actually kept this model as "build_config" for build_runner. The idea here
> is that users that consume generators would have to do little or no work.
>
> https://github.com/dart-lang/build/tree/master/build_config

Interesting. Are you sating that build_config can be used to generate bazel BUILD files?

That was the original goal. Today it will be used to generate build scripts.

Here is one such example.

 

The rules_go repo has a tool called gazelle which functions in a similar manner to dazel (walks a directory tree, parses *.go files, generates BUILD.bazel files). My dislike for dazel (and maybe build_config?) is that it depends on pubspec.yaml files which are "pub specific" files (vs simply dart source code). In the gazelle case only go source files are required.

Dazel currently isn't a serious project. Internally we have something else, called Daze, that doesn't use pub at all, which is closer to what you want. Again, something we might open source as prioritization calls for it.

 

[Hochhaus, Andy]

On Sun, Nov 26, 2017 at 1:20 PM, Matan Lurey <ma...@lurey.org> wrote:
> ... we're trying to double down on the small-medium size case to completion first, and let the other cases be more use-case driven.

Makes sense.

> If you/your company would find immediate value from better Bazel support in Dart, please contact us and I'd be happy to schedule some sort of higher-level chat about your needs - you can email me at work (mat...@google.com).

Thanks to your guidance I'm hopeful I can get something which is "good
enough for the time being" using build_runner. At the point that it
becomes prohibitive I will ping you.

> Dazel currently isn't a serious project. Internally we have something else, called Daze, that doesn't use pub at all, which is closer to what you want. Again, something we might open source as prioritization calls for it.

This sounds like it has potential to meet my long term needs.

Best,
Andy

[Bob Nystrom]

Everything Matan says here and below is spot-on. A bit more historical context / commentary:

On Sun, Nov 26, 2017 at 11:31 AM, Matan Lurey <ma...@lurey.org> wrote:

tl;dr, at one point we had no build system. Dart 1.0 was predicated around having a JIT (just-in-time) source-code VM, so similar to the JS ecosystem in ~2011/2012, there wasn't an apparent need to have a platform-level build system (remember, Dart->JS was just going to be for legacy browsers, and you did most browser development in Dartium).

Once pub, as a package manager, was built and shipped, there were various opinions on whether it or another tool should play a build/asset system role, and the current implementation of pub does via "pub serve", "pub build", and asset transformers (commonly called "transformers").

The reason we chose to bake the build system into pub is a little obscure, but I think it was based on a reasonable goal. The idea was this: Let's say you're using some groovy widget package called, uh, "dootstrap". It exposes a CSS file that your app can use. This is a bread-and-butter use case for a package manager like pub: You depend on the dootstrap package, and then you reference the CSS file using some path that the package manager sets up for you.

The use case we wanted to support was, that CSS file might actually be generated by a build step. The maintainers of dootstrap could actually author it as a SASS file and then build the CSS from that. To you, as a user of dootstrap/widgets.css, that's purely an implementation detail. All you care is that the CSS is made available to you somehow.

OK, so the easy solution is to have the dootstrap maintainers just rebuild the CSS file and check it into the package. In other words, the build step is run at maintainer-time before the package is shipped. That generally works OK, but consider a more complex use case:

Let's you're working on an application called my_rad_game. It uses some third-party 3D models (of goats — your game is about goats) from a package called goat_models. Those models are loaded and rendered by a 3D engine your game uses called engine_3d. So the goat_models packages needs to expose these models as files in a form that the engine can read and load at runtime. Again, the maintainers of goat_models could run the build step to generate those model files and add them the goat_models package before publishing it.

Here's where it gets tricky. Different versions of engine_3d may change the model format. Maybe engine_3d version 2.0.0 starts compressing floating point values or something. If the maintainers of goat_models ran an old version of engine_3d's model builder and I'm using a new version of engine_3d, then it won't be able to load the old models. I really need to ensure that the same version of engine_3d is used to build the models and the load them at runtime.

This is even harder because goat_models — like any good package — wants to support a range of versions of engine_3d. If it has to contain the pre-built models, that may lock it down to an artificially narrow range of versions and make it harder for users to consume the package.

So, our idea was that we would delay the build step until after pub has resolved all of your dependencies. Then it would run any build steps right then, using the exact versions of the dependencies that your application itself uses. That way, you're guaranteed to have your build steps and your runtime use of the resulting assets in sync with each other.

Since that needed to happen after dependency resolution had happened, we felt it made sense to make it part of the package manager.

(In retrospect, a separate build system that runs after pub also works.)

 

This worked well enough for small/medium-ish size projects, but broke down quite a bit once your project(s) got larger - designed to be an in-memory monolithic transformation system, not a scaleable, modular build system.

Pub build/serve is in-memory (and its one of the design choices we knew was risky at the time), but it's not monolithic. It actually has a pretty smart dependency graph system internally and knows exactly which transformations need to be re-run when a source file has changed.

The problem isn't that the build system is monolithic, it's that using it to generate Dart code that relies on static analysis is. Pub's build system was designed for building data, not statically-analyzed code. Worse, because Dart freely allows cyclic imports, it's even hard to statically analyze it in a modular fashion. That's why all of our other build tools have to do this extra work around finding "library cycles" and stuff like that. The language simply wasn't designed for modularity.

Using pub's build system for code generation is just a poor fit. It was the best tool we had at the time, but it was never designed for that task.

I'm really glad we're moving away from pub's build system to a separate one. Some of the design choices we made didn't pan out well, and a new build system can be better tailored for the use cases that we now know we have, like code generation. Also, it simplifies pub.

Cheers!

– bob

[Hochhaus, Andy]

Given the announcement of the new dart build tool today, could someone provide an update on the state of bazel support?

Best,

Andy

[Nate Bosch]

See the README at https://github.com/dart-lang/rules_dart

We're not officially supporting bazel for Dart externally. We have enough code there that it is technically possible to get Dart projects working - but we're not making guarantees so if something isn't working we might not prioritize a fix. We also use pieces of that code internally and when we make changes for our internal users it could break external use cases. We have not nailed down our long term strategy, so it's possible that external bazel support will become a goal.

[Hochhaus, Andy]

Thanks Nate.

--
For more ways to connect visit https://www.dartlang.org/community

---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/d52e7898-fdbd-4d24-8a2a-2436b27be1e4%40dartlang.org.