[dart-announce] Dart 2 intent to remove: the build and serve commands from pub

['Kevin Moore' via Dart Announcements]

pub is getting out of the build system business and going back to its roots: being an amazing package manager for the Dart ecosystem.

In the next few weeks, we plan to remove support for the build and serve commands from pub. This change will happen in our regular dev channel releases.

Don't panic!

All of the package management features will remain unchanged: get, upgrade, run, publish, and friends will continue to work as you expect.

Replacing pub build and pub serve

If you use pub build and pub serve to develop and deploy web applications, we've outlined the migration steps in our Dart 2 Migration Guide. The documentation for package:build_runner is also a good place to start.

Not ready to migrate off pub build and pub serve?

That's okay. Your workflow will not be affected if either of these is true:

• You're on a Dart 1.x release.

• You don't upgrade to the latest Dart 2-dev until you have updated your workflow to use build_runner.

Where can I get help?

Start with the links provided. If you have questions or get stuck, post a question on Stack Overflow or join us on the dart-lang/build Gitter room. If you find an issue, please file it in the dart-lang/build repository.

--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups "Dart Announcements" group.
Visit this group at https://groups.google.com/a/dartlang.org/group/announce/.

[Kevin Moore]

FYI:

Nate just posted a write-up with details about our build system evolution: http://lessworsemorebetter.com/2018/04/17/dart-builds-how-did-we-get-here.html

Almost in the same breath, he also landed the pull request that removes build/serve from pub. (Points for naming the branch "kill-bill".)

To follow the overall progress for this change in the SDK, subscribe to https://github.com/dart-lang/sdk/issues/32593

[tatumizer-v0.2]

A bit off topic:  I'd like to see some abstract mathematical definition of the problem the build is trying to solve. I tried to google it, but either my keywords were wrong, or said definition simply doesn't exist.

I mean, a definition in terms of graphs, or constraint solvers, or something like that. The very fact that we have  race conditions during the build (mentioned in Nate's article) strongly suggests that the build isn't fully aware of what problem it's trying to solve, and instead just *doing something* towards an unknown goal :)

My conjecture is that as soon as the problem gets defined, it will turn out that the only way to solve it is to honestly write a program (procedural one, in a Turing-complete language) that does it for a given specific case. This would make any shortcuts via declarative syntax impossible in principle. I'm not sure though.

[Bob Nystrom]

Barback can get itself into weird race conditions because of the combination of a few things:

1. Cyclic dependencies between packages are allowed. Dart the language has always allowed cyclic imports and they are widely used in practice. Because of that, we felt it import that pub allow cyclic package dependencies. Cycles at that granularity aren't in common, but they do happen, especially around dev dependencies. For example, the test package uses args for its arg parsing, and args uses test for its tests.
   
   
2. The set of inputs and outputs for a transformer is imperatively determined. Some build systems like Bazel require you to explicitly declaratively list all of the inputs and outputs of each build step. This lets you construct the entire build graph before performing any work, which is nice. The downside is that it can be verbose, tedious, and error prone to write all of those down. Consider a transformer that takes a folder full of PNG files and produces one big sprite sheet out of all of them. In barback, you can write a transformer that imperatively walks the input directory, discovers the files, and lists them as inputs. Drop a new PNG in there and it automatically gets picked up.
   
   Barback was designed for relatively simple dependency graphs with lots of small assets — things like spriting images, transpiling CSS, etc. so it optimized for ease-of-use over untangling giant dependency graphs. But, since Dart has no other metaprogramming story, barback ended up being the de facto solution for things like Angular template compilation and other much more complex build scenarios that it wasn't intended for.
   
   
3. A transformer can change a file "in place". Barback allows you to produce an output file whose path is the same as a previous input file. We felt this was important to allow transforming files that are referenced by other files. Consider a transformer that, say, compiles async/await in Dart files to vanilla future code. If we don't allow transformers to overwrite files, then the transformer has to output that file to some new path. But all of the other Dart files importing that one are still importing the old path and won't pick up the transformed one. So you'd have to transform those too to fix their imports. But then those files need new paths, so you need to transform everything that imports them. The result is you have to transform every single Dart file just to make a change to a single one.

When you put these together, you can get into this situation:

1. Package a reads an input from package b.
   
   
2. We need to wait for b's transformers to finish running before that input can be provided because we don't know if it will be produced (rule 2 above) or modified (rule 3) by some transformer in b.
   
   
3. But package b also reads an input from package a (rule 1), so we need to wait for *its* transformers to run.
   
   
4. Deadlock.
   

I'm not an expert on the new build stuff, but I believe it addresses these like so:

1. I don't think it allows dependency cycles between packages at all. Or it may allow them between packages but not between build targets, which are fine-grained than entire packages.
   
   
2. Inputs and outputs are explicitly listed, though I think some tooling is provided to auto-generate these manifests for common cases.
   
   
3. You can't output to a path where a file already exists. No "overwriting". Instead, all build steps produce new files at new paths. This means those outputs can be written to disc. Thus, to solve the "modify a single file without breaking references to it" problem, you write your imports to point to the *output* path not the *original* path. Since the build outputs will be on disc, tools like IDEs can find them. It's a little weird because it means when authoring code, you need to be more aware of what the build system outputs where, but it simplifies a lot of other problems.
   

Cheers!

– bob

--
For more ways to connect visit https://www.dartlang.org/community
---

You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.
To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-58c011f66c1f%40dartlang.org.

[Jake Macdonald]

1. I don't think it allows dependency cycles between packages at all. Or it may allow them between packages but not between build targets, which are fine-grained than entire packages.

It does allow package cycles, but it essentially tries to treat them like "one" package as much as possible.

Essentially, for any package which is not in a cycle we order the phases such that all its dependencies are built in their entirety first. However, if there is a package cycle then we choose a deterministic ordering between the two packages, and apply each builder one at a time to each package.

As an example, if there is a cycle between package "a" and package "b", and they each have the angular, summary, and ddc builders applied, then we would add conceptually these phases:

1. angular on package:a
2. angular on package:b
3. summaries on package:a
4. summaries on package:b
5. ddc on package:a
6. ddc on package:b

If they were not in a cycle, say "a" depends on "b" but not the other way around, the ordering would look like:

1. angular on package:b
2. summaries on package:b
3. ddc on package:b
4. angular on package:a
5. summaries on package:a
6. ddc on package:a

This "just works, in all existing real world scenarios we know of, although you could certainly devise a scenario where it wouldn't do what you wanted if you really tried. For that case the new build system also allows you to create a custom build script if the automatic ordering doesn't work for you, where you can choose the exact ordering of all the builders.

1. Inputs and outputs are explicitly listed, though I think some tooling is provided to auto-generate these manifests for common cases.
   

Yes builder authors have to declare their mapping of input files to output files. This allows us to build the graph ahead of time.

[Nate Bosch]

bcc ann...@dartlang.org

On Thu, Apr 19, 2018 at 9:36 AM 'Bob Nystrom' via Dart Misc <mi...@dartlang.org> wrote:

Barback can get itself into weird race conditions because of the combination of a few things:

1. Cyclic dependencies between packages are allowed. Dart the language has always allowed cyclic imports and they are widely used in practice. Because of that, we felt it import that pub allow cyclic package dependencies. Cycles at that granularity aren't in common, but they do happen, especially around dev dependencies. For example, the test package uses args for its arg parsing, and args uses test for its tests.
   
   
2. The set of inputs and outputs for a transformer is imperatively determined. Some build systems like Bazel require you to explicitly declaratively list all of the inputs and outputs of each build step. This lets you construct the entire build graph before performing any work, which is nice. The downside is that it can be verbose, tedious, and error prone to write all of those down. Consider a transformer that takes a folder full of PNG files and produces one big sprite sheet out of all of them. In barback, you can write a transformer that imperatively walks the input directory, discovers the files, and lists them as inputs. Drop a new PNG in there and it automatically gets picked up.
   
   Barback was designed for relatively simple dependency graphs with lots of small assets — things like spriting images, transpiling CSS, etc. so it optimized for ease-of-use over untangling giant dependency graphs. But, since Dart has no other metaprogramming story, barback ended up being the de facto solution for things like Angular template compilation and other much more complex build scenarios that it wasn't intended for.
   
   
3. A transformer can change a file "in place". Barback allows you to produce an output file whose path is the same as a previous input file. We felt this was important to allow transforming files that are referenced by other files. Consider a transformer that, say, compiles async/await in Dart files to vanilla future code. If we don't allow transformers to overwrite files, then the transformer has to output that file to some new path. But all of the other Dart files importing that one are still importing the old path and won't pick up the transformed one. So you'd have to transform those too to fix their imports. But then those files need new paths, so you need to transform everything that imports them. The result is you have to transform every single Dart file just to make a change to a single one.

When you put these together, you can get into this situation:

1. Package a reads an input from package b.
   
   
2. We need to wait for b's transformers to finish running before that input can be provided because we don't know if it will be produced (rule 2 above) or modified (rule 3) by some transformer in b.
   
   
3. But package b also reads an input from package a (rule 1), so we need to wait for *its* transformers to run.
   
   
4. Deadlock.
   

I'm not an expert on the new build stuff, but I believe it addresses these like so:

1. I don't think it allows dependency cycles between packages at all. Or it may allow them between packages but not between build targets, which are fine-grained than entire packages.
   

We really _wanted_ to disallow dependency cycles but we found it wasn't practical in our current ecosystem. If and when we revisit bazel support we will need to disallow target-level cycles. Today build_runner allows either and we interleave builders across them so there is a supported solution to "I need information from other sources in a cycle" which is to split up your builder into multiple steps.
 

       2. Inputs and outputs are explicitly listed, though I think some tooling is provided to auto-generate these manifests for common cases.

Outputs are all determined by file extensions, inputs are allowed dynamically. The important thing is that the set of possible outputs is static.
 

        3. You can't output to a path where a file already exists. No "overwriting". Instead, all build steps produce new files at new paths. This means those outputs can be written to disc. Thus, to solve the "modify a single file without breaking references to it" problem, you write your imports to point to the *output* path not the *original* path. Since the build outputs will be on disc, tools like IDEs can find them. It's a little weird because it means when authoring code, you need to be more aware of what the build system outputs where, but it simplifies a lot of other problems.

Cheers!

– bob

On Wed, Apr 18, 2018 at 5:48 PM, tatumizer-v0.2 <tatu...@gmail.com> wrote:

A bit off topic:  I'd like to see some abstract mathematical definition of the problem the build is trying to solve. I tried to google it, but either my keywords were wrong, or said definition simply doesn't exist.

I mean, a definition in terms of graphs, or constraint solvers, or something like that. The very fact that we have  race conditions during the build (mentioned in Nate's article) strongly suggests that the build isn't fully aware of what problem it's trying to solve, and instead just *doing something* towards an unknown goal :)

My conjecture is that as soon as the problem gets defined, it will turn out that the only way to solve it is to honestly write a program (procedural one, in a Turing-complete language) that does it for a given specific case. This would make any shortcuts via declarative syntax impossible in principle. I'm not sure though.

--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.
To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-58c011f66c1f%40dartlang.org.

--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/CAF8T8Lz6ORU1a6XGfc8YxJpaefX0Rs%2Bx%3DKG%2BaezAXAiMtcWRTg%40mail.gmail.com.

['Kevin Moore' via Dart Announcements]

Dart SDK 2.0.0-dev.51.0 has just been released with this change.