A program to generate a program to generate a program

[pauld...@gmail.com]

It seems typical in the Bazel ecosystem to have users run a program to generate a program to generate a program.

For example, (1) gazelle to generate (2) BUILD to generate a (3) binary.

Some examples I know of:

Gazelle - https://github.com/bazelbuild/rules_go
Angular - https://medium.com/@Jakeherringbone/building-angular-apps-at-scale-813ef42add04
Scala - internal tool at Stripe
Maven (ok, maybe bad example, as this is partially due to repeatability concerns) - https://github.com/pgr0ss/bazel-deps

This "configure-make" paradigm is common among build tools that lack abstraction capabilities (e.g., make), but this is an uncommon paradigm for many other non-Bazel build tools (e.g. gradle).

---

This encourages two behaviors, which are generally antipatterns in build systems:

(1) Checking products into version control, permitting them to get out of sync with their sources.
(2) Requiring the user to know which steps to run at the appropriate times.

Is this state of affairs due to historical accident, temporary limitations, or purposeful design? Is there a way to make Bazel a complete and automatic build tool? Or is there some essential complexity that can't really be solved in a 100% streamlined way?

[Damien Martin-Guillerez]

Hi Paul,

+Carmi Grushko 

BUILD file contains more than what the code contains but we have been talking a lot about integrating those step automatically but there are some shortage in most case. E..g. it is difficult to make that step incremental and  it is generally a slow step that rarely needs to be run. Go is of course special since the language contains build information in it and in most case the build files can be generated on the fly. IIUC Typescript has similar information in it.

AFAICT gradle is no different if you think of it, Android Studio is actually generating most of the gradle files...

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/617c70b6-409e-4964-bb9c-334b91b67517%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ittai zeidman]

Paul,
The stripe internal tool for scala means developers don't write BUILD files but it's automatically generated on the fly?
I'm interested since I wrote such a tool to migrate maven based scala/java codebases but it heavily relies on a code index since Scala allows a lot of freedom layout wise (multiple classes in same file and more)

[Ian O'Connell]

I'm not really sure whats being referred to in the first email, at Stripe we write our build files manually still.

We have some macros (and more internal variants  similarly) like https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L1014 to generate multiple library targets to speed up re-compiling.

We use https://github.com/johnynek/bazel-deps to handle external dependencies in the repos. (Yaml file that you update, it does the maven resolution + generates BUILD files in a 3rdparty folder tree with exports/runtime dependencies setup). But relative to normal operations this changes rarely enough (once or twice a day in the busier repos). 

Twitter with pants doesn't write much of their scala build files manually anymore i'm told? But for bazel we haven't deployed anything to auto-divide scala files or figure out their interdependencies to generate the finest build targets yet. 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/141bdcde-f50d-4ee2-9fd1-1998e0c25079%40googlegroups.com.

[Paul Draper]

> I'm not really sure whats being referred to in the first email, at Stripe we write our build files manually still.

I misremembered. This was actually a suggestion (not an extant implemenation) for fast Scala builds from P. Oscar Boykin:

> build a zinc rules that could use zinc to automatically create minimal build files for a directory. In this way, the user only manages he dependencies of one directory but then periodically regenerates static targets that are minimized. This may make the tiny build target approach more palatable.

https://groups.google.com/forum/#!topic/bazel-discuss/3iUy5jxS3S0

> AFAICT gradle is no different if you think of it, Android Studio is actually generating most of the gradle files...

I don't do Android development, but I doubt I'd depend on an IDE for creating a build process(!).

> Go is of course special since the language contains build information in it and in most case the build files can be generated on the fly. IIUC Typescript has similar information in it.

I'd like to be generated on the fly; it seems that they are generated via a manually initiated step. (Or do I not undersand this correctly?)

---

> E..g. it is difficult to make that step incremental and  it is generally a slow step that rarely needs to be run

Venerable make does this incrementally.

```

include deps.makefile

deps.makefile: source1 source2 source3

    command > $@

```

Generating BUILD files isn't necessarily the only way this problem has to be solved. I'm not yet famililar enough with Bazel to suggest what a solution might be (or maybe if one already exists, and gazelle, etc. choose to do things the hard way).

Intuition tells me there's a good way to not have users juggle multiple commands dependending on the part of the file that was edited.

On Thursday, August 31, 2017 at 9:27:27 AM UTC-6, Ian O'Connell wrote:
> I'm not really sure whats being referred to in the first email, at Stripe we write our build files manually still.
> 
> 
> We have some macros (and more internal variants  similarly) like https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L1014 to generate multiple library targets to speed up re-compiling.
> 
> 
> We use https://github.com/johnynek/bazel-deps to handle external dependencies in the repos. (Yaml file that you update, it does the maven resolution + generates BUILD files in a 3rdparty folder tree with exports/runtime dependencies setup). But relative to normal operations this changes rarely enough (once or twice a day in the busier repos). 
> 
> 
> 
> 
> Twitter with pants doesn't write much of their scala build files manually anymore i'm told? But for bazel we haven't deployed anything to auto-divide scala files or figure out their interdependencies to generate the finest build targets yet. 
> 
> 
> 
> 
> 
> 
> On Thu, Aug 31, 2017 at 6:42 AM, ittai zeidman <itt...@gmail.com> wrote:
> Paul,
> 
> The stripe internal tool for scala means developers don't write BUILD files but it's automatically generated on the fly?
> 
> I'm interested since I wrote such a tool to migrate maven based scala/java codebases but it heavily relies on a code index since Scala allows a lot of freedom layout wise (multiple classes in same file and more)
> 
> 
> 
> --
> 
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> 

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.