Using shake as a backend for overloaded ant systems

[Black]

Hello,

I'm currently working on a product that use an overloaded ant build system (you have to understand that you have a custom XML that is interpreted and partially given to ant). The build mechanism is quiet complicated, you have an OOP-like language based on XML with optional parts and so on. The build process is also rather complicated, they compile some JAVA with a kind of options, some other with other options, they get different outputs, they extract some informations to generate some C files (and headers). Then they compile the C to a myriad of binaries with different targets. My main concern is that it's damn slow.
I don't have the time to produce a complete migration of this system, my goal is pretty simple: let the current system make the "orchestration" job and change the build chain. My idea is to replace all the command calls to a "tracer system" and generate a Shakefile executed each time I need it.

Have you ever met such kind of needs?
Is there already some ant-to-shake utilities?
Have you got any other ideas to tackle this problem (a speed problem)?
Have you any starting points in mind?

Thanks in advance for your help.
Regards.

[Neil Mitchell]

Hi,

Running a tracer to capture the command lines could work, although you'll also need to capture the dependencies so Shake can rebuild the right pieces. Once you have that information you can generate a Shake Haskell file, or generate text and load it up and interpret it with Shake, or even generate a Ninja file and execute that with Shake (or Ninja, if that was easier for you). 

To get a size of scale, how many lines of XML ant files do you have? It may turn out to be simpler to port directly, although that very much depends on the size. 

Finally, Shake is exceptionally well suited to writing interpreters for other build systems. It ships with a complete Ninja interpreter, that as far as I can tell, is faster and might actually have fewer bugs than Ninja. It ships with a proof of concept Make interpreter that can deal with tutorial examples, but little real-world Make. The Ant format looks a lot simpler than Make, so perhaps Shake could interpret it directly and you could just type "shake" instead of "ant". 

Thanks, Neil 

[Gautier DI FOLCO]

Hello,
Thanks for your answer.

2015-06-16 23:25 GMT+02:00 Neil Mitchell <ndmit...@gmail.com>:

Hi,

Running a tracer to capture the command lines could work, although you'll also need to capture the dependencies so Shake can rebuild the right pieces. Once you have that information you can generate a Shake Haskell file, or generate text and load it up and interpret it with Shake, or even generate a Ninja file and execute that with Shake (or Ninja, if that was easier for you). 

Good idea

 

To get a size of scale, how many lines of XML ant files do you have? It may turn out to be simpler to port directly, although that very much depends on the size. 

Around 5 millions of LoC in 5.000 files, so it's too to be ported without help.

 

Finally, Shake is exceptionally well suited to writing interpreters for other build systems. It ships with a complete Ninja interpreter, that as far as I can tell, is faster and might actually have fewer bugs than Ninja. It ships with a proof of concept Make interpreter that can deal with tutorial examples, but little real-world Make. The Ant format looks a lot simpler than Make, so perhaps Shake could interpret it directly and you could just type "shake" instead of "ant". 

I'm afraid to not even know the whole rules :/

 

Thanks for all :)

[Neil Mitchell]

>> To get a size of scale, how many lines of XML ant files do you have? It
>> may turn out to be simpler to port directly, although that very much depends
>> on the size.
>
> Around 5 millions of LoC in 5.000 files, so it's too to be ported without
> help.

I mean just the size of the Ant build-description files (which is all
that needs porting for the build system), not all the source. I really
hope you don't have 5M LoC of build files! (I typically find big
projects tap out around 10K)

>> Finally, Shake is exceptionally well suited to writing interpreters for
>> other build systems. It ships with a complete Ninja interpreter, that as far
>> as I can tell, is faster and might actually have fewer bugs than Ninja. It
>> ships with a proof of concept Make interpreter that can deal with tutorial
>> examples, but little real-world Make. The Ant format looks a lot simpler
>> than Make, so perhaps Shake could interpret it directly and you could just
>> type "shake" instead of "ant".
>
> I'm afraid to not even know the whole rules :/

You don't need to know all the rules for Ant, just the ones which your
project makes use of. It also has the benefit of benefiting everyone,
and the work can be shared much better (it's a general problem, not
something specific to your company).

Thanks, Neil

[Gautier DI FOLCO]

2015-06-17 13:32 GMT+02:00 Neil Mitchell <ndmit...@gmail.com>:

> Around 5 millions of LoC in 5.000 files, so it's too to be ported without
> help.

I mean just the size of the Ant build-description files (which is all
that needs porting for the build system), not all the source. I really
hope you don't have 5M LoC of build files! (I typically find big
projects tap out around 10K)

What do you exactly call 'Ant build-description files'?

5M LoC is the sum of the non-blank and non-commented lines in the XML files used by our tool to build our product.

 

>> Finally, Shake is exceptionally well suited to writing interpreters for
>> other build systems. It ships with a complete Ninja interpreter, that as far
>> as I can tell, is faster and might actually have fewer bugs than Ninja. It
>> ships with a proof of concept Make interpreter that can deal with tutorial
>> examples, but little real-world Make. The Ant format looks a lot simpler
>> than Make, so perhaps Shake could interpret it directly and you could just
>> type "shake" instead of "ant".
>
> I'm afraid to not even know the whole rules :/

You don't need to know all the rules for Ant, just the ones which your
project makes use of. It also has the benefit of benefiting everyone,
and the work can be shared much better (it's a general problem, not
something specific to your company).

Thanks, Neil

Ok, I have to do some research to find them in that case.

[Neil Mitchell]

> What do you exactly call 'Ant build-description files'?
> 5M LoC is the sum of the non-blank and non-commented lines in the XML files
> used by our tool to build our product.

Wow, that is what I was thinking of. That's a lot more than I was
expecting! I really didn't know build systems came that large...

Given that size, I suspect porting it by hand to Shake is infeasible.
If it has a fairly regular structure, then extracting the information
from it to a Shake file might work.

Thanks, Neil

[Gautier DI FOLCO]

2015-06-17 13:18 GMT+00:00 Neil Mitchell <ndmit...@gmail.com>:

Wow, that is what I was thinking of. That's a lot more than I was
expecting! I really didn't know build systems came that large...

Well, let 100 engineers work on a system that they don't understand during 7 years and you get that mess.

 

Given that size, I suspect porting it by hand to Shake is infeasible.
If it has a fairly regular structure, then extracting the information
from it to a Shake file might work.

Sadly it's a directories hierarchy nightmare, with different kinds of plugins with a more or less copy and paste configuration, it will be difficult without a bit a clean up.

[Neil Mitchell]

Are you hoping to speed up the time to do a zero build when nothing
has changed, or the time to do a complete rebuild? What is the current
zero build time?

Looking at Ant, it might be more reasonable to describe it as a DSL
for running statements, rather than as a build system. There are
dependencies, but it's not specified in a traditional "make" style
dependency with one file having some dependencies, it's more a script
that can be executed linearly and with some parallelism and sharing.
While some build systems work that way (fabricate springs to mind,
https://code.google.com/p/fabricate/), it's relatively rare. As such,
it may turn out to be quite difficult to convert into a
dependency-driven build system. I'm sure many simple Ant build systems
can be converted to a dependency-driven system, but if you have 5M
lines there's inevitably going to be plenty that can't. I'm not sure
there will be "easy" wins in converting to Shake.

[Gautier DI FOLCO]

2015-06-17 20:09 GMT+00:00 Neil Mitchell <ndmit...@gmail.com>:

Are you hoping to speed up the time to do a zero build when nothing
has changed, or the time to do a complete rebuild? What is the current
zero build time?

The full build (all parts the first time) takes between 18 minutes to 24 minutes.
An incremental build (one file has been modified) takes 7-10 minutes.
If there is no modification, it takes 4-5 minutes.
It's way too long (even Clang compilation time is smaller and Clang is full of templates).
 

Looking at Ant, it might be more reasonable to describe it as a DSL
for running statements, rather than as a build system. There are
dependencies, but it's not specified in a traditional "make" style
dependency with one file having some dependencies, it's more a script
that can be executed linearly and with some parallelism and sharing.
While some build systems work that way (fabricate springs to mind,
https://code.google.com/p/fabricate/), it's relatively rare. As such,
it may turn out to be quite difficult to convert into a
dependency-driven build system. I'm sure many simple Ant build systems
can be converted to a dependency-driven system, but if you have 5M
lines there's inevitably going to be plenty that can't. I'm not sure
there will be "easy" wins in converting to Shake.

I think there are no smart parts in our build part, it's too complicated, very few of us have this knowledge, most of us are just copy and paste old stuff.
I'm pretty sure that it will be a huge win to convert it it to Shake, but I will be the only to master it, so to maintain it, to yell at people and so on. My goal is to make my life easier and maybe one day I'll share it and maybe my management will hear of that and will take a decision about it but I don't care if my work is easier.