Flush long output

57 views
Skip to first unread message

Nicolas Desprès

unread,
Jul 28, 2012, 9:44:27 AM7/28/12
to ninja-build
Hi,

Although, Ninja's short output status is fine 99% of the time, it is not always the case. For long lasting commands generating a long output like configuration stage, documentation generation, test executions, etc...

I tried to address this issue in this pull-request: https://github.com/martine/ninja/pull/382. Specially the case of the configuration stage. It introduces a new rule attribute. This attribute is just a hint for flushing the output. If the command is run in parallel of other jobs, no flush happens. Actually, this most of the time the case. You can also use a variable to flush on a per build basis. I needed this behavior for the CMake generator (the CUSTOM_COMMAND rule is used in many different context some requiring to flush and some not) but I finally did not use it since it would have been too intrusive in the CMake code. I have rather added a -F flag. It is way easier to implement, use and it does the job for documentation generation, test executions, etc...

Comments?

Cheers,
Nico

Evan Martin

unread,
Jul 28, 2012, 2:12:59 PM7/28/12
to ninja...@googlegroups.com
I wonder, can we do this automatically? Whenever we're running a
single command, we know that all the rest of the build is blocked on
that command completing. We won't start up another command in
parallel until that command completes, so it's safe to print its
output while it runs.

Related issue:
https://github.com/martine/ninja/issues/111

Nicolas Desprès

unread,
Jul 28, 2012, 3:46:07 PM7/28/12
to ninja...@googlegroups.com


On Saturday, July 28, 2012, Evan Martin <mar...@danga.com> wrote:
> I wonder, can we do this automatically?  Whenever we're running a
> single command, we know that all the rest of the build is blocked on
> that command completing.  We won't start up another command in
> parallel until that command completes, so it's safe to print its
> output while it runs.

A friend of mine suggested roughly the safe. I have to confess that it is simpler. However, I know how to test if there is one task in the plan, but I don't know how to test if a task is a connection point between two sub-graph in the graph. Maybe, just the first test would be enough.

>
> Related issue:
>   https://github.com/martine/ninja/issues/111

Did not about this one.

PS: Sorry for the poor English of my previous post.


>
> On Sat, Jul 28, 2012 at 6:44 AM, Nicolas Desprès
> <nicolas...@gmail.com> wrote:
>> Hi,
>>
>> Although, Ninja's short output status is fine 99% of the time, it is not
>> always the case. For long lasting commands generating a long output like
>> configuration stage, documentation generation, test executions, etc...
>>
>> I tried to address this issue in this pull-request:
>> https://github.com/martine/ninja/pull/382. Specially the case of the
>> configuration stage. It introduces a new rule attribute. This attribute is
>> just a hint for flushing the output. If the command is run in parallel of
>> other jobs, no flush happens. Actually, this most of the time the case. You
>> can also use a variable to flush on a per build basis. I needed this
>> behavior for the CMake generator (the CUSTOM_COMMAND rule is used in many
>> different context some requiring to flush and some not) but I finally did
>> not use it since it would have been too intrusive in the CMake code. I have
>> rather added a -F flag. It is way easier to implement, use and it does the
>> job for documentation generation, test executions, etc...
>>
>> Comments?
>>
>> Cheers,
>> Nico
>>
>

--
Nicolas Desprès

Mathias Gaunard

unread,
Jul 29, 2012, 10:16:14 AM7/29/12
to ninja...@googlegroups.com
On Saturday, July 28, 2012 9:46:07 PM UTC+2, Nicolas Desprès wrote:

On Saturday, July 28, 2012, Evan Martin wrote:
> I wonder, can we do this automatically?  Whenever we're running a
> single command, we know that all the rest of the build is blocked on
> that command completing.  We won't start up another command in
> parallel until that command completes, so it's safe to print its
> output while it runs.

A friend of mine suggested roughly the safe. I have to confess that it is simpler. However, I know how to test if there is one task in the plan, but I don't know how to test if a task is a connection point between two sub-graph in the graph. Maybe, just the first test would be enough.

I believe there are two situations where output should be flushed:
- when a command is run, and we know that no other command will be run or none will be run until the command finishes. Dependencies and -j N can affect this.
- when a command finishes, a command already running could now satisfy the above, in which case its output should be flushed as well.

Doing this would cover all use cases automatically. Not only would it not need explicit flushing on the user part, it would also necessarily be safe and guarantee that output of commands running in parallel will never be interleaved.

Nicolas Desprès

unread,
Jul 31, 2012, 8:24:22 AM7/31/12
to ninja...@googlegroups.com
Well. I kind of agree with that, but I can't find an easy way to implement it. I have pushed a new version in the branch auto_flush: https://github.com/martine/ninja/pull/387. This version is fully automatic. It does not cover all the cases mentioned by Mathias but at least it covers the main ones: configuration, tests execution, documentation, etc... I think it will be enough to satisfy what people have been asking for on the CMake mailing list.

Cheers,
Nico

Nicolas Desprès

unread,
Aug 6, 2012, 3:39:33 AM8/6/12
to ninja...@googlegroups.com
I have closed this pull request based on Maxim Kalaev's remarks. I'll try to send a new proposal soon.

--
Nicolas Desprès

Reply all
Reply to author
Forward
0 new messages