Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Capturing output of parallel make

27 views
Skip to first unread message

James

unread,
Apr 24, 2017, 4:21:55 PM4/24/17
to
How to capture the output of parallel make (make -jN) without truncation?

Is there a message logging utility?

Can strace be used for this purpose?

TIA
James

Bit Twister

unread,
Apr 24, 2017, 4:57:02 PM4/24/17
to
On Mon, 24 Apr 2017 13:20:12 -0700 (PDT), James wrote:
> How to capture the output of parallel make (make -jN) without truncation?

Have you tried something like
script -c "make -jN" mk_par.log
and checked mk_par.log.

Kaz Kylheku

unread,
Apr 24, 2017, 5:55:20 PM4/24/17
to
On 2017-04-24, James <hsle...@yahoo.com> wrote:
> How to capture the output of parallel make (make -jN) without truncation?

What truncation?

Dan Espen

unread,
Apr 24, 2017, 6:00:19 PM4/24/17
to
James <hsle...@yahoo.com> writes:

> How to capture the output of parallel make (make -jN) without truncation?

I've run a lot of make -jN runs.
I don't remember any truncation.
Make can interleave output, the jobs are running
in parallel after all.

If you see real truncation perhaps you should file a bug report.


> Is there a message logging utility?

No. Make just lets the commands it is running do their thing.

> Can strace be used for this purpose?

No, not in any reasonable way.

--
Dan Espen

James

unread,
Apr 24, 2017, 7:25:34 PM4/24/17
to
What I meant was the output was interleaved. It is not about make, but about logging in parallel. For example,

for f in `seq 1000`; do
(echo Printing $f `date`; sleep 3) &
done > output
wait

James



Kaz Kylheku

unread,
Apr 24, 2017, 7:59:42 PM4/24/17
to
On 2017-04-24, James <hsle...@yahoo.com> wrote:
I would solve this problem by fixing the makefile.

Decent makefiles nowadays don't display the steps, but a summary of what
is going on which is output by make itself. That output doesn't get
interleaved: e.g.

CC foo.o
CC bar.o
LINK program foo.o bar.o

The nice macros which achieve this sort of thing can easily be extended
with the feature of capturing the command output and then assembling it
into a coherent log.

Or, heck, **not** assembling it into a single log.

You can have it so that when a target like foo.o is updated (or
fails to be), the build step and the output of that build step are
logged in foo.log.

If you want a big log, you can find all the *.log files,
sort them by modification time stamp and catenate.

Ivan Shmakov

unread,
Apr 25, 2017, 3:50:45 AM4/25/17
to
>>>>> Kaz Kylheku <686-67...@kylheku.com> writes:
>>>>> On 2017-04-24, James <hsle...@yahoo.com> wrote:

>> What I meant was the output was interleaved. It is not about make,
>> but about logging in parallel. For example,

>> for f in `seq 1000`; do (echo Printing $f `date`; sleep 3) & done >
>> output ; wait

> I would solve this problem by fixing the makefile.

> Decent makefiles nowadays don't display the steps, but a summary of
> what is going on which is output by make itself. That output
> doesn't get interleaved: e.g.

> CC foo.o
> CC bar.o
> LINK program foo.o bar.o

... The feature that makes troubleshooting so easy that
I invariably have to turn it off to have my challenge back...

[...]

--
FSF associate member #7257 58F8 0F47 53F5 2EB2 F6A5 8916 3013 B6A0 230E 334A

Icarus Sparry

unread,
Apr 28, 2017, 7:48:43 PM4/28/17
to
On Mon, 24 Apr 2017 23:59:36 +0000, Kaz Kylheku wrote:

> On 2017-04-24, James <hsle...@yahoo.com> wrote:
>> What I meant was the output was interleaved. It is not about make, but
>> about logging in parallel. For example,
>>
>> for f in `seq 1000`; do
>> (echo Printing $f `date`; sleep 3) &
>> done > output wait
>
> I would solve this problem by fixing the makefile.
>
> Decent makefiles nowadays don't display the steps, but a summary of what
> is going on which is output by make itself. That output doesn't get
> interleaved: e.g.
>
> CC foo.o
> CC bar.o
> LINK program foo.o bar.o

Short answer:
The output can still be interleaved, it typically isn't because the
entire "CC foo.o" is output as a single write call.

These days GNU make has a "-Otarget" option which causes make itself to
buffer all the output for each target and output it in a non interleaved
manner.

Qualifications:
If you are not doing a recursive make,
and your make program is single threaded (like GNU make)
and you are actually doing the outputting from make itself rather than
using an external program like "echo" as the linux kernel build system
does,
then
you can have a "Decent makefile" in the sense that Kaz is using it.

James

unread,
May 1, 2017, 7:37:34 PM5/1/17
to
Using flock might do the job.

( flock -x -w 600 200; \
make -j10; \
)200>/dev/null

James

Joerg.S...@fokus.fraunhofer.de

unread,
May 3, 2017, 10:10:04 AM5/3/17
to
In article <d54ae58c-e51b-4271...@googlegroups.com>,
James <hsle...@yahoo.com> wrote:
>How to capture the output of parallel make (make -jN) without truncation?

I am not sure what you men by "truncation"....

SunPro make collects the output and error streams for every parallel build task
and later combines them in a way that grants a non-mixed output.

If you like a portable version of SunPro make, you may fetch the "schilytools"
tarball. It includes a portable version of the SuunPro Make program.

--
EMail:jo...@schily.net (home) Jörg Schilling D-13353 Berlin
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/
0 new messages