collect .ninja_log (to find out slowness in build time)

[Fumitoshi Ukai (鵜飼文敏)]

Hi,

I'm thinking to collect .ninja_log from bots to analyze where is a bottleneck in compile step.

My plan is fix compile.py to upload .ninja_log to cloud storage (gs://chrome-goma-log?), and setup AppEngine app to show analyzed results.

what do you think?

which cloud storage bucket to use?

is there any other data source to investigate build speed?

https://docs.google.com/a/chromium.org/document/d/16TdPTIIZbtAarXZIMJdiT9CePG5WYCrdxm5u9UuHXNY/edit

Thanks,

ukai

[Paweł Hajdan, Jr.]

On Mon, Aug 18, 2014 at 9:00 AM, Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org> wrote:

Hi,

I'm thinking to collect .ninja_log from bots to analyze where is a bottleneck in compile step.

My plan is fix compile.py to upload .ninja_log to cloud storage (gs://chrome-goma-log?), and setup AppEngine app to show analyzed results.

what do you think?

Sounds great to me. Uploading compiler_proxy logs was a big step forward in investigating goma failures.

 

which cloud storage bucket to use?

It seems chrome-goma-log would be a good start.

 

is there any other data source to investigate build speed?

https://docs.google.com/a/chromium.org/document/d/16TdPTIIZbtAarXZIMJdiT9CePG5WYCrdxm5u9UuHXNY/edit

I couldn't get access to the above doc.

Paweł

 

Thanks,

ukai

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE8n1nf3VB-N4_nd1izM3Jg1%2BB17oxpaewg8YTkKTA%2BbVTAEbw%40mail.gmail.com.

[Fumitoshi Ukai (鵜飼文敏)]

2014-08-18 22:54 GMT+09:00 Paweł Hajdan, Jr. <phajd...@chromium.org>:

On Mon, Aug 18, 2014 at 9:00 AM, Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org> wrote:

Hi,

I'm thinking to collect .ninja_log from bots to analyze where is a bottleneck in compile step.

My plan is fix compile.py to upload .ninja_log to cloud storage (gs://chrome-goma-log?), and setup AppEngine app to show analyzed results.

what do you think?

Sounds great to me. Uploading compiler_proxy logs was a big step forward in investigating goma failures.

 

which cloud storage bucket to use?

It seems chrome-goma-log would be a good start.

 

is there any other data source to investigate build speed?

https://docs.google.com/a/chromium.org/document/d/16TdPTIIZbtAarXZIMJdiT9CePG5WYCrdxm5u9UuHXNY/edit

I couldn't get access to the above doc.

Sorry, I forgot setting share option.

I changed it to Anyone who has the link can comment.

Thanks,

ukai

[Fumitoshi Ukai (鵜飼文敏)]

I've landed change of compile.py to upload .ninja_log to gs://chrome-goma-log at the end of compile step.

Next, I'd like to develop AppEngine app to show stats, trace view or something like that.

code location: infra/appengine/chromium_build_stats (or infra_internal/ ?)

app id: chromium-build-stats

runtime: go

Any comments/suggestions?

Thanks,

ukai

[Eric Seidel]

Do ninja_logs contain build errors? If so, I expect the
sheriff-o-matic feeder (infra/tools/builder_alerts) might like to pull
down/parse these logs instead of stdio for understanding compile
errors:
https://chromium.googlesource.com/infra/infra/+/master/infra/tools/builder_alerts/reasons_splitter.py#296

On Wed, Aug 20, 2014 at 8:06 AM, Fumitoshi Ukai (鵜飼文敏)

> https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE8n1ncJBAuXqo%3D8B71DWKN-fQmhgypDM9%3D9sooDtZi6-1NN-Q%40mail.gmail.com.

[Fumitoshi Ukai (鵜飼文敏)]

ninja_log doesn't contain build error. it only has timing logs (when each target is started/ended)

compile.py adds exit status of ninja in json at the end of ninja_log when uploaded, but it doesn't have any more information about compile error (error message or so)

[Ojan Vafai]

While we're talking about this, is there any structured way to get at the list of compile failures or is stdio parsing really the only way? If stdio is the only way, how hard would to be to create a structured (i.e. json) output as well? This would help make sheriff-o-matic work a ton better for compile failures if it could actually know which file (and ideally line and specific error message) the error happened in.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE8n1nfWdBc7a3nJC3Mtz8U73%3D14a7bVAEkZcmYd0Kz2PK%2BOVA%40mail.gmail.com.

[Nico Weber]

There's no way. I think ninja always prints "failed:", so looking for that should be fairly reliable.

[Ojan Vafai]

Would it be hard to add a flag to ninja that causes it to spit out a json file?

[Eric Seidel]

Or whatever file type would be appropriate for ninja, but somehow
separate the output from the failing command from previous passing
commands. :)

[Nico Weber]

Is it hard to look for "failed:" strings? It'd have to write json to a second file since streaming json to build output would look bad, the json file would be much larger than the current output, etc.

[Ojan Vafai]

With the stdio output from the tests, we've had a surprisingly large number of bugs around issues scraping the output format (esp as the format has changed over time), so I'm sensitive to this. But, maybe the ninja output format is so simple and unchanging that none of this matters.

We already parse the log output and get something out of it. It's not complicated code: https://chromium.googlesource.com/infra/infra.git/+/master/infra/tools/builder_alerts/reasons_splitter.py#296. But it would give us a more reliable, future-proof signal.

BTW, do linker errors output the same way?

Log-parsing just isn't the most reliable thing. What if the ninja output format changes slightly in some edge case? Or, currently ninja does a thing where it ellipsizes content that's too large for the terminal. That doesn't affect the bots now (I think?), but you could imagine something like this happening. It's a lossy system and the design of the output is for human beings, not machines. As you make future design decisions about the output format, backwards-compatibility for machines parsing the output (rightfully) won't be your primary concern. 

You already have the data in a structured form, no?

Why would the json file be much larger? Isn't it just putting the failure data in a structured form?

FWIW, I agree that it would of course have to be written to a second file.

[Scott Graham]

On Wed, Aug 20, 2014 at 8:06 AM, Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org> wrote:

I've landed change of compile.py to upload .ninja_log to gs://chrome-goma-log at the end of compile step.

Next, I'd like to develop AppEngine app to show stats, trace view or something like that.

code location: infra/appengine/chromium_build_stats (or infra_internal/ ?)

app id: chromium-build-stats

runtime: go

Any comments/suggestions?

Hi,

Great, I'm glad they're being saved now. That's these lines

C:\b\build\scripts\slave\gsutil.bat cp file://c:\users\chrome~1\appdata\local\temp\1\tmp4viuk5 gs://chrome-goma-log/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368.gz

at the bottom of the compile step?

 

As a first step, it would be helpful to have a link emitted there, something like:

"""

Visit http://chromium-build-stats.appspot.com/view/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368 to view compile timing data.

"""

If it's not too hard, a first step might just be to have that autoconvert like https://github.com/nico/ninjatracing and make the .json available for download to view in about:tracing.

Directly serving the about:tracing UI with the data loaded would be even nicer (but might still be helpful to have the download available).

Of course, many other views of statistics across builds might be helpful, but I'm not sure what I'd want to know yet.

nico/ninjatracing mentions getting better data via either a clean or -t recompact first. Is it possible to approximate that after the fact given that we don't do that in compile.py?

--
You received this message because you are subscribed to the Google Groups "Chromium Hackability Code Yellow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hackability-c...@chromium.org.
To post to this group, send email to hackabi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/hackability-cy/CAE8n1ncJBAuXqo%3D8B71DWKN-fQmhgypDM9%3D9sooDtZi6-1NN-Q%40mail.gmail.com.

[Fumitoshi Ukai (鵜飼文敏)]

2014-08-22 7:03 GMT+09:00 Scott Graham <sco...@chromium.org>:

On Wed, Aug 20, 2014 at 8:06 AM, Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org> wrote:

I've landed change of compile.py to upload .ninja_log to gs://chrome-goma-log at the end of compile step.

Next, I'd like to develop AppEngine app to show stats, trace view or something like that.

code location: infra/appengine/chromium_build_stats (or infra_internal/ ?)

app id: chromium-build-stats

runtime: go

Any comments/suggestions?

Hi,

Great, I'm glad they're being saved now. That's these lines

C:\b\build\scripts\slave\gsutil.bat cp file://c:\users\chrome~1\appdata\local\temp\1\tmp4viuk5 gs://chrome-goma-log/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368.gz

at the bottom of the compile step?

 

As a first step, it would be helpful to have a link emitted there, something like:

"""

Visit http://chromium-build-stats.appspot.com/view/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368 to view compile timing data.

"""

Yeah, I'm planning to do it at first once chromium-build-stats are ready.

 (waiting for https://code.google.com/p/chromium/issues/detail?id=397543, and https://codereview.chromium.org/489703003/ and then land https://codereview.chromium.org/467303003/ )

If it's not too hard, a first step might just be to have that autoconvert like https://github.com/nico/ninjatracing and make the .json available for download to view in about:tracing.

I think it's not so hard.  experimental code could be found in google3/experimental/users/ukai/go/ninja_log/tracer.go

 

Directly serving the about:tracing UI with the data loaded would be even nicer (but might still be helpful to have the download available).

Of course, many other views of statistics across builds might be helpful, but I'm not sure what I'd want to know yet.

nico/ninjatracing mentions getting better data via either a clean or -t recompact first. Is it possible to approximate that after the fact given that we don't do that in compile.py?

I think ninja reads/outputs .ninja_log at startup, at that point, ninja_log may have log entries in random order.

while building, ninja emits log entries as soon as target finishes, so we would get log entries for the last build in ascending order of end time.

Thanks,

ukai

[Fumitoshi Ukai (鵜飼文敏)]

2014-08-22 14:24 GMT+09:00 Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org>:

2014-08-22 7:03 GMT+09:00 Scott Graham <sco...@chromium.org>:

On Wed, Aug 20, 2014 at 8:06 AM, Fumitoshi Ukai (鵜飼文敏) <uk...@chromium.org> wrote:

I've landed change of compile.py to upload .ninja_log to gs://chrome-goma-log at the end of compile step.

Next, I'd like to develop AppEngine app to show stats, trace view or something like that.

code location: infra/appengine/chromium_build_stats (or infra_internal/ ?)

app id: chromium-build-stats

runtime: go

Any comments/suggestions?

Hi,

Great, I'm glad they're being saved now. That's these lines

C:\b\build\scripts\slave\gsutil.bat cp file://c:\users\chrome~1\appdata\local\temp\1\tmp4viuk5 gs://chrome-goma-log/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368.gz

at the bottom of the compile step?

 

As a first step, it would be helpful to have a link emitted there, something like:

"""

Visit http://chromium-build-stats.appspot.com/view/2014/08/21/build51-m1/ninja_log.build51-m1.chrome-bot.20140821-143447.3368 to view compile timing data.

"""

Yeah, I'm planning to do it at first once chromium-build-stats are ready.

 (waiting for https://code.google.com/p/chromium/issues/detail?id=397543, and https://codereview.chromium.org/489703003/ and then land https://codereview.chromium.org/467303003/ )

I've deployed initial version of chromium-build-stats (w/  https://codereview.chromium.org/561633002/ ) to http://chromium-build-stats.appspot.com.

-- 

ukai