Add tool for bots to download a web directory. (issue 10391151)

[chri...@chromium.org]

Reviewers: cmp,

Message:
I use to do this using rsync before moving to buildbot. I'm not sure if
there's
a better way to do this.

Description:
Add tool for bots to download a web directory.

This script allow buildbot slaves to download dependency files before it run
tests. For example 30+ chromebots will retrieve the list of urls file and
binary files that are not to be checked in the repository.

TEST=None
BUG=None

Please review this at https://chromiumcodereview.appspot.com/10391151/

SVN Base: http://src.chromium.org/svn/trunk/tools/build/

Affected files:
A scripts/slave/download_from_url.py

[c...@chromium.org]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode1
scripts/slave/download_from_url.py:1: #!/usr/bin/python
there's a general problem with this approach that makes it hard to
reproduce builds

for example, we cannot go back and say "rebuild the earlier version" and
get the same result since the contents of the files at the given URL may
have changed.

i don't think this is right. i think instead you should find a way to
check these dependencies in somewhere and reference them from a deps
file. how big are they? are they all binaries? how often do they
change? do they change from build to build?

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode77
scripts/slave/download_from_url.py:77: 'If ommited, default URL will be
used.')
ommited -> omitted

later you wrote:
if not options.url:
...
return 1

so if it's omitted, it looks like the script will exit. that's fine if
it's the case, just drop the "if omitted" line from the help doc.

https://chromiumcodereview.appspot.com/10391151/

[chri...@google.com]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode1
scripts/slave/download_from_url.py:1: #!/usr/bin/python

On 2012/05/16 17:53:19, cmp wrote:
> there's a general problem with this approach that makes it hard to
reproduce
> builds

> for example, we cannot go back and say "rebuild the earlier version"
and get the
> same result since the contents of the files at the given URL may have
changed.

> i don't think this is right. i think instead you should find a way to
check
> these dependencies in somewhere and reference them from a deps file.
how big
> are they? are they all binaries? how often do they change? do they
change
> from build to build?

The files:

top_million_urls 20megs

minidump_stackwalk 1meg
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/processor/minidump_stackwalk.cc

crash_report 4megs
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/tools/mac/crash_report/crash_report.mm

minidump_stackwalk and crash_report just parse the dump files. They
don't change from build to build. So I was thinking these are the files
that I can build on one machine and distribute to the rest of the test
machines.

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode77
scripts/slave/download_from_url.py:77: 'If ommited, default URL will be
used.')

On 2012/05/16 17:53:19, cmp wrote:
> ommited -> omitted

> later you wrote:
> if not options.url:
> ...
> return 1

> so if it's omitted, it looks like the script will exit. that's fine
if it's the
> case, just drop the "if omitted" line from the help doc.

Done.

https://chromiumcodereview.appspot.com/10391151/

[c...@chromium.org]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode1
scripts/slave/download_from_url.py:1: #!/usr/bin/python

Okay, as for the size of the files, I don't think it will be a problem.
Have you tried gzip compressing these files to see how much that saves
us? And if the data needs to be internal, we can check them into an
internal repo. Once they're in a repo, we can reference them from the
DEPS file and then we won't need this script. (Any new updates will be
pulled via the update_scripts step.)

https://chromiumcodereview.appspot.com/10391151/

[chri...@google.com]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download_from_url.py#newcode1
scripts/slave/download_from_url.py:1: #!/usr/bin/python

On 2012/05/16 20:23:19, cmp wrote:
> Okay, as for the size of the files, I don't think it will be a
problem. Have
> you tried gzip compressing these files to see how much that saves us?
And if
> the data needs to be internal, we can check them into an internal
repo. Once
> they're in a repo, we can reference them from the DEPS file and then
we won't
> need this script. (Any new updates will be pulled via the
update_scripts step.)

gzip saves about 50%. It doesn't need to be internal. Where should I
put them?

https://chromiumcodereview.appspot.com/10391151/

[c...@chromium.org]

In that case this could go into chrome/trunk/tools/chromebot. WDYT?

https://chromiumcodereview.appspot.com/10391151/

[chri...@google.com]

On 2012/05/16 20:48:42, cmp wrote:
> In that case this could go into chrome/trunk/tools/chromebot. WDYT?

Actually awhile ago nirnimesh told me not to check these in. But it would
nice
to check them in somewhere.

https://chromiumcodereview.appspot.com/10391151/

[c...@chromium.org]

+nirnimesh, was there a specific reason not to check them in besides the
binary
size?

https://chromiumcodereview.appspot.com/10391151/

[Chris Phan]

This is for the files:

top_million_urls

minidump_stackwalk (process linux crash dumps)

crash_report (process mac crash dumps)

[nirn...@chromium.org]

Chris, all these 3 files (top_million, minidump_stackwalk, crash_report)
should
be checked in.

https://chromiumcodereview.appspot.com/10391151/