Official release source tarballs

[Eric Hameleers]

Hi

An official release tarball of the chromium 32.0.170.77 sources is nowhere to be found. I found the 32.0.1700.76 sources, but those are supposedly only for Windows, right?

As announced, "The Chrome Team is excited to announce the promotion of Chrome 32 to the Stable channel. 32.0.1700.76 for Windows and Chrome Frame and 32.0.1700.77 for Mac and Linux".

As a result of my failed search I started wondering how the official source tarballs are created. The src/tools/export_tarball/export_tarball.py script creates a tarball that is way bigger than an official tarball (almost 650 MB instead of a little over 100 MB). Also the gclient seems to have skipped the checkout of the v8 directory and who knows what else.

When can we expect a tarball for chromium 32.0.1700.77 sources? 

Eric

[Saikrishna Arcot]

With regards to the script, I believe using the
--remove-nonessential-files option gets rid of unnecessary files (those
listed in NONESSENTIAL_DIRS), and should cut down on the size of the
tarball.

Saikrishna Arcot

On Wed 15 Jan 2014 08:50:12 AM EST, Eric Hameleers wrote:
> Hi
>
> An official release tarball of the chromium 32.0.170.77 sources is
> nowhere to be found. I found the 32.0.1700.76 sources, but those are
> supposedly only for Windows, right?
> As announced, "The Chrome Team is excited to announce the promotion of
> Chrome 32 to the Stable channel. 32.0.1700.76 for Windows and Chrome
> Frame and 32.0.1700.77 for Mac and Linux".
>
> As a result of my failed search I started wondering how the official
> source tarballs are created.
> The src/tools/export_tarball/export_tarball.p

> <http://src.chromium.org/svn/trunk/src/tools/export_tarball/export_tarball.py>y
> script creates a tarball that is *way* bigger than an official tarball

> (almost 650 MB instead of a little over 100 MB). Also the gclient
> seems to have skipped the checkout of the v8 directory and who knows
> what else.
>
> When can we expect a tarball for chromium 32.0.1700.77 sources?
>
> Eric
>

> --
> You received this message because you are subscribed to the Google
> Groups "chromium-packagers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to chromium-packag...@chromium.org.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/3ef1dfc3-f072-4e56-8995-a5c1bee9473c%40chromium.org.

[Chad Miller]

On Wed, Jan 15, 2014 at 8:50 AM, Eric Hameleers <al...@slackware.com> wrote:

When can we expect a tarball for chromium 32.0.1700.77 sources? 

It takes several hours to create and upload. If it's not here 12 hours after release announcement, something may be wrong.

- chad

Ubuntu

[René Ladan]

2014/1/15 Chad Miller <chad....@canonical.com>:

Still not working here:
fetch: http://commondatastorage.googleapis.com/chromium-browser-official/chromium-32.0.1700.77.tar.xz:
Not Found

René

[Saikrishna Arcot]

It seems the last tarball published was almost 48 hours ago (or more
than 48 hours in case the times are GMT). Perhaps something is wrong
with the builder?

Saikrishna Arcot

[Eric Hameleers]

I was able to create a chromium-32.0.1700.77.tar.xz source tarball and compiled a working package from it. Still testing to verify that I did not miss anything but I am typing in the new browser at the moment.

I translated the steps I executed into a script - beware, I did not actually run it yet:

#

# Checking out the sources of an official chromium release, by Eric Hameleers <alien at slackware.com>:

#

# In a working directory, you will create three directories:

# - depot_tools: this contains all the tools needed for creating the tarball

# - checkout: here almost 7 GB of sourcecode will be checked out

# - output: here the chromium-${RELEASE}.tar.xz source tarball will be created.

# Some variables:

WORKING_DIR="${HOME}/chromium_src"

CHROMIUM_REL="32.0.1700.77"

DEPOT_TOOLS_REPO="https://chromium.googlesource.com/chromium/tools/depot_tools.git"

RELEASES_URL="https://src.chromium.org/chrome/releases"

# The actual work (takes a while to checkout 1 GB of source and pack it up:

mkdir -p ${WORKING_DIR}

cd ${WORKING_DIR}

# Clone the tools:

git clone ${DEPOT_TOOLS_REPO}

# Check out the sources:

mkdir ckeckout

cd checkout

PATH="../depot_tools/:$PATH" gclient config ${RELEASES_URL}/${CHROMIUM_REL}

# At this moment, your config will have been written to a file '.gclient'

PATH="../depot_tools/:$PATH" gclient sync 2>&1 | tee ../sync.log

# The sync will download everything, based on what it found in this file:

# ${CHROMIUM_REL}/DEPS 

PATH="../depot_tools/:$PATH" gclient sync 2>&1 | tee ../sync.2.log

# Pack up the source tarball:

mkdir ../output

python src/tools/export_tarball/export_tarball.py --remove-nonessential-files ../output/chromium-${CHROMIUM_REL}

echo ""

echo "Resulting source tarball is: ${WORKING_DIR}/output/chromium-${CHROMIUM_REL}.tar.xz"

echo ""

It produced a 169MB tarball. I needed a couple of "gclient sync" commands before I finally had all the source code (hopefully all... I have no way of checking if anything is missing).

Eric 

[Paweł Hajdan, Jr.]

Sorry about the breakage.

I've now identified the root cause and it's the gsutil repository that moved from http://gsutil.googlecode.com/svn/trunk/src to https://github.com/GoogleCloudPlatform/gsutil

For some reason the first URL seems to be still working for Googlers, but not for anyone else. That's probably how this was only detected now.

I'm working on fixing this, and I'll update this thread with progress info.

You should have good chance of success with taking https://src.chromium.org/chrome/releases/32.0.1700.77/DEPS and removing all references to gsutil and syncing from that.

Paweł

--

You received this message because you are subscribed to the Google Groups "chromium-packagers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-packag...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/d359b9a6-6842-443e-995d-1f565b018cf0%40chromium.org.

[Paweł Hajdan, Jr.]

Please see https://code.google.com/p/chromium/issues/detail?id=334327 for more info. I continue the efforts to fix this.

Paweł

[Paweł Hajdan, Jr.]

Tarball for 32.0.1700.77 is available now. I apologize for the delays.

Tarball for 33.0.1750.29 should appear within 24 hours.

I'm also working on systems used to generate these tarballs to make them more robust. One improvement hopefully coming soon is uploading failure logs to https://gsdview.appspot.com/chromium-browser-official/ so anyone can see the errors, and so that we can notice these problems faster.

Paweł

[Paweł Hajdan, Jr.]

Tarballs for beta release 33.0.1750.29 are available now.

Paweł

[Paweł Hajdan, Jr.]

To help with further breakages, there'll now be a .log file uploaded for every release we fail to package, e.g. https://gsdview.appspot.com/chromium-browser-official/chromium-34.0.1785.0.log .

Currently it's one line, but one way to more easily see it would be "cat ~/Downloads/chromium-34.0.1785.0.log  | sed 's/\\n/\n/g' | less"

I'll also get an e-mail alerting me about these errors.

By the way, currently the log file sometimes contains gibberish because gclient sync is running several jobs in parallel and apparently they get all mixed up. I plan to eventually address that too (possibly with just forcing a single job).

Paweł

[Raymond Wooninck]

On Tuesday 21 January 2014 14:22:33 Paweł Hajdan, Jr. wrote:
> To help with further breakages, there'll now be a .log file uploaded for
> every release we fail to package, e.g.
> https://gsdview.appspot.com/chromium-browser-official/chromium-34.0.1785.0.l
> og.

Hi Pawel,

This time the tarball for the new DEV channel build was available directly
with the announcement. I only noticed that somehow the date of the files are
set to 1 January 1970 :)

Also the new DEV channel build now seems to require depot_tools to be
installed during build. It would be great if this somehow could be announced
as well to save some additional debugging work :)

Thanks for the great job.

Regards

Raymond
openSUSE Chromium packager

[Paweł Hajdan, Jr.]

On Tue, Jan 21, 2014 at 2:27 PM, Raymond Wooninck <titti...@gmail.com> wrote:

On Tuesday 21 January 2014 14:22:33 Paweł Hajdan, Jr. wrote:
> To help with further breakages, there'll now be a .log file uploaded for
> every release we fail to package, e.g.
> https://gsdview.appspot.com/chromium-browser-official/chromium-34.0.1785.0.l
> og.

This time the tarball for the new DEV channel build was available directly
with the announcement. I only noticed that somehow the date of the files are
set to 1 January 1970 :)

Oh, another weird bug.

 

Also the new DEV channel build now seems to require depot_tools to be
installed during build. It would be great if this somehow could be announced
as well to save some additional debugging work :)

You probably mean GN, right? Please see https://code.google.com/p/chromium/issues/detail?id=336441 . I think it's also related to your earlier post https://groups.google.com/a/chromium.org/d/msg/chromium-packagers/zqWr0MHxNbo/1dxLH9P2QvYJ .

My opinion is that for dev channel I don't always announce things since they change so quickly... but questions are fine. Also, sometimes I don't start packaging a dev channel release immediately after it's released. In fact, sometimes I've skipped releases focusing on upstream instead. I might do that for the GN bootstrapping bug.

Paweł

[Saikrishna Arcot]

I believe he's referring to the following stacktrace, and changes 245412 (primary) and 245705.

Failed to find depot_tools
Traceback (most recent call last):
  File "build/gyp_chromium", line 41, in <module>
    import find_depot_tools
  File "/build/buildd/chromium-browser-34.0.1797.2/src/tools/find_depot_tools.py", line 40, in <module>
    import breakpad
ImportError: No module named breakpad

Apparently, this change is mainly for WIndows and Cygwin, but the import line gets applied to all OSes, and so Linux gets pulled in into this as well.

Saikrishna Arcot

--
You received this message because you are subscribed to the Google Groups "chromium-packagers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-packag...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CAATLsPaq3Ch6oUcVqNrt807dhNnVsLW2hNCYmPx7nkYB_zkM8g%40mail.gmail.com.

[Paweł Hajdan, Jr.]

The above should be fixed by https://codereview.chromium.org/137203009/

Paweł

[Raymond Wooninck]

Hi Pawel,

On Tuesday 21 January 2014 18:37:04 Paweł Hajdan, Jr. wrote:
> > Also the new DEV channel build now seems to require depot_tools to be
> > installed during build. It would be great if this somehow could be
> > announced
> > as well to save some additional debugging work :)
>
> You probably mean GN, right? Please see
> https://code.google.com/p/chromium/issues/detail?id=336441 . I think it's
> also related to your earlier post
> https://groups.google.com/a/chromium.org/d/msg/chromium-packagers/zqWr0MHxNb
> o/1dxLH9P2QvYJ.

Nope. I have the GN binaries already there and this time it really failed on
the search for depot_tools (especially on the part "import breakpad"). Adding
the depot_tools tarball into the sourcepad and set the path to include this
depot_tools directory resolved the build issue.

Regards

Raymond

[Lei Zhang]

Are logs only uploaded in case of a failure?

Someone on #chromium mentioned the courgette folder is missing files
in chromium-36.0.1933.0.tar.xz. This is indeed the case, but I have no
idea how to diagnose it.

On Thu, Jan 16, 2014 at 3:39 PM, Paweł Hajdan, Jr.

> https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CAATLsPajFReh2mdmWT%3Do5Aok%2BXb0hn%3DO-PryyZVxre1K9S4eqA%40mail.gmail.com.

[Raymond Wooninck]

On Thursday 10 April 2014 23:16:51 Lei Zhang wrote:
> Are logs only uploaded in case of a failure?
>
> Someone on #chromium mentioned the courgette folder is missing files
> in chromium-36.0.1933.0.tar.xz. This is indeed the case, but I have no
> idea how to diagnose it.
>

Hi Lei,

As far as I can tell, this directory has always been empty in the release
source tarballs. This is a listing of the file of the latest Stable source
tarball:

hq000006@HQVMT4XX20
/sources/openSUSE/network:chromium/chromium/chromium-34.0.1847.116/courgette %
ls -l
total 16
-rw-r--r-- 1 hq000006 users 6438 Apr 2 21:03 courgette.gyp
drwxr-xr-x 2 hq000006 users 4096 Apr 2 21:03 testdata
drwxr-xr-x 2 hq000006 users 4096 Apr 2 21:03 third_party

So I guess that with the latest Dev build, a dependency was created that now
require courgette to be part of the release source tarball. Or that dependency
has to be reverted again.

Regards

Raymond
openSUSE Chromium maintainer

[Raymond Wooninck]

On Friday 11 April 2014 14:53:28 Raymond Wooninck wrote:
> On Thursday 10 April 2014 23:16:51 Lei Zhang wrote:
> > Someone on #chromium mentioned the courgette folder is missing files
> > in chromium-36.0.1933.0.tar.xz. This is indeed the case, but I have no
> > idea how to diagnose it.

Ok, after some checking it seems that the build issue is being generated due
to the content/content_utility.gypi file that has a reference to courgette.

This change is based on the following review
https://codereview.chromium.org/25883006/

I am currently building the Dev source tarball with a patched version of
content_utility.gypi (removed the courgette line), but given this commit I am
not sure if this will build or not.

Regards

Raymond

[Paweł Hajdan, Jr.]

We're no longer removing it, please see https://groups.google.com/a/chromium.org/d/msg/chromium-packagers/6hDjpnB4c-U/-vBUz1FMUTsJ

Thanks for reporting this.

Paweł

--
You received this message because you are subscribed to the Google Groups "chromium-packagers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-packag...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/16082978.tR7CPUAssx%40hqvmt4xx20.eur.cchbc.com.