Get `gn gen` to succeed on Windows (issue 694353003 by scottmg@chromium.org)

[sco...@chromium.org]

Reviewers: jam,

Message:
fyi

Description:
Get `gn gen` to succeed on Windows

1. Pull in NSS for net, crypto
2. Copy sqlite from chromium for NSS
3. Some mucking in build/ to avoid DEPSing gyp

I believe 1&2 can be deleted later once net is moved out.

This gets as far as successful project generation, and builds a bunch
of stuff but there's still quite a few compile errors.

Result of `ninja -C out\Debug root -k0 >log.txt` here:
https://gist.github.com/sgraham/7e58a022595db1c0ae60

They don't look too major, but I don't know if mojo wants to take on
longer term support/maintenance of Windows, with bots/etc. so not sure
this is worth it or not.

(To work fully this also requires a ninja binary at ninja ToT that's
not pushed to depot_tools yet. But the compile errors are unrelated,
and ninja will get there eventually.)

R=j...@chromium.org
BUG=

Please review this at https://codereview.chromium.org/694353003/

Base URL: https://github.com/domokit/mojo.git@master

Affected files (+493996, -7551 lines):
M .gitignore
M DEPS
M build/config/compiler/BUILD.gn
M build/toolchain/win/BUILD.gn
M build/vs_toolchain.py
A build/win/win_tool.py
M net/BUILD.gn
A third_party/sqlite/BUILD.gn
A third_party/sqlite/LICENSE
A third_party/sqlite/OWNERS
A third_party/sqlite/README.chromium
A third_party/sqlite/amalgamation/README
A third_party/sqlite/amalgamation/sqlite3.h
A + third_party/sqlite/build_as_cpp.cc
A third_party/sqlite/fts2.patch
A third_party/sqlite/fts3.patch
A third_party/sqlite/fts3_85522.patch
A third_party/sqlite/google_generate_amalgamation.sh
A third_party/sqlite/icu-shell.patch
A third_party/sqlite/mac_time_machine.patch
A third_party/sqlite/memcmp.patch
A third_party/sqlite/misalignment.patch
A third_party/sqlite/misc.patch
A third_party/sqlite/recover.patch
A third_party/sqlite/separate_cache_pool.patch
A third_party/sqlite/sqlite.gyp
A
third_party/sqlite/sqlite-3.7.6.3-fix-out-of-scope-memory-reference.patch
A third_party/sqlite/sqlite3.h
A third_party/sqlite/src/Makefile.arm-wince-mingw32ce-gcc
A third_party/sqlite/src/Makefile.in
A third_party/sqlite/src/Makefile.linux-gcc
A third_party/sqlite/src/Makefile.vxworks
A third_party/sqlite/src/README
A third_party/sqlite/src/VERSION
A + third_party/sqlite/src/aclocal.m4
A third_party/sqlite/src/addopcodes.awk
A third_party/sqlite/src/art/2005osaward.gif
A third_party/sqlite/src/art/SQLite.eps
A third_party/sqlite/src/art/SQLite.gif
A third_party/sqlite/src/art/SQLiteLogo3.tiff
A third_party/sqlite/src/art/SQLite_big.gif
A third_party/sqlite/src/art/nocopy.gif
A third_party/sqlite/src/art/powered_by_sqlite.gif
A third_party/sqlite/src/art/sqlite370.eps
A third_party/sqlite/src/art/sqlite370.ico
A third_party/sqlite/src/art/sqlite370.jpg
A third_party/sqlite/src/art/src_logo.gif
A + third_party/sqlite/src/config.guess
A + third_party/sqlite/src/config.sub
A third_party/sqlite/src/config.h.in
A + third_party/sqlite/src/configure
A third_party/sqlite/src/configure.ac
A third_party/sqlite/src/contrib/sqlitecon.tcl
A third_party/sqlite/src/doc/lemon.html
A third_party/sqlite/src/doc/pager-invariants.txt
A third_party/sqlite/src/doc/vfs-shm.txt
A third_party/sqlite/src/ext/README.txt
A third_party/sqlite/src/ext/async/README.txt
A third_party/sqlite/src/ext/async/sqlite3async.h
A third_party/sqlite/src/ext/async/sqlite3async.c
A third_party/sqlite/src/ext/fts1/README.txt
A third_party/sqlite/src/ext/fts1/ft_hash.h
A third_party/sqlite/src/ext/fts1/ft_hash.c
A third_party/sqlite/src/ext/fts1/fts1.h
A third_party/sqlite/src/ext/fts1/fts1.c
A third_party/sqlite/src/ext/fts1/fts1_hash.h
A third_party/sqlite/src/ext/fts1/fts1_hash.c
A third_party/sqlite/src/ext/fts1/fts1_porter.c
A third_party/sqlite/src/ext/fts1/fts1_tokenizer.h
A third_party/sqlite/src/ext/fts1/fts1_tokenizer1.c
A third_party/sqlite/src/ext/fts1/fulltext.h
A third_party/sqlite/src/ext/fts1/fulltext.c
A third_party/sqlite/src/ext/fts1/simple_tokenizer.c
A third_party/sqlite/src/ext/fts1/tokenizer.h
A third_party/sqlite/src/ext/fts2/README.tokenizers
A third_party/sqlite/src/ext/fts2/README.txt
A third_party/sqlite/src/ext/fts2/fts2.h
A third_party/sqlite/src/ext/fts2/fts2.c
A third_party/sqlite/src/ext/fts2/fts2_hash.h
A third_party/sqlite/src/ext/fts2/fts2_hash.c
A third_party/sqlite/src/ext/fts2/fts2_icu.c
A third_party/sqlite/src/ext/fts2/fts2_porter.c
A third_party/sqlite/src/ext/fts2/fts2_tokenizer.h
A third_party/sqlite/src/ext/fts2/fts2_tokenizer.c
A third_party/sqlite/src/ext/fts2/fts2_tokenizer1.c
A third_party/sqlite/src/ext/fts2/mkfts2amal.tcl
A third_party/sqlite/src/ext/fts3/README.syntax
A third_party/sqlite/src/ext/fts3/README.tokenizers
A third_party/sqlite/src/ext/fts3/README.txt
A third_party/sqlite/src/ext/fts3/fts3.h
A third_party/sqlite/src/ext/fts3/fts3.c
A third_party/sqlite/src/ext/fts3/fts3Int.h
A third_party/sqlite/src/ext/fts3/fts3_aux.c
A third_party/sqlite/src/ext/fts3/fts3_expr.c
A third_party/sqlite/src/ext/fts3/fts3_hash.h
A third_party/sqlite/src/ext/fts3/fts3_hash.c
A third_party/sqlite/src/ext/fts3/fts3_icu.c
A third_party/sqlite/src/ext/fts3/fts3_porter.c
A third_party/sqlite/src/ext/fts3/fts3_snippet.c
A third_party/sqlite/src/ext/fts3/fts3_tokenizer.h
A third_party/sqlite/src/ext/fts3/fts3_tokenizer.c
A third_party/sqlite/src/ext/fts3/fts3_tokenizer1.c
A third_party/sqlite/src/ext/fts3/fts3_write.c
A third_party/sqlite/src/ext/fts3/fts3speed.tcl
A third_party/sqlite/src/ext/fts3/mkfts3amal.tcl
A third_party/sqlite/src/ext/icu/README.txt
A third_party/sqlite/src/ext/icu/icu.c
A third_party/sqlite/src/ext/icu/sqliteicu.h
A third_party/sqlite/src/ext/rtree/README
A third_party/sqlite/src/ext/rtree/rtree.h
A third_party/sqlite/src/ext/rtree/rtree.c
A third_party/sqlite/src/ext/rtree/rtree1.test
A third_party/sqlite/src/ext/rtree/rtree2.test
A third_party/sqlite/src/ext/rtree/rtree3.test
A third_party/sqlite/src/ext/rtree/rtree4.test
A third_party/sqlite/src/ext/rtree/rtree5.test
A third_party/sqlite/src/ext/rtree/rtree6.test
A third_party/sqlite/src/ext/rtree/rtree7.test
A third_party/sqlite/src/ext/rtree/rtree8.test
A third_party/sqlite/src/ext/rtree/rtree9.test
A third_party/sqlite/src/ext/rtree/rtreeA.test
A third_party/sqlite/src/ext/rtree/rtreeB.test
A third_party/sqlite/src/ext/rtree/rtree_perf.tcl
A third_party/sqlite/src/ext/rtree/rtree_util.tcl
A third_party/sqlite/src/ext/rtree/sqlite3rtree.h
A third_party/sqlite/src/ext/rtree/tkt3363.test
A third_party/sqlite/src/ext/rtree/viewrtree.tcl
A + third_party/sqlite/src/install-sh
A + third_party/sqlite/src/ltmain.sh
A third_party/sqlite/src/main.mk
A third_party/sqlite/src/manifest
A third_party/sqlite/src/manifest.uuid
A third_party/sqlite/src/mkdll.sh
A third_party/sqlite/src/mkextu.sh
A third_party/sqlite/src/mkextw.sh
A third_party/sqlite/src/mkopcodec.awk
A third_party/sqlite/src/mkopcodeh.awk
A third_party/sqlite/src/mkso.sh
A third_party/sqlite/src/publish.sh
A third_party/sqlite/src/publish_osx.sh
A third_party/sqlite/src/spec.template
A third_party/sqlite/src/sqlite.pc.in
A third_party/sqlite/src/sqlite3.1
A third_party/sqlite/src/sqlite3.pc.in
A third_party/sqlite/src/src/alter.c
A third_party/sqlite/src/src/analyze.c
A third_party/sqlite/src/src/attach.c
A third_party/sqlite/src/src/auth.c
A third_party/sqlite/src/src/backup.c
A third_party/sqlite/src/src/bitvec.c
A third_party/sqlite/src/src/btmutex.c
A third_party/sqlite/src/src/btree.h
A third_party/sqlite/src/src/btree.c
A third_party/sqlite/src/src/btreeInt.h
A third_party/sqlite/src/src/build.c
A third_party/sqlite/src/src/callback.c
A third_party/sqlite/src/src/complete.c
A third_party/sqlite/src/src/ctime.c
A third_party/sqlite/src/src/date.c
A third_party/sqlite/src/src/delete.c
A third_party/sqlite/src/src/expr.c
A third_party/sqlite/src/src/fault.c
A third_party/sqlite/src/src/fkey.c
A third_party/sqlite/src/src/func.c
A third_party/sqlite/src/src/global.c
A third_party/sqlite/src/src/hash.h
A third_party/sqlite/src/src/hash.c
A third_party/sqlite/src/src/hwtime.h
A third_party/sqlite/src/src/insert.c
A third_party/sqlite/src/src/journal.c
A third_party/sqlite/src/src/legacy.c
A third_party/sqlite/src/src/lempar.c
A third_party/sqlite/src/src/loadext.c
A third_party/sqlite/src/src/main.c
A third_party/sqlite/src/src/malloc.c
A third_party/sqlite/src/src/mem0.c
A third_party/sqlite/src/src/mem1.c
A third_party/sqlite/src/src/mem2.c
A third_party/sqlite/src/src/mem3.c
A third_party/sqlite/src/src/mem5.c
A third_party/sqlite/src/src/memjournal.c
A third_party/sqlite/src/src/mutex.h
A third_party/sqlite/src/src/mutex.c
A third_party/sqlite/src/src/mutex_noop.c
A third_party/sqlite/src/src/mutex_os2.c
A third_party/sqlite/src/src/mutex_unix.c
A third_party/sqlite/src/src/mutex_w32.c
A third_party/sqlite/src/src/notify.c
A third_party/sqlite/src/src/os.h
A third_party/sqlite/src/src/os.c
A third_party/sqlite/src/src/os_common.h
A third_party/sqlite/src/src/os_os2.c
A third_party/sqlite/src/src/os_unix.c
A third_party/sqlite/src/src/os_win.c
A third_party/sqlite/src/src/pager.h
A third_party/sqlite/src/src/pager.c
A third_party/sqlite/src/src/parse.y
A third_party/sqlite/src/src/pcache.h
A third_party/sqlite/src/src/pcache.c
A third_party/sqlite/src/src/pcache1.c
[[ 771 additional files ]]

[sco...@chromium.org]

Actually, after I DEPS bison and gperf, it's not terrible looking. ~10-20
linux-ism. Maybe the right time to do this is once net is moved out, though
I
guess various things will rot more the longer you wait.

https://codereview.chromium.org/694353003/

[j...@chromium.org]

lgtm, thanks!

re the ninja binary, I'm not sure what you mean that this needs a new
binary but
the compile errors are unrelated?

On 2014/11/03 21:25:37, scottmg wrote:
> Actually, after I DEPS bison and gperf, it's not terrible looking. ~10-20
> linux-ism. Maybe the right time to do this is once net is moved out,
> though I
> guess various things will rot more the longer you wait.

The net stuff is going to happen, but I don't know when.


https://codereview.chromium.org/694353003/diff/10001/build/vs_toolchain.py
File build/vs_toolchain.py (left):

https://codereview.chromium.org/694353003/diff/10001/build/vs_toolchain.py#oldcode1
build/vs_toolchain.py:1: # Copyright 2014 The Chromium Authors. All
rights reserved.
ditto for all these updates to files in build?

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py
File build/win/win_tool.py (right):

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py#newcode1
build/win/win_tool.py:1: #!/usr/bin/env python
are you going to check a copy of this to chromium as well?

https://codereview.chromium.org/694353003/

[Scott Graham]

On Mon, Nov 3, 2014 at 2:04 PM, <j...@chromium.org> wrote:

lgtm, thanks!

re the ninja binary, I'm not sure what you mean that this needs a new binary but
the compile errors are unrelated?

Sorry, that was a bit terse.

Using the ninja.exe that's in depot_tools at the moment, you'll get errors on the second and subsequent builds (like "blah.obj not mentioned in grit/xyz.d"). At ninja ToT that's fixed, but it's not in a released ninja version yet.

Those errors are unrelated to the ones linked in the CL description which are just various things that need to be ported from Linux-specific code.

So, you want this landed? It seems a bit ugly to add sqlite, but I guess there's already a lot of stuff copied from chromium/src.

[sco...@chromium.org]

https://codereview.chromium.org/694353003/diff/10001/build/vs_toolchain.py
File build/vs_toolchain.py (right):

https://codereview.chromium.org/694353003/diff/10001/build/vs_toolchain.py#newcode22
build/vs_toolchain.py:22: def NameValueListToDict(name_value_list):
This was also to avoid pulling gyp. Maybe I should just do that instead?

... hmm, actually, GYP_DEFINES doesn't make any sense in mojo so I'll
just delete that code instead.

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py
File build/win/win_tool.py (right):

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py#newcode1
build/win/win_tool.py:1: #!/usr/bin/env python

On 2014/11/03 22:04:49, jam wrote:
> are you going to check a copy of this to chromium as well?

This was copied from gyp, so no. (GN currently uses it on Mac and
Windows but didn't make its own copy.)

https://codereview.chromium.org/694353003/

[j...@chromium.org]

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py
File build/win/win_tool.py (right):

https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py#newcode1
build/win/win_tool.py:1: #!/usr/bin/env python

On 2014/11/03 22:28:34, scottmg wrote:
> On 2014/11/03 22:04:49, jam wrote:
> > are you going to check a copy of this to chromium as well?

> This was copied from gyp, so no. (GN currently uses it on Mac and
Windows but
> didn't make its own copy.)

i see. so when chrome is gn-only, then this file will be created there
as well?

https://codereview.chromium.org/694353003/

[John Abd-El-Malek]

On Mon, Nov 3, 2014 at 2:28 PM, Scott Graham <sco...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:04 PM, <j...@chromium.org> wrote:

lgtm, thanks!

re the ninja binary, I'm not sure what you mean that this needs a new binary but
the compile errors are unrelated?

Sorry, that was a bit terse.

Using the ninja.exe that's in depot_tools at the moment, you'll get errors on the second and subsequent builds (like "blah.obj not mentioned in grit/xyz.d"). At ninja ToT that's fixed, but it's not in a released ninja version yet.

ah, got it, thanks.

when will this be released?

 

Those errors are unrelated to the ones linked in the CL description which are just various things that need to be ported from Linux-specific code.

So, you want this landed?

yes please, we filed a ticket for a win builder

 

It seems a bit ugly to add sqlite, but I guess there's already a lot of stuff copied from chromium/src.

right, there are tons of dependencies already.

[Scott Graham]

On Mon, Nov 3, 2014 at 2:37 PM, John Abd-El-Malek <j...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:28 PM, Scott Graham <sco...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:04 PM, <j...@chromium.org> wrote:

lgtm, thanks!

re the ninja binary, I'm not sure what you mean that this needs a new binary but
the compile errors are unrelated?

Sorry, that was a bit terse.

Using the ninja.exe that's in depot_tools at the moment, you'll get errors on the second and subsequent builds (like "blah.obj not mentioned in grit/xyz.d"). At ninja ToT that's fixed, but it's not in a released ninja version yet.

ah, got it, thanks.

when will this be released?

it's causing some problems for non-chromium/non-mojo cmake projects, so needs to be tweaked, but next couple weeks.

 

 

Those errors are unrelated to the ones linked in the CL description which are just various things that need to be ported from Linux-specific code.

So, you want this landed?

yes please, we filed a ticket for a win builder

ok, sg.

[sco...@chromium.org]

On 2014/11/03 22:36:50, jam wrote:
> https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py
> File build/win/win_tool.py (right):


https://codereview.chromium.org/694353003/diff/10001/build/win/win_tool.py#newcode1
> build/win/win_tool.py:1: #!/usr/bin/env python

> On 2014/11/03 22:28:34, scottmg wrote:
> > On 2014/11/03 22:04:49, jam wrote:

> > > are you going to check a copy of this to chromium as well?
> >

> > This was copied from gyp, so no. (GN currently uses it on Mac and
> Windows
but
> > didn't make its own copy.)

> i see. so when chrome is gn-only, then this file will be created there as
well?

Yeah, or something similar/simpler anyway.

https://codereview.chromium.org/694353003/

[John Abd-El-Malek]

On Mon, Nov 3, 2014 at 2:42 PM, Scott Graham <sco...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:37 PM, John Abd-El-Malek <j...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:28 PM, Scott Graham <sco...@chromium.org> wrote:

On Mon, Nov 3, 2014 at 2:04 PM, <j...@chromium.org> wrote:

lgtm, thanks!

re the ninja binary, I'm not sure what you mean that this needs a new binary but
the compile errors are unrelated?

Sorry, that was a bit terse.

Using the ninja.exe that's in depot_tools at the moment, you'll get errors on the second and subsequent builds (like "blah.obj not mentioned in grit/xyz.d"). At ninja ToT that's fixed, but it's not in a released ninja version yet.

ah, got it, thanks.

when will this be released?

it's causing some problems for non-chromium/non-mojo cmake projects, so needs to be tweaked, but next couple weeks.

I thought the fix is already committed in ninja? If I'm not minunderstanding that, why do we need to wait to pull the binary to depot_tools which is just for chromium?

[Scott Graham]

We could, we just generally have tried to avoid having "un-released" ninja binaries in depot_tools (because for a while ninja/mac/windows were different revisions and it was a mess.)

But Nico controls the release schedule. :) So once the code seems safe enough to ship to chromium it can have a version number and be in depot_tools right away. I'll try to get it sorted out this week.

[sco...@chromium.org]

Committed patchset #3 (id:20001) manually as
d19529d27cdfff8a1a8d663828e5a927b08dcabf.

https://codereview.chromium.org/694353003/