x64 and building

[Peter Mayo]

I am thinking around and investigating how to more efficiently support development on an experimental configuration concurrently with keeping the stable configuration working with any needed refactorings and embellishments.

I was wondering if any of the people involved in setting up Release_x64 and Debug_x64 support from our gyp configs would be so kind as to share their identity and any references to what was done, or what their encountered pitfalls were?

Thanks,

Peter.

[Scott Graham]

I'm not sure what your question is. Are you talking about doing some other thing than x64 in gyp?

This is probably a good starting place: https://code.google.com/p/chromium/codesearch#search/&q=_x64%20file:gyp&sq=package:chromium&type=cs

There were a variety of people that worked on it. The main things I remember:

- various difference in platform headers (warnings, blahblah)

- bot setup (output directories hardcoded to something x86-ish)

- a lot of |int|s that should be |size_t|s

- there was a tool in third_party that parses .obj files during the build, but didn't support x64 coff.

{jschuh, wolenetz, iannucci}@ might recall more pain.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

[Peter Mayo]

Yes, I am talking about something other than x64.  Imagine working on aura before aura was published & released.  Changes had to land that broke neither use_aura=1 nor use_aura=0.  Switching back and forth was annoying and time consuming.  If we had the intermediates and executables segregated into Release and Release_aura (or any other name, like Release_x64 was) then a dev could more quickly validate the state of both configurations from their local source for their local platform.

Plumbing the correct support of Release_64 through gyp, and chromium's .gyp and .gypi files is the expertise/experience I wanted to pull from before evaluating whether this is a worthwhile path.

Thanks,

Peter.

--

Peter Mayo |

Waterloo |

pete...@google.com |

519-880-3439

[Peter Mayo]

And the other guy is too.

[Antoine Labour]

On Thu, Jul 11, 2013 at 2:32 PM, Peter Mayo <pete...@google.com> wrote:

Yes, I am talking about something other than x64.  Imagine working on aura before aura was published & released.  Changes had to land that broke neither use_aura=1 nor use_aura=0.  Switching back and forth was annoying and time consuming.  If we had the intermediates and executables segregated into Release and Release_aura (or any other name, like Release_x64 was) then a dev could more quickly validate the state of both configurations from their local source for their local platform.

Plumbing the correct support of Release_64 through gyp, and chromium's .gyp and .gypi files is the expertise/experience I wanted to pull from before evaluating whether this is a worthwhile path.

Thanks,

Peter.

I do a lot of multi-config builds from the same source tree (e.g. Chrome OS vs Aura vs Linux vs Android, cross Component vs not, cross gcc vs clang vs asan, cross official vs not, etc.)

What I do personally is replace not "Release" by "Release_$config" but gyp's output_dir (out/ is the default), by simply setting the output_dir in GYP_GENERATOR_FLAGS, that varies per-platform. It's a lot more scalable (though some scripts, e.g. landmines.py or android tools don't know about it and break).

Antoine

[Nico Weber]

On Thu, Jul 11, 2013 at 2:38 PM, Antoine Labour <pi...@google.com> wrote:

On Thu, Jul 11, 2013 at 2:32 PM, Peter Mayo <pete...@google.com> wrote:

Yes, I am talking about something other than x64.  Imagine working on aura before aura was published & released.  Changes had to land that broke neither use_aura=1 nor use_aura=0.  Switching back and forth was annoying and time consuming.  If we had the intermediates and executables segregated into Release and Release_aura (or any other name, like Release_x64 was) then a dev could more quickly validate the state of both configurations from their local source for their local platform.

Plumbing the correct support of Release_64 through gyp, and chromium's .gyp and .gypi files is the expertise/experience I wanted to pull from before evaluating whether this is a worthwhile path.

Thanks,

Peter.

I do a lot of multi-config builds from the same source tree (e.g. Chrome OS vs Aura vs Linux vs Android, cross Component vs not, cross gcc vs clang vs asan, cross official vs not, etc.)

What I do personally is replace not "Release" by "Release_$config" but gyp's output_dir (out/ is the default), by simply setting the output_dir in GYP_GENERATOR_FLAGS, that varies per-platform. It's a lot more scalable (though some scripts, e.g. landmines.py or android tools don't know about it and break).

+1, that's the way to go.

GYP_GENERATORS=ninja GYP_DEFINES=aura=1 build/gyp_chromium -Goutput_dir=out_aura

[Thiago Farina]

On Thu, Jul 11, 2013 at 6:59 PM, Nico Weber <tha...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 2:38 PM, Antoine Labour <pi...@google.com> wrote:

On Thu, Jul 11, 2013 at 2:32 PM, Peter Mayo <pete...@google.com> wrote:

Yes, I am talking about something other than x64.  Imagine working on aura before aura was published & released.  Changes had to land that broke neither use_aura=1 nor use_aura=0.  Switching back and forth was annoying and time consuming.  If we had the intermediates and executables segregated into Release and Release_aura (or any other name, like Release_x64 was) then a dev could more quickly validate the state of both configurations from their local source for their local platform.

Plumbing the correct support of Release_64 through gyp, and chromium's .gyp and .gypi files is the expertise/experience I wanted to pull from before evaluating whether this is a worthwhile path.

Thanks,

Peter.

I do a lot of multi-config builds from the same source tree (e.g. Chrome OS vs Aura vs Linux vs Android, cross Component vs not, cross gcc vs clang vs asan, cross official vs not, etc.)

What I do personally is replace not "Release" by "Release_$config" but gyp's output_dir (out/ is the default), by simply setting the output_dir in GYP_GENERATOR_FLAGS, that varies per-platform. It's a lot more scalable (though some scripts, e.g. landmines.py or android tools don't know about it and break).

+1, that's the way to go.

GYP_GENERATORS=ninja GYP_DEFINES=aura=1 build/gyp_chromium -Goutput_dir=out_aura

 

Hooray! Ah, that is something I was missing and didn't know how to do. I have chromeos set in my ~/.gyp/include.gypi, so the back and forth to plain Linux (gtk) was always a pain. So if I can split the output per configuration that may save a lot of time when recompiling.

-- 

Thiago

[Nico Weber]

It's documented on the ninja page: https://code.google.com/p/chromium/wiki/NinjaBuild#Custom_build_configs

 

-- 

Thiago

[Thiago Farina]

On Thu, Jul 11, 2013 at 7:14 PM, Nico Weber <tha...@chromium.org> wrote:
> It's documented on the ninja page:
> https://code.google.com/p/chromium/wiki/NinjaBuild#Custom_build_configs
>

Yep, I just found
https://github.com/natduca/home/blob/master/bin/chromeos.env :)

--
Thiago

[Slavomir Kaslev]

What are the right GYP_DEFINES to build x64 Chrome with Aura on Windows today? I tried GYP_DEFINES="component=shared_library use_aura=1 enable_hidpi=1 host_arch=x64" but hit compile errors for inline asm in NaCl.

--
Thiago

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
    http://groups.google.com/a/chromium.org/group/chromium-dev

--

Slavomir Kaslev | Software Engineer | ska...@google.com | 562 217 8497

[Scott Graham]

I'm not sure if anyone's spent much time on x64+aura yet so I'm not sure how well it works, but it'd be target_arch=x64 and make sure to ninja -C out\Release_x64 chrome (rather than just out\Release).

[Justin Schuh]

To answer the original question, the (Release|Debug)_x64 thing is an ugly hack that we're not proud of. It was the least evil option, however, in light of an extremely large number of configuration quirks currently in place to support a pile of x64 glue for Win32 Chrome on 64-bit Windows. I would strongly discourage emulating the approach, and rather suggest that you simply redirect your out/ directory (as suggested by others).

-j

[Hazir Krasniqi]

Ju falenderoji per ket shkrim tendin,pra nese mundeni te me ndihmoni atehere do te kontaktohemi dhe me shkruani detalisht, se une kam njohuri te madhe me Internet, edhe njehere ju falenderoji.

Am Donnerstag, 11. Juli 2013 22:57:20 UTC+2 schrieb Peter Mayo:

Unë jam duke menduar rreth dhe duke hetuar se si për të mbështetur zhvillimin në mënyrë më efikase në një konfiguracion eksperimental njëkohësisht me mbajtjen e konfigurimit të qëndrueshme pune me ndonjë refactorings nevojshme dhe embellishments.

Unë u pyesin nëse ndonjë prej njerëzve të përfshirë në ngritjen Release_x64 dhe Debug_x64 mbështetje nga configs batakçi tona do të jetë kështu lloj si të ndajnë identitetin e tyre dhe të gjitha referencat në atë që është bërë, apo çfarë kurthesh e tyre të hasura ishin?

Thanks,

Peter.

[Peter Mayo]

Are the lessons learned, the initial design decisions or the motivations captured in a doc/bug/initial CL ?

[Peter Mayo]

On Thu, Jul 11, 2013 at 5:59 PM, Nico Weber <tha...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 2:38 PM, Antoine Labour <pi...@google.com> wrote:

+1 - This works great for what I want to do.

I will suggest a follow on though, and that is to add custom hooks in your .gclient so that these parallel directories are all maintained for you when you sync/runhooks.

Then running other configurations you use/want can be done by just referencing the one(s) you want by the name you set up, rather than using invisible state in your system/environment.  In particular, having the normal/natural/ tree breaking state close at hand and easily regression tested would seem like a win.

ninja -C src/out_foo chrome my_unittests  ...

[Zachary Turner]

Unfortunately this still doesn't work very well when using msvs or msvs-ninja, because the msvc project files are output directly into the source tree rather than into the build output directory.  I asked about fixing this once before and it seemed like there was some support, but I haven't had the time to look into it yet.

Because of that, I actually had to abandon the multiple output trees and go back to doing a complete re-build when I want to switch between aura and non-aura.  Time consuming, but seems like the best option.

[Shezan Baig]

Sorry, I'm just now catching up on chromium-dev and I saw this message from a few days ago.

I believe with msvs-ninja, something like this might work (I haven't tried it, so please pardon any typos :) ), but start by creating a gypi file with all the configurations you want:

msvs_configs.gypi:
{
    "target_defaults": {
        "configurations": {
            "no_aura_debug": {
                "msvs_external_builder_out_dir": "<DEPTH)/out_no_aura/Debug",
            },
            "no_aura_release": {
                "msvs_external_builder_out_dir": "<DEPTH)/out_no_aura/Release",
            },
            "aura_debug": {
                "msvs_external_builder_out_dir": "<DEPTH)/out_aura/Debug",
            },
            "aura_release": {
                "msvs_external_builder_out_dir": "<DEPTH)/out_aura/Release",
            },
         },
    },
}

Then, run gyp_chromium like:
$ GYP_GENERATORS=msvs-ninja python gyp_chromium -I msvs_configs.gypi
$ GYP_GENERATORS=ninja python gyp_chromium -Duse_aura=0 -Goutput_dir=out_non_aura
$ GYP_GENERATORS=ninja python gyp_chromium -Duse_aura=1 -Goutput_dir=out_aura

Now, when you open your msvs solution, you should see configurations for "no_aura_debug", "no_aura_release", "aura_debug", "aura_release".  Depending on which configuration you select from the IDE when you build, ninja will run in the corresponding output directory.

Note that the msvs project files are still in the source tree, but that doesn't really matter because the build output directory is redirected to ninja's output directory.  So you don't need to do complete rebuilds when switching between aura and non-aura.

Hope this helps,
-shez-

[Zachary Turner]

Yes, but the project files contain various #defines and other compiler settings that are configuration specific.  You cannot use an aura project file to build non-aura for example, because if your project file was generated with use_aura=1, then USE_AURA will be #define'd on the C++ side.  Even if you build using ninja and use msvs just as a source browser, there are still problems because certain source files aren't even included in the projects, etc.  The only way to fix this, as far as I can tell, is for the project files to go in the output directories.

[Zachary Turner]

Sorry I misunderstood what you wrote at first, but I think the latter problem I described still exists with this method.  namely, that source browsing will be a little off since the project files will be for the last configuration that was generated (aura in this case).  So Intellisense (or Visual Assist X) will always think that USE_AURA is defined, and various source files will not be included in the solution.

Can you specify use_aura=1 under each specific configuration, next to where you've specified msvs_external_builder_out_dir?  Something like that would probably fix the problme I'm describing.

[Shezan Baig]

No, you're right.  If use_aura causes different files/projects to be built, then you would need to physically have different vcxproj/sln files, which means, yeah they will need to be in separate output directories.

I don't have access to a development machine this weekend, so I can't tell what (if any) support there is in the msvs generator for the "-Goutput_dir" option.  But I don't imagine it would be difficult to add this support.  I can probably take a look in the next week and upload a CL for you to try.  I think this would be useful in some of my workflows as well.

Best,
-shez-

[Scott Graham]

You could hack it a bit by adding 'defines': ['USE_AURA=1'] or similar in the config blocks. You'd still have the wrong file list though.

[Zachary Turner]

Either way, it got me to thinking.  It actually would be really cool if you could do it with a single solution.  MSBuild obviously supports it, it's just a matter of making the generator support it.  I mean in MSBuild you can have per-configuration excluded files, #defines, etc.  This is probably a much bigger task, but maybe I'll think about it down the line.  It would be really convenient to have all the configurations you ever use in a single .sln file.

[Shezan Baig]

That does sound convenient, but definitely would be a much bigger task.  Since the list of files and projects is determined by gyp before the msvs generator is invoked, I think it will require either a change to the way gyp itself is processed (which is outside the scope of the msvs generator), or make the msvs generator intelligent enough to merge different configurations together in multiple gyp runs (I'm not sure how trivial this would be).

In the meantime, I think something like this doesn't seem too bad, and gives you almost what you want (no need to rebuild when switching, and also correct intellisense):

msvs.gypi:
{
    "target_defaults": {
        "msvs_external_builder_out_dir": "<(DEPTH)/<(output_dir)/$(CONFIGURATION_NAME)",
    },
}

# Generate the ninja files  (do this only once)

$ GYP_GENERATORS=ninja python gyp_chromium -Duse_aura=1 -Goutput_dir=out_aura
$ GYP_GENERATORS=ninja python gyp_chromium -Duse_aura=0 -Goutput_dir=out_non_aura

Then, say you're working on aura, generate msvs projects like this:
$ GYP_GENERATORS=msvs-ninja python gyp_chromium -Duse_aura=1 -Doutput_dir=out_aura -I msvs.gypi

Then, when you want to work on non-aura, regenerate your msvs projects like this:
$ GYP_GENERATORS=msvs-ninja python gyp_chromium -Duse_aura=0 -Doutput_dir=out_non_aura -I msvs.gypi

You'll need to regenerate the msvs projects when switching, but this seems better than cleaning your build directory each time you switch, and you will get correct intellisense based on how you last invoked the msvs-ninja generator.