Buildbots for Ozone?

[Frédéric Wang]

Hi all,

This was mentioned on https://codereview.chromium.org/2408803002/ but
I'd like to bring this discussion to ozone-dev too.

FYI, the execution of "chrome --mash" on Wayland with chromeos=1 had
been broken twice in the past months and we only fixed these regressions
recently:

https://codereview.chromium.org/2387063002/
https://codereview.chromium.org/2389053003/

As mentioned above, issue 2408803002 tries and brings support for Ozone
X11/Wayland with chromeos=0 so that will be yet another untested
configuration.

More generally, we have been fixing several build errors and failures on
various Ozone platforms in the past two months, that were unfortunately
not detected by the current build bots.

IIUC there are two build bots for Ozone with chromeos=1, but I believe
we should try considering more build configs and tests for Ozone. Can
someone please indicate which Ozone configuration is covered? What would
be the process to add more build configs? What are the tests performed
and how to add new ones?

Thanks,

--
Frédéric 王 - Federico Juan - 프레드릭 왕
igalia.com/igalia-247/igalian/item/fwang

[Kyle Charbonneau]

Hey Frédéric,

The Ozone trybot that runs are part of CQ is linux_chromium_chromeos_ozone_rel_ng. It builds with the following GN args.

dcheck_always_on = true

ffmpeg_branding = "ChromeOS"

goma_dir = "/b/c/cipd/goma"

is_component_build = false

is_debug = false

proprietary_codecs = true

symbol_level = 1

target_os = "chromeos"

use_goma = true

use_ozone = true

Since it has use_ozone=true and target_os="chromeos" it hits the following block in ozone.gni. That means CQ only builds Ozone headless, x11 and gbm. Ozone wayland isn't built as part of CQ and breaking changes to it can be committed without anyone knowing :(

Does building Ozone wayland require any additional libraries? Running Ozone wayland does but if there are no build time dependencies then it might be possible to add ozone_platform_wayland=true to linux_chromium_chromeos_ozone_rel_ng. That would ensure the Ozone wayland compile doesn't get broken at least.

All of the tests on linux_chromium_chromeos_ozone_rel_ng run with --ozone-platform=headless right now. I've done a bit of work to make it possible to run tests using Ozone X11 but it's blocked right now. I'd imagine running tests using Ozone wayland isn't feasible. It would require a trybot with a wayland environment which we don't have.

Kyle

--
You received this message because you are subscribed to the Google Groups "Ozone-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ozone-dev+unsubscribe@chromium.org.

[Frédéric Wang]

Le 12/10/2016 à 15:40, 'Kyle Charbonneau' via Ozone-Dev a écrit :

Does building Ozone wayland require any additional libraries? Running Ozone
wayland does but if there are no build time dependencies then it might be
possible to add ozone_platform_wayland=true
to linux_chromium_chromeos_ozone_rel_ng. That would ensure the Ozone
wayland compile doesn't get broken at least.

Hi Kyle,

Yes, I think the minimum we want is to have ozone_platform_wayland=true
so that at least the code is built. I think this requires installing the
wayland development files. Also by default use_wayland_egl=true and I
believe we do want to test EGL support too. So this requires installing
additional wayland EGL development files.

(BTW, an additional Ozone platform that we "unbreak" recently and that
we might want to build is caca. You need ozone_platform_caca = true and
also the caca development files.)

All of the tests on linux_chromium_chromeos_ozone_rel_ng run with
--ozone-platform=headless right now. I've done a bit of work to make it
possible to run tests using Ozone X11 but it's blocked right now. I'd
imagine running tests using Ozone wayland isn't feasible. It would require
a trybot with a wayland environment which we don't have.

I see. So obviously this will require more runtime libraries. Also you
can also use weston (or even weston-launch) to run a Wayland
environment, but I guess this will be a bit more config work on the
buildbot. Probably, we should try and see the current status of existing
Wayland tests before and come back to this later.

Thanks,

[Robert Kroeger]

[+dpranke -- fyi and a question inline below.]

dpranke@: I'll drop by and explain my long term hope here if you're around.

On Wednesday, October 12, 2016, Frédéric Wang <fw...@igalia.com> wrote:

Le 12/10/2016 à 15:40, 'Kyle Charbonneau' via Ozone-Dev a écrit :

Does building Ozone wayland require any additional libraries? Running Ozone
wayland does but if there are no build time dependencies then it might be
possible to add ozone_platform_wayland=true
to linux_chromium_chromeos_ozone_rel_ng. That would ensure the Ozone
wayland compile doesn't get broken at least.

Hi Kyle,

Yes, I think the minimum we want is to have ozone_platform_wayland=true
so that at least the code is built. I think this requires installing the
wayland development files. 

Is there enough of Wayland in third_party now? If not, it could be added as separate CL.

 

Also by default use_wayland_egl=true and I
believe we do want to test EGL support too. So this requires installing
additional wayland EGL development files.

Same comment as above. :-)

 

(BTW, an additional Ozone platform that we "unbreak" recently and that
we might want to build is caca. You need ozone_platform_caca = true and
also the caca development files.)

In the interests of completeness, we should do this or (reluctantly) remove caca platform.

 

All of the tests on linux_chromium_chromeos_ozone_rel_ng run with
--ozone-platform=headless right now. I've done a bit of work to make it
possible to run tests using Ozone X11 but it's blocked right now. I'd
imagine running tests using Ozone wayland isn't feasible. It would require
a trybot with a wayland environment which we don't have.

I see. So obviously this will require more runtime libraries. Also you
can also use weston (or even weston-launch) to run a Wayland
environment, but I guess this will be a bit more config work on the
buildbot. 

Probably a lot more.  I have no idea how hard it would be to use wayland on a buildbot. Let's start with at least compiling the wayland platform. That will go a long time to keep the platform from breaking.

dpranke@: Am I correct in believing that ChromeOS board-overrides the ozone platform setting so it's a (safe) and (relatively easy) change to build all of the ozone platforms in linux_chromium_chromeos_ozone_* with the only consequence of (hopefully) a small increase in build times?

Probably, we should try and see the current status of existing
Wayland tests before and come back to this later.

There is a long list of tasks here. I propose:

* find out if third_party/wayland is sufficient to build the ozone wayland platform

* add what's missing if it's not

* change the builder (should be a matter of just setting the extra ozone flags in ToT per above?)

* repeat for caca

Rob.

[Dirk Pranke]

I believe so, but I'd need to double-check. I'll be around in the afternoon if you want to drop by.

-- Dirk 

[Michael Spang]

The CrOS ebuild definitely sets ozone_auto_platforms=false.

Michael

[Dirk Pranke]

Yup, looks safe.

-- Dirk

[Antonio Gomes]

For the sake of completeness, I think it is worth mentioning that
ozone_platform_wayland=true only builds for me when I use
use_sysroot=false.

Basically the sysroot lacks 'wayland-egl' installed:

Package wayland-egl was not found in the pkg-config search path.
Perhaps you should add the directory containing `wayland-egl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'wayland-egl' found

> --
> You received this message because you are subscribed to the Google Groups
> "Ozone-Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ozone-dev+...@chromium.org.



--
--Antonio Gomes

[Michael Forney]

On Thu, Oct 13, 2016 at 1:32 PM, Antonio Gomes <toni...@gmail.com> wrote:

For the sake of completeness, I think it is worth mentioning that
ozone_platform_wayland=true only builds for me when I use
use_sysroot=false.

Basically the sysroot lacks 'wayland-egl' installed:

  Package wayland-egl was not found in the pkg-config search path.
  Perhaps you should add the directory containing `wayland-egl.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'wayland-egl' found

​Yes, this is one of the reasons why I made wayland-egl optional and controlled by use_wayland_egl. With use_wayland_egl=0, everything required is available in the chromium tree.

libwayland-egl is normally provided by mesa in typical linux distributions. I'm not sure how NVIDIA's wayland EGL platform works. Maybe you can still use the libwayland-egl from mesa with it?

Perhaps it is easy to build in third_party/mesa/BUILD.gn?

[Robert Kroeger]

On Thursday, October 13, 2016, Antonio Gomes <toni...@gmail.com> wrote:

For the sake of completeness, I think it is worth mentioning that
ozone_platform_wayland=true only builds for me when I use
use_sysroot=false.

Basically the sysroot lacks 'wayland-egl' installed:

Seems like the first order of business then is to get the system root extended then? Because otherwise, the rest of issues seem comparatively minor.

Rob.

[Dirk Pranke]

You should file a bug and work with thomasanderson@ to get things updated and building.

-- Dirk

> email to ozone-dev+unsubscribe@chromium.org.



--
--Antonio Gomes

[Tom Anderson]

On 10/13/2016 02:26 PM, Dirk Pranke wrote:

You should file a bug and work with thomasanderson@ to get things updated and building.

-- Dirk

On Thu, Oct 13, 2016 at 2:17 PM, Robert Kroeger <rjkr...@chromium.org> wrote:

On Thursday, October 13, 2016, Antonio Gomes <toni...@gmail.com> wrote:

For the sake of completeness, I think it is worth mentioning that
ozone_platform_wayland=true only builds for me when I use
use_sysroot=false.

Basically the sysroot lacks 'wayland-egl' installed:

Looks like the wayland-egl package doesn't exist on wheezy.  Unfortunately that means building with use_sysroot=true is blocked on updating the sysroot to jessie, which is blocked on updating the builders from Precise to Trusty.

[Tom Anderson]

On 10/13/2016 03:34 PM, Tom Anderson wrote:

On 10/13/2016 02:26 PM, Dirk Pranke wrote:

You should file a bug and work with thomasanderson@ to get things updated and building.

-- Dirk

On Thu, Oct 13, 2016 at 2:17 PM, Robert Kroeger <rjkr...@chromium.org> wrote:

On Thursday, October 13, 2016, Antonio Gomes <toni...@gmail.com> wrote:

For the sake of completeness, I think it is worth mentioning that
ozone_platform_wayland=true only builds for me when I use
use_sysroot=false.

Basically the sysroot lacks 'wayland-egl' installed:

Looks like the wayland-egl package doesn't exist on wheezy.  Unfortunately that means building with use_sysroot=true is blocked on updating the sysroot to jessie, which is blocked on updating the builders from Precise to Trusty.

Ok I take this back.  I've added the wayland dev files to the wheezy, jessie, and precise sysroots.  https://codereview.chromium.org/2415933004/

I can build with use_ozone = true, ozone_platform_wayland=true, target_os = "chromeos"

However, desktop Linux builds are giving me some linker errors:

../../chrome/browser/fullscreen_aurax11.cc:21: error: undefined reference to 'views::DesktopWindowTreeHostX11::GetAllOpenWindows()'
../../chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:82: error: undefined reference to 'BuildGtk2UI()'
../../chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:97: error: undefined reference to 'views::X11DesktopHandler::get()'
../../chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:97: error: undefined reference to 'views::X11DesktopHandler::AddObserver(views::X11DesktopHandlerObserver*)'
../../chrome/browser/ui/views/frame/browser_frame.cc:243: error: undefined reference to 'views::X11DesktopHandler::get()'
../../chrome/browser/ui/views/frame/browser_frame.cc:243: error: undefined reference to 'views::X11DesktopHandler::GetWorkspace()'
obj/chrome/browser/ui/libui.a(window_finder.o):../../chrome/browser/ui/views/tabs/window_finder.cc:vtable for WindowFinder: error: undefined reference to 'WindowFinder::GetLocalProcessWindowAtPoint(gfx::Point const&, std::set<aura::Window*, std::less<aura::Window*>, std::allocator<aura::Window*> > const&)'
../../chrome/browser/ui/views/frame/desktop_browser_frame_aura.cc:54: error: undefined reference to 'BrowserDesktopWindowTreeHost::CreateBrowserDesktopWindowTreeHost(views::internal::NativeWidgetDelegate*, views::DesktopNativeWidgetAura*, BrowserView*, BrowserFrame*)'

At least the X11DesktopHandler ones shouldn't be too bad to fix.

[Frédéric Wang]

Le 14/10/2016 à 03:42, 'Tom Anderson' via Ozone-Dev a écrit :
> Ok I take this back. I've added the wayland dev files to the wheezy,
> jessie, and precise sysroots. https://codereview.chromium.org/2415933004/
>
> I can build with use_ozone = true, ozone_platform_wayland=true,
> target_os = "chromeos"
>
> However, desktop Linux builds are giving me some linker errors:

Thank you everybody for the progress on this!

@Tom: yes, chromeos is supported but desktop Linux is not yet. For the
latter, you can currently build target "ozone" and there is a CL here to
allow at least to build target "chrome":

https://codereview.chromium.org/2408803002/

[Antonio Gomes]

Thank you, Tom. It tested it locally, and ozone x11/wayland build
worked for both chromeos (ToT) and linux (with
https://codereview.chromium.org/2408803002/).

I believe once it https://codereview.chromium.org/2408803002/ gets
approved/merged, we should consider an extra non-chromeos ozone
buildbot (build only for now).
--
--Antonio Gomes

[Robert Kroeger]

My impression is that it's generally hard to find machine resources for CQ and builders for a new combination of build flags. 

If we don't want to add a brand new builder/CQ stack, then I see two choices (but maybe I'm missing some?)

*  add ozone=true, ozone_x11=true, mus configuration to Linux Chrome and it runs in this mode when enabled by a command line flag.

*  add a new no-ash external-window mode to CrOS Chrome enabled by a sibling flag to --mash. 

It's not clear to me which of these is better. I think Antonio's current patch is better aligned with the second option. Ash would still get compiled in this case but (thanks to mustash) wouldn't necessarily need to execute. And when mustash is complete, CrOS Chrome and Linux Chrome ought to be almost entirely the same.

Rob.

 

--
--Antonio Gomes

[Dirk Pranke]

On Sun, Oct 16, 2016 at 1:52 PM, Robert Kroeger <rjkr...@chromium.org> wrote:

On Friday, October 14, 2016, Antonio Gomes <toni...@gmail.com> wrote:

On Fri, Oct 14, 2016 at 2:00 AM, Frédéric Wang <fw...@igalia.com> wrote:
> Le 14/10/2016 à 03:42, 'Tom Anderson' via Ozone-Dev a écrit :
>> Ok I take this back.  I've added the wayland dev files to the wheezy,
>> jessie, and precise sysroots. https://codereview.chromium.org/2415933004/
>>
>> I can build with use_ozone = true, ozone_platform_wayland=true,
>> target_os = "chromeos"
>>
>> However, desktop Linux builds are giving me some linker errors:
>
> Thank you everybody for the progress on this!
>
> @Tom: yes, chromeos is supported but desktop Linux is not yet. For the
> latter, you can currently build target "ozone" and there is a CL here to
> allow at least to build target "chrome":
>
> https://codereview.chromium.org/2408803002/

Thank you, Tom. It tested it locally, and ozone x11/wayland build
worked for both chromeos (ToT) and linux (with
https://codereview.chromium.org/2408803002/).

I believe once it https://codereview.chromium.org/2408803002/ gets
approved/merged, we should consider an extra non-chromeos ozone
buildbot (build only for now).

My impression is that it's generally hard to find machine resources for CQ and builders for a new combination of build flags. 

It's not hard to get resources on Linux, but whether this configuration is important enough to justify yet another

builder on the CQ is another question.

If we don't want to add a brand new builder/CQ stack, then I see two choices (but maybe I'm missing some?)

*  add ozone=true, ozone_x11=true, mus configuration to Linux Chrome and it runs in this mode when enabled by a command line flag.

*  add a new no-ash external-window mode to CrOS Chrome enabled by a sibling flag to --mash. 

It's not clear to me which of these is better. I think Antonio's current patch is better aligned with the second option. Ash would still get compiled in this case but (thanks to mustash) wouldn't necessarily need to execute. And when mustash is complete, CrOS Chrome and Linux Chrome ought to be almost entirely the same.

If I understood you correctly the other day, we want to get to a place where we do a single build on Linux and

whether to use Wayland or X11 would be a runtime flag, right? If so, it seems like the way to go would be to add

an optional trybot configuration and a FYI builder to support a different build configuration until such time as we

had the single unified build.

WDYT?

-- Dirk

 

Rob.

 

--
--Antonio Gomes

[Antonio Gomes]

On Sun, Oct 16, 2016 at 4:52 PM, Robert Kroeger <rjkr...@chromium.org> wrote:
>
>
> On Friday, October 14, 2016, Antonio Gomes <toni...@gmail.com> wrote:
>>
>> On Fri, Oct 14, 2016 at 2:00 AM, Frédéric Wang <fw...@igalia.com> wrote:
>> > Le 14/10/2016 à 03:42, 'Tom Anderson' via Ozone-Dev a écrit :
>> >> Ok I take this back. I've added the wayland dev files to the wheezy,
>> >> jessie, and precise sysroots.
>> >> https://codereview.chromium.org/2415933004/
>> >>
>> >> I can build with use_ozone = true, ozone_platform_wayland=true,
>> >> target_os = "chromeos"
>> >>

>> (...)

>
>
> My impression is that it's generally hard to find machine resources for CQ
> and builders for a new combination of build flags.

Since 'wayland-egl' was the only missing dependency on the builtin
sysroots, and it is now fixed, I think the simpler solution for now to
avoid ozone/wayland build breakages, is to add 'wayland' as part of
the list of backends built by the headless buildbot, as highlight'ed
bellow by Kyle Charbonneau:

> dcheck_always_on = true
> ffmpeg_branding = "ChromeOS"
> goma_dir = "/b/c/cipd/goma"
> is_component_build = false
> is_debug = false
> proprietary_codecs = true
> symbol_level = 1
> target_os = "chromeos"
> use_goma = true
> use_ozone = true
>
> Since it has use_ozone=true and target_os="chromeos" it hits the following
> block in ozone.gni. That means CQ only builds Ozone headless, x11 and gbm.
> Ozone wayland isn't built as part of CQ and breaking changes to it can be
> committed without anyone knowing :(

.. and I have submitted a quick patch to it in
https://codereview.chromium.org/2423833002/.

[Frédéric Wang]

Le 13/10/2016 à 23:01, 'Michael Forney' via Ozone-Dev a écrit :
> Yes, this is one of the reasons why I made wayland-egl optional and
> controlled by use_wayland_egl. With use_wayland_egl=0, everything
> required is available in the chromium tree.

Hi Michael,

Thank you for the information. I think the required packages have been
added to the sysroot in
https://chromium.googlesource.com/chromium/src/+/ec1f1a08d336158429f96c8e09d590066696b41b
so it probably does not make sense to keep this use_wayland_egl option?

I've uploaded https://chromiumcodereview.appspot.com/2431383002/

[Frédéric Wang]

Hi all,

Just to follow-up on this:

* The wayland packages were added to sysroot in
https://codereview.chromium.org/2415933004 and the use_wayland_egl flag
removed in https://codereview.chromium.org/2431383002/
* The wayland platform is enabled by default in
https://codereview.chromium.org/2423833002/

So I believe the ozone chromeos try bot now builds the Wayland platform
and runs the existing tests. This will hopefully that reduce the number
of breakages we have encountered recently. I guess it would still be
nice to have some simple test running chrome in a Wayland environment,
but I understand it might not be so easy to setup.

I don't know if some people are interested in doing the same for the
caca platform that was discussed in this thread...

[Robert Kroeger]

On Thursday, October 20, 2016, Frédéric Wang <fw...@igalia.com> wrote:

Hi all,

Just to follow-up on this:

* The wayland packages were added to sysroot in
https://codereview.chromium.org/2415933004 and the use_wayland_egl flag
removed in https://codereview.chromium.org/2431383002/
* The wayland platform is enabled by default in
https://codereview.chromium.org/2423833002/

So I believe the ozone chromeos try bot now builds the Wayland platform
and runs the existing tests. This will hopefully that reduce the number
of breakages we have encountered recently. I guess it would still be
nice to have some simple test running chrome in a Wayland environment,
but I understand it might not be so easy to setup.

w00t!

 

I don't know if some people are interested in doing the same for the
caca platform that was discussed in this thread...

caca is fun but I don't think I can justify having people spending time on it. I don't want to lose it completely either. 

Ozone was intended to permit having platforms exist out-of-tree. Any objections to me moving it to separate repo in the chromium github?

Rob.

[Robert Kroeger]

Yes. 

 

If so, it seems like the way to go would be to add

an optional trybot configuration and a FYI builder to support a different build configuration until such time as we

had the single unified build.

WDYT?

I think this would be awesome as it would probably facilitate progress in the short term. What's the best way to proceed -- file an infra bug?

Rob.

[Dirk Pranke]

Yes, and assign it to andybons@ to triage/delegate (and cc me).

(Sorry for the delay, was out for a few days).

-- Dirk

[Tom Anderson]

I wanted to provide an update on this thread.  The Ozone Linux FYI builder went green for the first time last night.

I think the next step here is to add builders to the CQ and the main waterfall.  We would also want to try rebasing/landing this CL so we at least have something working on Ozone/Wayland.  Also, just as an FYI, Fedora 25 shipped yesterday, and it uses Wayland by default.

[Frédéric Wang]

Le 09/11/2016 à 19:57, Tom Anderson a écrit :

I wanted to provide an update on this thread.  The Ozone Linux FYI builder went green <https://build.chromium.org/p/chromium.fyi/builders/Ozone%20Linux?numbuilds=100> for the first time last night.

Yeah! Thanks for your help on this!

I think the next step here is to add builders to the CQ and the main waterfall.  We would also want to try rebasing/landing this CL <https://codereview.chromium.org/2027943002/> so we at least have something working on Ozone/Wayland.  Also, just as an FYI, Fedora 25 shipped yesterday, and it uses Wayland by default.

As discussed in a previous thread, it's not clear we want to follow the approach in this CL. Anyway, most of the work is now merged to be able to run some minimal chrome under Wayland. The only bit missing is about ash resources https://codereview.chromium.org/2485673002/. With that last patch, the following should work (maybe one need do something with https://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md):

chrome --mash --ozone-platform=wayland

[Antonio Gomes]

Hi Tom.

Does not <out>/chrome --mash --ozone-platform=wayland work for you?

PS: You need to either be on a Wayland based environment (say
Gnome/Wayland) or run it within a Wayland compositor (e.g. Weston).

> --
> You received this message because you are subscribed to the Google Groups
> "Ozone-Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ozone-dev+...@chromium.org.



--
--Antonio Gomes

[Tom Anderson]

On 11/09/2016 11:19 AM, Antonio Gomes wrote:
> Hi Tom.
>
> Does not <out>/chrome --mash --ozone-platform=wayland work for you?
>
> PS: You need to either be on a Wayland based environment (say
> Gnome/Wayland) or run it within a Wayland compositor (e.g. Weston).

Thanks, I was able to get it working with your patch

[Yuly Novikov]

Thank you very much for the work done to make this happen!

I've looked at a few builds and I see this bot finishes after compilation.

Am I missing something?

Do you have plans for adding more tests to it in the future, something like linux_chromium_chromeos_ozone_rel_ng?

email to ozone-dev+unsubscribe@chromium.org.

[Tom Anderson]

The bot is compile-only for now, but we will definitely want a linux_chromium_ozone_rel_ng in the future.  Not sure exactly how far-off that is atm.