Mus + OzonePlatform::InitializeForGPU

[Antonio Gomes]

(I was going to ask on IRC #ozone-wayland, but decided to mail
ozone-dev for the record)

Hey all/rjkroege!

I would like to confirm what is the envisioned way to have
service/ui's connector instance available to
OzonePlatform::InitiliazeForGPU .

For instance, I see that for OzonePlatform::InitializeForUI, it is
called directly from Service::OnStart (services/ui/service.cc), with
the proper connector instance as parameter.

In case of OzonePlatform::InitializeForGPU, though, it is called indirectly with

ui::Service::OnStart
ui::ws::WindowServer::WindowServer
ui::ws::GpuServiceProxy::GpuServiceProxy
ui::GpuMain::GpuMain
gpu::GpuInit::InitializeAndStartSandbox
gl::init::InitializeGLOneOff
OzonePlatform::InitiliazeForGPU

Fred Wang (in cc) experimented with a CL where the "connector"
instance is passed through the code path, as parameter:
https://codereview.chromium.org/2342003003/

What do you guys think?

--
--Antonio Gomes

[Robert Kroeger]

Hi,

On Friday, September 16, 2016, Antonio Gomes <toni...@gmail.com> wrote:

(I was going to ask on IRC #ozone-wayland, but decided to mail
ozone-dev for the record)

Hey all/rjkroege!

I would like to confirm what is the envisioned way to have
service/ui's connector instance available to
OzonePlatform::InitiliazeForGPU .

I'm still sorting this out but the high-level details are something like this:

service/ui will be two processes (it isn't because we have been figuring out the details of the division):

1. the window server (https://cs.chromium.org/chromium/src/services/ui/main.cc)

2. the gpu server (which doesn't have a mainline yet.) But when it does, it will look something like so:

MojoResult ServiceMain(MojoHandle service_request_handle) {

  shell::ServiceRunner runner(new ui::GpuService);

  runner.set_message_loop_type(base::MessageLoop::TYPE_UI);

  return runner.Run(service_request_handle);

}

The ui::GpuService's shell::Service implementation will have a ui::GpuService::OnStart and that method will call OzonePlatform::InitiliazeForGPU like the window server (1) calls it now (https://cs.chromium.org/chromium/src/services/ui/service.cc?rcl=0&l=159)

At that point, Ozone will have a mojo connector() in both ws and gpu processes and they can establish connections to each other as mediated by the manifest files.

 

For instance, I see  that for OzonePlatform::InitializeForUI, it is
called directly from Service::OnStart (services/ui/service.cc), with
the proper connector instance as parameter.

In case of OzonePlatform::InitializeForGPU, though, it is called indirectly with

ui::Service::OnStart
ui::ws::WindowServer::WindowServer
ui::ws::GpuServiceProxy::GpuServiceProxy
ui::GpuMain::GpuMain
gpu::GpuInit::InitializeAndStartSandbox
gl::init::InitializeGLOneOff
OzonePlatform::InitiliazeForGPU

Yeah. For a variety of reasons, I want to move that to where I talk about it above. And currently, it's not setup correctly at all in standard Chrome. (Standard Chrome doesn't use mojo in ozone so this doesn't matter yet.)

 

Fred Wang (in cc) experimented with a CL where the "connector"
instance is passed through the code path, as parameter:
https://codereview.chromium.org/2342003003/

What do you guys think?

I was still thinking about it. I don't think we know how we want it to be in a non-mus context. The mus approach (at least at some level of abstraction -- many details are undecided) is the broadly agreed to plan of record so my preference for wayland support would be to head towards where we're going and sort out how wayland can integrate with mus. Which probably requires some design discussion. But doing this does have one large advantage: mus (ws) mostly solves how to structure support for (I'll invent terminology here) internal-window vs external-window problem. [1]. I presume that our eventual goal is Chrome built ozone=true, cros=false to support both X11 and wayland. But it will take a while.

[1] internal-window is a name that I just made up for how ChromeOS-Chrome manages windows: there is a single acceleratedWidget for a single physical display and all of Chrome's windows (popups, browsers, etc.) are (aura) windows inside of this single display. external-window is the converse: the way that {mac,linux,windows}-Chrome makes a new platform window (and hence acceleratedWidget) for each Chrome window.

Rob.

 

--
--Antonio Gomes

[Antonio Gomes]

Hi,

(..)

>>
>> I would like to confirm what is the envisioned way to have
>> service/ui's connector instance available to
>> OzonePlatform::InitiliazeForGPU .
>
> I'm still sorting this out but the high-level details are something like
> this:
>
> service/ui will be two processes (it isn't because we have been figuring out
> the details of the division):
>
> 1. the window server
> (https://cs.chromium.org/chromium/src/services/ui/main.cc)
>
> 2. the gpu server (which doesn't have a mainline yet.) But when it does, it
> will look something like so:
>
> MojoResult ServiceMain(MojoHandle service_request_handle) {
> shell::ServiceRunner runner(new ui::GpuService);
> runner.set_message_loop_type(base::MessageLoop::TYPE_UI);
> return runner.Run(service_request_handle);
> }
>
> The ui::GpuService's shell::Service implementation will have a
> ui::GpuService::OnStart and that method will call
> OzonePlatform::InitiliazeForGPU like the window server (1) calls it now
> (https://cs.chromium.org/chromium/src/services/ui/service.cc?rcl=0&l=159)
>
> At that point, Ozone will have a mojo connector() in both ws and gpu
> processes and they can establish connections to each other as mediated by
> the manifest files.

This is a good point.

If I understand things correctly, from a Mojo perspective, it would
not matter whether the Mus-Ws and Mus-GPU code are on different
*processes* or on different *threads* ; the communication through Mojo
happens transparently to it.

So, in order to prepare the GPU side in Ozone/Wayland for the future,
I was thinking if it would make things any easier if Mus-WS and
Must-GPU code run on different threads (my reading of the code is that
they are on the same process, same thread?).
For instance, DRM-GBM Ozone backend is able to "clone" Mus-WS's
connector and establish a communication channel just fine.

>> For instance, I see that for OzonePlatform::InitializeForUI, it is
>> called directly from Service::OnStart (services/ui/service.cc), with
>> the proper connector instance as parameter.
>>
>> In case of OzonePlatform::InitializeForGPU, though, it is called
>> indirectly with
>>
>> ui::Service::OnStart
>> ui::ws::WindowServer::WindowServer
>> ui::ws::GpuServiceProxy::GpuServiceProxy
>> ui::GpuMain::GpuMain
>> gpu::GpuInit::InitializeAndStartSandbox
>> gl::init::InitializeGLOneOff
>> OzonePlatform::InitiliazeForGPU
>
> Yeah. For a variety of reasons, I want to move that to where I talk about it
> above. And currently, it's not setup correctly at all in standard Chrome.
> (Standard Chrome doesn't use mojo in ozone so this doesn't matter yet.)

So Fred (CC'ed) have been experimenting with Mojo in Ozone/X11, on
ChromeOS builds of Chrome, launched --mash parameter, and make it
Mojo-capable.
This configuration was chosen because it is what the most functional
Ozone backend (X11) at the moment.

>>
>> Fred Wang (in cc) experimented with a CL where the "connector"
>> instance is passed through the code path, as parameter:
>> https://codereview.chromium.org/2342003003/
>>
>> What do you guys think?
>
> I was still thinking about it. I don't think we know how we want it to be in
> a non-mus context. The mus approach (at least at some level of abstraction
> -- many details are undecided) is the broadly agreed to plan of record so my
> preference for wayland support would be to head towards where we're going
> and sort out how wayland can integrate with mus. Which probably requires
> some design discussion. But doing this does have one large advantage: mus
> (ws) mostly solves how to structure support for (I'll invent terminology
> here) internal-window vs external-window problem. [1]. I presume that our
> eventual goal is Chrome built ozone=true, cros=false to support both X11 and
> wayland. But it will take a while.
>
>
> [1] internal-window is a name that I just made up for how ChromeOS-Chrome
> manages windows: there is a single acceleratedWidget for a single physical
> display and all of Chrome's windows (popups, browsers, etc.) are (aura)
> windows inside of this single display. external-window is the converse: the
> way that {mac,linux,windows}-Chrome makes a new platform window (and hence
> acceleratedWidget) for each Chrome window.

Understood.

In Fred's CL above, the main goal was to make a Connector instance
available to Ozone/Wayland GPU side. As I said before it might still
be helpful thing to do, even before we have the WS / GPU process
separation.
In other words, if we run them on different thread, making using of
mojo to communicate them would be helpful, and should just work when
the split up happens.

--
--Antonio Gomes

[Frédéric Wang]

Hi Robert,

Thank you for your message and for your reviews!

As I understand the situation, on the one hand there is a clear will
from Chromium developers to migrate to:

i) Mojo as a new IPC API.
ii) Mus for window management.

As you said, there is also a good high-level picture of how to handle
these two points for Ozone, but that's unfortunately still
work-in-progress. Do you have an estimate of how long it would take? Or
even better, how we can contribute on the effort?

On the other hand, people are willing to upstream Intel's Ozone backend
for Wayland but the current implementation does not use Mojo
communication and only works for the non-mus path. Hence the question is
whether old IPC and/or the non-mus path should be considered in the
short term?

Anyway, I agree with you regarding how the initialization parameters
will be passed for the mus case. Here is my analysis for the non-mus case:

1) One process is launched, runs BrowserMain and calls
BrowserMainRunnerImpl::Initialize which in turn calls:
1.1) BrowserMainLoop::InitializeToolkit => aura::Env::CreateInstance
=> aura::Env::Init => ui::OzonePlatform::InitializeForUI
1.2) BrowserMainLoop::CreateStartupTasks =>
BrowserMainLoop::BrowserThreadsStarted =>
BrowserMainLoop::InitializeMojo which creates a MojoShellContext
connecting to the service manager and provides a Mojo connector for the
io thread.

2) Another process is launched, runs GpuMain and creates:
2.1) A gpu::GpuInit instance and calls
gpu::GpuInit::InitializeAndStartSandbox =>
gl::init::InitializeGLOneOff() => gl::init::GetAllowedGLImplementations
=> OzonePlatform::InitializeForGPU
2.2) A GpuChildThread instance which at some point will execute
GpuChildThread::OnInitialize to create a GpuServiceFactory.

In 2.2), we could modify GpuServiceFactory::RegisterServices to insert a
factory for a new "mojo::gpu" service (as it is done for "mojo::media"
under some build flag) and then we would have a Mojo connector
associated to it.

The main problem here is that the Mojo connectors are obtained in 2.2)
and 1.2) *after* the respective calls to InitializeForUI and
InitializeForGPU. So if we want to use Mojo communications in the
non-mus path we would have either to refactor the initialization process
(which seems hard to do) or introduce some OzonePlatform functions to
pass the mojo connector later (e.g. something like "ConnectUI" and
"ConnectGPU"). What do you think?

Frédéric

[Frédéric Wang]

Le 21/09/2016 à 16:42, Frédéric Wang a écrit :
> In 2.2), we could modify GpuServiceFactory::RegisterServices to insert a
> factory for a new "mojo::gpu" service (as it is done for "mojo::media"
> under some build flag) and then we would have a Mojo connector
> associated to it.
>

> introduce some OzonePlatform functions to pass the mojo connector later (e.g. something like "ConnectUI" and "ConnectGPU").

Just for the record, I tried this today and was indeed able to
experiment some basic mojo communication in Ozone/Wayland for the
non-mus path. However, for 2.2) the mojo connector can actually be
obtained by GpuChildThread::GetMojoShellConnection()->GetConnector() so
it is somewhat simpler than what I thought.

[Robert Kroeger]

comments inline

yes. This is one of the points of Mojo: it makes different process arrangements possible. 

 

So, in order to prepare the GPU side in Ozone/Wayland for the future,
I was thinking if it would make things any easier if Mus-WS and
Must-GPU code run on different threads (my reading of the code is that
they are on the same process, same thread?).

It would appear that it is currently like this. But this is a temporary state that does not reflect the desired intent. Soon, mus-gpu will be a separate process and the architecture will be more as it should be.

 

For instance, DRM-GBM Ozone backend is able to "clone" Mus-WS's
connector and establish a communication channel just fine.

The code for this in ozone ought to do the "right thing" whether or not there mus-ws and mus-gpu are co-located in one process or two.

 

>> For instance, I see  that for OzonePlatform::InitializeForUI, it is
>> called directly from Service::OnStart (services/ui/service.cc), with
>> the proper connector instance as parameter.
>>
>> In case of OzonePlatform::InitializeForGPU, though, it is called
>> indirectly with

>>

>> ui::Service::OnStart
>> ui::ws::WindowServer::WindowServer
>> ui::ws::GpuServiceProxy::GpuServiceProxy 

>> ui::GpuMain::GpuMain
>> gpu::GpuInit::InitializeAndStartSandbox
>> gl::init::InitializeGLOneOff
>> OzonePlatform::InitiliazeForGPU
>
> Yeah. For a variety of reasons, I want to move that to where I talk about it
> above. And currently, it's not setup correctly at all in standard Chrome.
> (Standard Chrome doesn't use mojo in ozone so this doesn't matter yet.)

So Fred (CC'ed) have been experimenting with Mojo in Ozone/X11, on
ChromeOS builds of Chrome, launched --mash parameter, and make it

Mojo-capable.This configuration was chosen because it is what the most functional

Ozone backend (X11) at the moment.

I'm not clear on what you mean by "most functional".  We built ozone/x11 primarily for testing. Ozone/Drm is the shipping configuration of ozone for CrOS yes?

 

>>
>> Fred Wang (in cc) experimented with a CL where the "connector"
>> instance is passed through the code path, as parameter:
>> https://codereview.chromium.org/2342003003/
>>
>> What do you guys think?

afaik: Fred has set this aside for the moment?

 

>
> I was still thinking about it. I don't think we know how we want it to be in
> a non-mus context. The mus approach (at least at some level of abstraction
> -- many details are undecided) is the broadly agreed to plan of record so my
> preference for wayland support would be to head towards where we're going
> and sort out how wayland can integrate with mus. Which probably requires
> some design discussion. But doing this does have one large advantage: mus
> (ws) mostly solves how to structure support for (I'll invent terminology
> here) internal-window vs external-window problem. [1]. I presume that our
> eventual goal is Chrome built ozone=true, cros=false to support both X11 and
> wayland. But it will take a while.
>
>
> [1] internal-window is a name that I just made up for how ChromeOS-Chrome
> manages windows: there is a single acceleratedWidget for a single physical
> display and all of Chrome's windows (popups, browsers, etc.) are (aura)
> windows inside of this single display. external-window is the converse: the
> way that {mac,linux,windows}-Chrome makes a new platform window (and hence
> acceleratedWidget) for each Chrome window.

Understood.

In Fred's CL above, the main goal was to make a Connector instance
available to Ozone/Wayland GPU side. As I said before it might still
be helpful thing to do, even before we have the WS / GPU process
separation.
In other words, if we run them on different thread, making using of
mojo to communicate them would be helpful, and should just work when
the split up happens.

fwiw: the process split should happen by mid October?

Is wayland ready for this eventually where mus-ws's ozone implementation gets events, display interrogation and cursor control while mus-gpu's ozone implementation provides gl contexts and manages gpu memory buffers.

Rob.

 

--
--Antonio Gomes

[Robert Kroeger]

Hi Frédéric,

On Wednesday, September 21, 2016, Frédéric Wang <fw...@igalia.com> wrote:

Hi Robert,

Thank you for your message and for your reviews!

No problem. If you think I've missed you (i.e. more than 24 hour latency), you should pester me on irc.

 

As I understand the situation, on the one hand there is a clear will
from Chromium developers to migrate to:

i) Mojo as a new IPC API.

yes.

 

ii) Mus for window management.

yes. But mus is more than this. I'm suppose to be updating the feature summary this week so will share.

 

As you said, there is also a good high-level picture of how to handle
these two points for Ozone, but that's unfortunately still
work-in-progress. Do you have an estimate of how long it would take?

I'm probably not suppose to commit to dates in a public list. :-) The process split should happen in a small number of weeks. The larger graphics changes will happen incrementally over several quarters.

 

Or
even better, how we can contribute on the effort?

How about the following (these can happen in parallel)

1. Make Chromium ozone=1, cros=0 on ozone-x11 actually build and work? (Maybe this is what Antonio meant in the previous email where I was confused?) Anyway: the goal would be to have this kind of Chromium work correctly in external-window mode on x11.

2. Convert the wayland platform to mojo IPC. (Note that upstreaming this code will require security review.)

At the end of this, we would have a single Chromium build that would run on Linux with either an X11 or wayland backend and we might seriously discuss making this be the supported Chrome Linux configuration.

As far as helping with the mus process split and ancillary work: it's maybe not very easy to contribute to that as it seems to require a lot of communication and shared knowledge. There will be a bunch of bugs to pick up over the next few weeks though as we (finally) get the bugs organized.

 

On the other hand, people are willing to upstream Intel's Ozone backend
for Wayland but the current implementation does not use Mojo
communication and only works for the non-mus path. Hence the question is
whether old IPC and/or the non-mus path should be considered in the
short term?

I'd say no. Without "external-window" mode, wayland Chromium would seem to be a demo but not something that we could imagine shipping for Chrome. 

 

Anyway, I agree with you regarding how the initialization parameters
will be passed for the mus case. Here is my analysis for the non-mus case:

1) One process is launched, runs BrowserMain and calls
BrowserMainRunnerImpl::Initialize which in turn calls:
  1.1) BrowserMainLoop::InitializeToolkit => aura::Env::CreateInstance
=> aura::Env::Init => ui::OzonePlatform::InitializeForUI
  1.2) BrowserMainLoop::CreateStartupTasks =>
BrowserMainLoop::BrowserThreadsStarted =>
BrowserMainLoop::InitializeMojo which creates a MojoShellContext
connecting to the service manager and provides a Mojo connector for the
io thread.

2) Another process is launched, runs GpuMain and creates:
  2.1) A gpu::GpuInit instance and calls
gpu::GpuInit::InitializeAndStartSandbox =>
gl::init::InitializeGLOneOff() => gl::init::GetAllowedGLImplementations
=> OzonePlatform::InitializeForGPU
  2.2) A GpuChildThread instance which at some point will execute
GpuChildThread::OnInitialize to create a GpuServiceFactory.

yes. 

 

In 2.2), we could modify GpuServiceFactory::RegisterServices to insert a
factory for a new "mojo::gpu" service (as it is done for "mojo::media"
under some build flag) and then we would have a Mojo connector
associated to it.

Hm. Maybe not. I'll have to ask others about this.

 

The main problem here is that the Mojo connectors are obtained in 2.2)
and 1.2) *after* the respective calls to InitializeForUI and
InitializeForGPU. So if we want to use Mojo communications in the
non-mus path we would have either to refactor the initialization process
(which seems hard to do) or introduce some OzonePlatform functions to
pass the mojo connector later (e.g. something like "ConnectUI" and
"ConnectGPU"). What do you think?

I don't like either of these choices very much. I vote for presuming mus architecture for Wayland Chromium.

As for how we would use mojo in the ozone with the existing Chrome architecture, someone (me?) on Chrome team will probably be fixing this in the next month. 

Rob.

 

Frédéric

[Frédéric Wang]

Hi Robert,

Just an update on this, as maybe we were not really clear on our status
& experiments:

1) Last month, we were able to run "chrome --ozone-platform=wayland" for
Ozone/Wayland/ChromeOS using the following configuration:

use_ozone = true
ozone_platform_wayland = true
target_os="chromeos"

and a small patch to
content/browser/compositor/gpu_process_transport_factory.cc to force
software fallback to happen. This is not using mus and we have two
separate processes for UI and GPU. As said above, it's hard to pass the
Mojo connectors to Initalize* functions. However, we were able to make a
quick hack to pass
GpuChildThread::GetMojoShellConnection()->GetConnector() after the init
calls and verified it makes communication from GPU to UI possible.

2) Antonio also wrote some patches some times ago (the first one in the
series being https://codereview.chromium.org/2024953007/) and was able
to run chrome for Ozone/Wayland/Linux using this configuration:

use_ozone = true
ozone_platform_wayland = true

(i.e. ChromeOS=0 here). However, he did not rely on mus and instead
follow Intel's approach (i.e. code in ui/views/widget/desktop_aura etc)
that you indicated is not the long term approach.

3) We were able to run "chrome --mash --ozone-platform=x11" on
Ozone/X11/ChromeOS with the following configuration:

use_ozone = true
ozone_platform_x11 = true
ozone_platform_wayland = true
enable_package_mash_services = true
target_os="chromeos"

This works without any additional patches. However, until recently we
were not able to run "chrome --mash --ozone-platform=wayland". This
seems to be due to some regressions: see
https://codereview.chromium.org/2387063002/ and
https://codereview.chromium.org/2389053003/. With these two patches, we
are able to run chrome for Ozone+Wayland+ChromeOS+mus again. For now,
Initalize* calls happen in the same process but (after
https://codereview.chromium.org/2366623002/) in different threads. Also
a mojo connector is passed to InitializeUI, so we are now able to do
Mojo communication between UI & GPU processes in Ozone/Wayland. This
means that we can now start experimenting with the mojo-fication of the
Wayland backend.

4) Currently, we probably don't have the prerequisites mentioned in
https://groups.google.com/a/chromium.org/forum/#!searchin/ozone-dev/drm/ozone-dev/PrqBxBmSG3s/vhMqWur5BgAJ
to run the DRM backend, and as said in 3) "chrome --mash
--ozone-platform=wayland" does not work on ToT so X11 was the "most
functional" backend for us. Still, we had build errors or runtime
failures when trying to run "chrome --mash" with non-chrome targets. So
this is another point we have to work on.

Frédéric

Le 25/09/2016 à 22:08, Robert Kroeger a écrit :
> Hi Frédéric,

[Frédéric Wang]

Le 04/10/2016 à 18:50, Frédéric Wang a écrit :
> For now,
> Initalize* calls happen in the same process but (after
> https://codereview.chromium.org/2366623002/) in different threads. Also
> a mojo connector is passed to InitializeUI, so we are now able to do
> Mojo communication between UI & GPU processes in Ozone/Wayland.

Obviously, I only meant UI & GPU "components" here, since "split mus" is
not complete yet :-)