Re: Multi-window placement

[Mike Wasserman]

Hey Zoltan, This is very interesting, thanks for reaching out!

There is no active Chromium effort to support our multi-screen APIs on Android or WebView, but we'd welcome your contributions.

I recently disabled the permission and then drafted a change to re-enable multi-screen info access on Android.

I'm more familiar with desktop platforms, but I can loop in Android/WebView experts when you have more specific questions, designs, or patches.

Feel free to also join #fugu at https://chromium.slack.com/ or email our fugu...@chromium.org list (+CC'ed, but it's sadly mostly spam these days).

Thanks,

Mike Wasserman

On Tue, Aug 23, 2022 at 10:42 AM Brad Triebwasser <btr...@chromium.org> wrote:

+Mike Wasserman who is the lead on our Window placement work. Mike can you help answer this?

On Mon, Aug 22, 2022 at 11:44 PM Zoltan Kuscsik <zol...@s57.io> wrote:

Hi Brad,

  we don't know each other, but I hope you don't mind this mail. My team is working on enabling Multi-Window placement for Android WebView. This work is done under the umbrella of Poly working with Google, to build a solution of running Google Meet on a dual-display Android conferencing system.

 We are making some good progress and I would like to make sure that we go in a direction that would allow us to upstream the solution into Chromium/WebView. Specifically, when it comes to WebView there are couple of degrees of freedom how this could integrate with the Android context.

Do you know if anyone in the Chromium/WebView team is looking into Android support? If yes, I would like to make sure that our work can aligns to that direction and we can make a meaningful contribution.

Thanks,

//Zoltan

Zoltan Kuscsik

CEO, SW Engineer

Solution57 ApS
Strømmen 6, 9400 Nørresundby, Denmark

http://s57.io

DK mobile: +45 26 127 372

US mobile: +1 (669) 224 9114

The contents of this e-mail and its attachments are addressed to theare addressed to the

intended recipient only, and are confidential and may contain legally

privileged information. If you are not the intended recipient, please

notify the sender by replying to this e-mail and delete the message from

your system.

[Zoltan Kuscsik]

Hi Mike,

thanks for the quick response. Me and Michael are the main devs on this feature. We figured out the permission switch already.

We are looking to extend the WebChromeClient Android API to provide hints about the targeted display for a new view and delegate

the actual window placement to the Android world.

Michael tried to join the fugu-dev, but got any response yet. 

Br,

//Zoltan

---

From: Mike Wasserman <m...@chromium.org>
Sent: Thursday, August 25, 2022 8:18 PM
To: Zoltan Kuscsik <zol...@s57.io>
Cc: btr...@chromium.org <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

 

[Mike Wasserman]

Hey Zoltan and Michael,

I sent out a meeting invite to chat with +Peter Conn (an engineer working on Chrome Android) and +Ben Morss (a PM supporting multi-screen web platform APIs).

This should be a good opportunity for us to discuss plans that will hopefully bring more multi-screen window placement support on Android and WebView!

If you don't mind, can you please help reiterate your context and goals here or in a doc (I'll add some naive starting points there), to help us get up to speed quicker.

Thanks,

Mike

[Zoltan Kuscsik]

Hi Mike,

thanks for organizing the call. Here we summarized the context, progress and discussion topics:

Muti-windows placement meeting points.docx

Looking forward to our call tomorrow,

//Zoltan

---

From: Mike Wasserman <m...@chromium.org>
Sent: Thursday, September 1, 2022 7:47 PM
To: Zoltan Kuscsik <zol...@s57.io>; Peter Conn <pec...@google.com>; Ben Morss <mo...@google.com>
Cc: btr...@chromium.org <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>; Michael Storgaard <mic...@s57.io>
Subject: Re: Multi-window placement

 

[Peter Conn]

Hey Zoltan and Michael,

I've been looking into window positioning on the Android side and I was wondering if you had come across the Presentation API - "A presentation is a special kind of dialog whose purpose is to present content on a secondary display.".

If so, were there any shortcomings that prevented you from using it?

Thanks,

Peter C

[Zoltan Kuscsik]

Hi Peter,

in the demo Michael showed is using the Presentation API. I do think it is the right way to place additional browser windows on the secondary screen when the window is full screen.

 Specially, that it is a Dialog type that is tied to the Activity running on the primary screen. The shortcomings are that it doesn't support windowing. 

These are the different approaches we are investigating:

1.) Presentation view for second screen support

Pros:

      - tied to the activity context running on the primary screen. It is a Dialog type not a new Activity.

      - the management of the dialog could be even delegated to the android_webview in chromium.

Cons:

       - no window positioning. Support for only full-screen windows on external/secondary displays.

2.) Window positioning achieved by placing new windows into the application's active view hierarchy (No-Android multi window features used)

3.) Window positioning by creating new Android windows.

      

     This would resemble the closest how the desktop version of Chrome is handling windows.

  However, this would only work on large screen devices that enabled freeform window management/PiP at system build time.

Ideally, we should propose an extension to WebChromeClient that allows any of these implementations on the application level. 

This would include the extension of WebChromeClient::onShowCustomview to provide the additional information of multiple windows/position/screen location. 

+ @Deprecated

public void onShowCustomView(View view, WebChromeClient.CustomViewCallback callback);

+ public void onShowCustomView(PopupView view, WebChromeClient.CustomViewCallback callback);

Where PopupView extends the android View adding providing the additional hints on the window placement and full screen mode.

WebChromeClient shall also support dismissing a specific window. 

//Zoltan

---

From: Peter Conn <pec...@google.com>
Sent: Friday, September 9, 2022 12:41 PM
To: Zoltan Kuscsik <zol...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>; Ben Morss <mo...@google.com>; btr...@chromium.org <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>; Michael Storgaard <mic...@s57.io>
Subject: Re: Multi-window placement

 

[Peter Conn]

Thanks for the really detailed response!

So it seems that the solutions we're looking at are really independent of how the Android side implementation works.

[Mike Wasserman]

That's an awesome video demo, and I'm so excited about continued progress here! Thanks for sharing!

I'll generally defer to Peter for WebChromeClient extension advice, but I have some questions/thoughts from my initial read:

Where in the stack are JS window.open() and window.moveTo() coordinates translated from multi-screen origin values to something consumed by Android?

The proposal suggests using the same multi-screen origin coordinate system for onCreateWindow and onMoveTo, but WindowManager.LayoutParams#x|y seem to generally convey coordinates relative to a specific screen's edge given a particular gravity?

Hopefully individual web content embedding applications won't need to do much complex Web <-> Android coordinate translation, especially for per-display scaling and other complex cases.

Also, you might like to skim my related considerations for specifying coordinates relative to the pertinent display or the multi-screen origin.

It might make sense for new APIs to specify a target display and coordinates relative to the display's top-left corner (and specific whether they're display-independent pixels or physical device pixels?).

Mike

On Mon, Oct 17, 2022 at 10:30 AM Michael Storgaard <mic...@s57.io> wrote:

Hey Peter

The displayId comes via DisplayAndroidManagerJni.get().updateDisplay when updateDisplayOnNativeSide is called. Then on native side in DisplayAndroidManager::UpdateDisplay the Display is created/updated using the sdkDisplayId int, which we get from DisplayAndroidManagerJni.get().updateDisplay. So the ID in native code is the same as we get from a particular Display object from the DisplayManager.

 

We have modified DisplayAndroidManager so we add all physical displays during initialization. Likewise we have modified DisplayAndroidManagerJni.get().updateDisplay so we can specify the x,y position in the virtual screen arrangement for the screen/display. For now we have hardcoded all displays to be horizontal aligned adjacent to each other with primary display being leftmost at (0,0). This is also done in DisplayAndroidManager for now.

 

I am off until next Monday.

 

Cheers,

 

Michael Storgaard

 

Senior SW Engineer

URL: https://s57.io

Phone: +45 41 6000 43

 

The contents of this e-mail and its attachments are addressed to the intended recipient only,

and are confidential and may contain legally privileged information. If you are not the intended

recipient, please notify the sender by replying to this e-mail and delete the message from your system.

 

From: Peter Conn <pec...@google.com>
Date: Monday, 17 October 2022 at 18.04
To: Michael Storgaard <mic...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>, Zoltan Kuscsik <zol...@s57.io>, Ben Morss <mo...@google.com>, btr...@chromium.org <btr...@chromium.org>, fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

Hey Michael,

 

Thanks for the dock, we'll give it a thorough read this week.

 

A quick question though - the APIs refer to screenId, an int that works with Android's DeviceManager. How does the web contents learn of these? I'm assuming not through some standard web API?

 

Thanks,

Peter C

 

On Fri, 14 Oct 2022 at 15:18, Michael Storgaard <mic...@s57.io> wrote:

Hi Mike and Peter

Attached is our initial proposal for the API extenstion required to WebChromeClient to enable multi-screen window placement; in particular for fullscreen requests and window placement.

 

In the document there is also a link to a demo-video showing the current state of our work using the suggested proposal.

 

We welcome your feedback.

 

Best regards

 

Michael Storgaard

 

Senior SW Engineer

URL: https://s57.io

Phone: +45 41 6000 43

 

The contents of this e-mail and its attachments are addressed to the intended recipient only,

[Mike Wasserman]

Great! Thank you, Peter!

Hopefully the JS API's existing feature detection is equally applicable for desktop and mobile. e.g. my demo does this:

if (!('getScreenDetails' in self) || !('isExtended' in screen) || !('onchange' in screen)) { ... }

There are open questions around detecting platform limitations for getting/setting windowed client bounds, etc. (i.e. on Wayland) that may also apply to Android.

Mike

On Fri, Oct 21, 2022 at 5:56 AM Peter Conn <pec...@google.com> wrote:

Hey,

I had a chat with some others about the changes to WebChromeClient and everything seems very sensible.

The one query is whether we care about feature detection - will the web page need to detect whether this is supported? If so, the current solution doesn't provide that, and doing so 

introduces more complexity.

Peter C

On Tue, 18 Oct 2022 at 19:30, Michael Storgaard <mic...@s57.io> wrote:

Thanks for the feedback. We will go through your document and Using cross-screen coordinates or per-screen coordinates. The actual implementation of virtual screen arrangement is something that we need to look further into; in particular the complex corner cases and what API extensions are needed for defining the screen arrangement from the Android app. We welcome your thoughts on this. As it is for now, the coordinate provided in WindowManager.RelativeLayout is the multi-screen origin coordinate for a certain window. It should be relative to the target display top-left corner as you also point out. Our suggestion is that the associated translation (and virtual screen configuration/management for android webview) should be done within the realm of DisplayAndroidManager using a new dedicated class for this. In the current version of our Android demo app, we are only relying on the screen ID provided via onCreateWindow in the proposed API. This is used for presenting the new window as a full screen child view in the activity on the target display. 

BR Michael

---

Fra: Mike Wasserman <m...@chromium.org>
Sendt: tirsdag, oktober 18, 2022 12:38 AM
Til: Michael Storgaard <mic...@s57.io>
Cc: Peter Conn <pec...@google.com>; Zoltan Kuscsik <zol...@s57.io>; Ben Morss <mo...@google.com>; btr...@chromium.org <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>
Emne: Re: Multi-window placement

 

[Mike Wasserman]

Thanks for sharing this update!

You might be interested to follow the ongoing effort to rename the "window-placement" provisional Permission API string to "window-management":

See https://crbug.com/1328581, https://github.com/w3c/window-placement/issues/114, and https://github.com/w3c/window-placement/pull/115

On Mon, Oct 31, 2022 at 6:03 AM Michael Storgaard <mic...@s57.io> wrote:

Sorry for the very late we reply, we have internally been busy with some (non-related) releases and had to revisit the proposed approach for permission management as well. Using android.permission.WINDOW_PLACEMENT was a wrong as you also point out, since this relates to Android itself.

 

However, we suggest to use onPermissionRequest delegate method on WebChromeClient for letting the host app controlling window placement permission, which seems to be aimed for this kind of purpose. In conjunction with this, we propose extending the list of PermissionRequest resources with RESOURCE_WINDOW_PLACEMENT (android.webkit.resource.WINDOW_PLACEMENT) for window placement.

 

From a high-level implementation perspective, we will do this by extending AwPermissionManager for window-placement according to the same principles / patterns used for protected media identifier permission, MIDI/Sysex messages and geolocation.

 

We have an implementation using this approach that partially woks now. See video here for grant and deny:

 

https://zkres-my.sharepoint.com/:v:/g/personal/michael_s57_io/EcdccriWg7FPkwbLPZTfmfsBxxryOoBqji1_ppRHBl6hhQ

 

However, the are some issues with ensuring having the correct permission state in the permission result cache. We are currently looking into this.

 

BR Michael

 

From: Peter Conn <pec...@google.com>
Date: Tuesday, 25 October 2022 at 09.54
To: Michael Storgaard <mic...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>, Zoltan Kuscsik <zol...@s57.io>, Ben Morss <mo...@google.com>, Brad Triebwasser <btr...@chromium.org>, fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

I think that's a good suggestion - but I'd like to see this part of the design explicitly called out.

 

Is WebView going to be responsible for checking what permissions the embedding app has?

 

Could you expand more on the WINDOW_PLACEMENT permission, it looks like it's a Chrome thing, not an Android thing. How will the user grant it?

 

On Mon, 24 Oct 2022, 14:11 Michael Storgaard, <mic...@s57.io> wrote:

Shouldn't this be controlled via the permission API only? If the manifest includes the permission declaration for android.permission.WINDOW_PLACEMENT and permission is granted, then the assumption is that the embedder will override these methods. However, in case permission is granted but the embedder has not implemented these delegate methods, then behavior will be as it is today when issuing window.open or fullscreen request; i.e. nothing happens.

 

BR Michael

 

From: Peter Conn <pec...@google.com>
Date: Monday, 24 October 2022 at 12.06
To: Mike Wasserman <m...@chromium.org>
Cc: Michael Storgaard <mic...@s57.io>, Zoltan Kuscsik <zol...@s57.io>, Ben Morss <mo...@google.com>, btr...@chromium.org <btr...@chromium.org>, fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

The problem with the standard JS feature detection is that it will detect whether the browser currently supports that API - but in this case we need to know whether the embedder supports multi-window placement. In other words, has the embedder overridden the methods that we're talking about adding?

 

 

[Ben Morss]

Hi all,

I just wanted to follow up on this interesting thread from November. Michael, were you able to make progress here?

Best,

Ben

Ben Morss | Product Manager

|

mo...@google.com |

212-565-0113

On Thu, Nov 3, 2022 at 5:37 PM Michael Storgaard <mic...@s57.io> wrote:

Yes, we have noticed that there was some ongoing renaming related to this and will follow the new naming convention and rename it to android.permission.WINDOW_MANAGEMENT. Thanks for links and for pointing this out

[Ben Morss]

Sounds great! Zoltan, thanks for the update.

On Tue, Jun 13, 2023 at 9:33 AM Michael Storgaard <mic...@s57.io> wrote:

Adding Todd Page from HP/Poly PM to the mail thread.

 

Michael Storgaard

 

Senior SW Engineer

URL: https://s57.io

Phone: +45 41 6000 43

 

The contents of this e-mail and its attachments are addressed to the intended recipient only,

and are confidential and may contain legally privileged information. If you are not the intended

recipient, please notify the sender by replying to this e-mail and delete the message from your system.

 

From: Zoltan Kuscsik <zol...@s57.io>
Date: Monday, 12 June 2023 at 16.43
To: Ben Morss <mo...@google.com>, Michael Storgaard <mic...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>, Peter Conn <pec...@google.com>, Brad Triebwasser <btr...@chromium.org>, fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

Hi Ben,

 

  we have prepared the patches for upstreaming, and we are waiting for the feedback from Poly/HP for a green light on releasing the code.

 

   We have made the changes of WebView SDK extension based on the spec. Our test target is the AOSP master emulator x86 + x86 WebView.

 

In addition, we also have the updated Android SDK working with Chromium. Michael is fixing some regression of the patch set to latest chromium main.

 

 

//Zoltan

 

 

 

 

 

 

 

---

From: Ben Morss <mo...@google.com>
Sent: Monday, June 12, 2023 4:32 PM
To: Michael Storgaard <mic...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>; Peter Conn <pec...@google.com>; Zoltan Kuscsik <zol...@s57.io>; Brad Triebwasser <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

 

Hi all,

Muti-windows placement meeting points.docx

[Ben Morss]

That's exciting! Thanks for letting us know, and for helping us be aware of your continued progress.

On Fri, Jul 7, 2023 at 3:32 AM Zoltan Kuscsik <zol...@s57.io> wrote:

Hi Ben,

we started the upstreaming process. The AOSP specific changes:

https://android-review.googlesource.com/c/platform/frameworks/base/+/2651659

Working through the corresponding WebView changes.

Br,

//Zoltan

---

From: Ben Morss <mo...@google.com>
Sent: Monday, June 12, 2023 4:32 PM

To: Michael Storgaard <mic...@s57.io>
Cc: Mike Wasserman <m...@chromium.org>; Peter Conn <pec...@google.com>; Zoltan Kuscsik <zol...@s57.io>; Brad Triebwasser <btr...@chromium.org>; fugu-dev <fugu...@chromium.org>
Subject: Re: Multi-window placement

 

[Mike Wasserman]

Hey Zoltan and s57.io team, I hope all is well!

Do you have any progress or plans to share regarding your work in this space?

I haven't noticed a CL for upstreaming support in Chromium to pair with the AOSP upstream CL. 

Window Management API support for Android was requested recently in Chromium and Github issue trackers.

Best,

Mike