Better orientation management in FirefoxOS

[Alive]

Hi,

Recently we found we have a poor control to this API in FirefoxOS:
1. mozLockOrientation/mozUnlockOrientation.
2. orientation property in app manifest.

The story is: this API is open to any web content, so any web page could lock the device orientation. This is good if the web content is foreground and there’s no app transition is happening. But if we are transitioning the app and it’s using this API - something weird will happen.

My question is: could we have better control from OS side to prevent this happen? I found background app could lock orientation long time ago and submitted a patch to fix it [1], but that seems to break again now. Moreover, even we could prevent a background page to lock screen orientation, we cannot prevent during app transition. And since we are introducing widgets now, I think OS should have the ability to ignore the lockOrientation request from a widget. Also haida is introducing a mechanism to switch between landscape and portrait apps but I am afraid this could not be easily implemented for now.[3] I am using Android phone and what I observed is android didn’t have a good UX when it’s switching between landscape and portrait apps. The screen usually goes black when you launches a landscape app from portrait home screen. iOS does better than android.

Screen orientation lock API is more like an UI-facing API but it’s now gecko controls we can or cannot lock. Some cases apply to requestFullScreen API. OS - gaia system app - should be able to take decision.

My proposal would be: let gaia system app has the ability to "return false" to any screen orientation lock/unlock request from any app/web content if now we are in a condition no body could lock the orientation.

Thanks for reading.

[1]https://bugzilla.mozilla.org/show_bug.cgi?id=868914
[2]https://bugzilla.mozilla.org/show_bug.cgi?id=868914#c26
[3]https://bugzilla.mozilla.org/show_bug.cgi?id=992085
-- 
Alive C. Kuo, Firefox OS, Senior Software Engineer at Mozilla Taiwan, Taipei office.

[Tim Chien]

My quick idea would be again re-invent these APIs to mozbrowser
event(s): let mozbrowser iframe dispatch a event named
"mozbrowserlockorientation" and "mozbrowserunlockorientation" and
expose a method within the event to deny() or allow() the request. The
event can follow my key event proposal and re-dispatch to grandparent
frame so it would always handled by System app.

That said, the above case does not give us the opportunity to display
two apps with different orientations at the same time, for haida
sheet/app opening/app closing transitions. This bug had hunt us from
1.0 and we need a better API than what I had in mind above to properly
fix it (or we will need to keep the screenshots, yuck)

Perhaps we can make each of the mozbrowser iframe comes with it's own
orientation, independent of the phone? I am not sure if the layout
code give us this kind of magic though.

Bug 868914 (deny lock orientation when not visible) sounds like a
meaningful patch for the current API, however that means we
unfortunately overload setVisible() again. We badly need technical
ownership and WebAPI leadership to pick what's left behind by :jlebar
here and come up with a overall plan on mozbrowser APIs.

--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

[Ehsan Akhgari]

On 2014-05-22, 2:22 AM, Tim Chien wrote:
> My quick idea would be again re-invent these APIs to mozbrowser
> event(s): let mozbrowser iframe dispatch a event named
> "mozbrowserlockorientation" and "mozbrowserunlockorientation" and
> expose a method within the event to deny() or allow() the request. The
> event can follow my key event proposal and re-dispatch to grandparent
> frame so it would always handled by System app.

If I understand the issue correctly, it's not limited to web content
loaded in the browser, it also affects apps loaded in a mozapp iframe
outside of the browser app, so that won't be sufficient, right?

> That said, the above case does not give us the opportunity to display
> two apps with different orientations at the same time, for haida
> sheet/app opening/app closing transitions. This bug had hunt us from
> 1.0 and we need a better API than what I had in mind above to properly
> fix it (or we will need to keep the screenshots, yuck)
>
> Perhaps we can make each of the mozbrowser iframe comes with it's own
> orientation, independent of the phone? I am not sure if the layout
> code give us this kind of magic though.

I'm not really familiar with the implementation of this API, so I'm not
sure if I can give you a good answer here. Perhaps try asking on
dev-b2g if nobody else here seems to have an answer? If you can't find
out the answer, let me know and I'll try to dig through the code to find
it for you. :-)

> Bug 868914 (deny lock orientation when not visible) sounds like a
> meaningful patch for the current API, however that means we
> unfortunately overload setVisible() again. We badly need technical
> ownership and WebAPI leadership to pick what's left behind by :jlebar
> here and come up with a overall plan on mozbrowser APIs.

Can you please clarify what you're asking for here? I'm not familiar
with the problem you're describing.

Thanks!
Ehsan

[Tim Chien]

On Fri, May 23, 2014 at 4:14 AM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> On 2014-05-22, 2:22 AM, Tim Chien wrote:
>>
>> My quick idea would be again re-invent these APIs to mozbrowser
>> event(s): let mozbrowser iframe dispatch a event named
>> "mozbrowserlockorientation" and "mozbrowserunlockorientation" and
>> expose a method within the event to deny() or allow() the request. The
>> event can follow my key event proposal and re-dispatch to grandparent
>> frame so it would always handled by System app.
>
>
> If I understand the issue correctly, it's not limited to web content loaded
> in the browser, it also affects apps loaded in a mozapp iframe outside of
> the browser app, so that won't be sufficient, right?

Everything in B2G is loaded in their nth level mozbrowser iframe
inside the system app mozbrowser iframe. mozapp iframe is special kind
of mozbrower iframe, and mozbrowser API is available there too. So
having system-facing API exposed through the mozbrowser iframe method
is sufficient.

Web content v.s. web app has no difference for mozbrowser APIs. Also,
web content can be loaded anywhere, inc. System app, not just the
Browser app which hides the web content in the 2rd level of the
supposedly "Web Phone".

>> That said, the above case does not give us the opportunity to display
>> two apps with different orientations at the same time, for haida
>> sheet/app opening/app closing transitions. This bug had hunt us from
>> 1.0 and we need a better API than what I had in mind above to properly
>> fix it (or we will need to keep the screenshots, yuck)
>>
>> Perhaps we can make each of the mozbrowser iframe comes with it's own
>> orientation, independent of the phone? I am not sure if the layout
>> code give us this kind of magic though.
>
>
> I'm not really familiar with the implementation of this API, so I'm not sure
> if I can give you a good answer here. Perhaps try asking on dev-b2g if
> nobody else here seems to have an answer? If you can't find out the answer,
> let me know and I'll try to dig through the code to find it for you. :-)
>

Yeah I will ask around. Thanks.

>
>> Bug 868914 (deny lock orientation when not visible) sounds like a
>> meaningful patch for the current API, however that means we
>> unfortunately overload setVisible() again. We badly need technical
>> ownership and WebAPI leadership to pick what's left behind by :jlebar
>> here and come up with a overall plan on mozbrowser APIs.
>
>
> Can you please clarify what you're asking for here? I'm not familiar with
> the problem you're describing.
>

Nothing specific here actually... I was just reiterate the fact we
overload setVisible() again in bug 868914 and the fact there isn't a
go-to person for mozbrowser API (and impl) anymore.

> Thanks!
>
> Ehsan

Thank you for your prompt reply!

[Dale Harvey]

I just wanted to agree on this point, I have a few mozbrowser patches in
progress and I dont know who I should be sending them to / consulting about
them on

>
> > Thanks!
> >
> > Ehsan
>
> Thank you for your prompt reply!
>
> --
> Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
> OS, Mozilla Corp. (Taiwan)

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

[Ehsan Akhgari]

On 2014-05-23, 12:29 AM, Tim Chien wrote:
> On Fri, May 23, 2014 at 4:14 AM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>> On 2014-05-22, 2:22 AM, Tim Chien wrote:
>>>
>>> My quick idea would be again re-invent these APIs to mozbrowser
>>> event(s): let mozbrowser iframe dispatch a event named
>>> "mozbrowserlockorientation" and "mozbrowserunlockorientation" and
>>> expose a method within the event to deny() or allow() the request. The
>>> event can follow my key event proposal and re-dispatch to grandparent
>>> frame so it would always handled by System app.
>>
>>
>> If I understand the issue correctly, it's not limited to web content loaded
>> in the browser, it also affects apps loaded in a mozapp iframe outside of
>> the browser app, so that won't be sufficient, right?
>
> Everything in B2G is loaded in their nth level mozbrowser iframe
> inside the system app mozbrowser iframe. mozapp iframe is special kind
> of mozbrower iframe, and mozbrowser API is available there too. So
> having system-facing API exposed through the mozbrowser iframe method
> is sufficient.
>
> Web content v.s. web app has no difference for mozbrowser APIs. Also,
> web content can be loaded anywhere, inc. System app, not just the
> Browser app which hides the web content in the 2rd level of the
> supposedly "Web Phone".

OK, I see what you mean now, sorry for the confusion. I think it makes
sense to allow the embedding app to decide on this behavior as you
suggest above.

>>> That said, the above case does not give us the opportunity to display
>>> two apps with different orientations at the same time, for haida
>>> sheet/app opening/app closing transitions. This bug had hunt us from
>>> 1.0 and we need a better API than what I had in mind above to properly
>>> fix it (or we will need to keep the screenshots, yuck)
>>>
>>> Perhaps we can make each of the mozbrowser iframe comes with it's own
>>> orientation, independent of the phone? I am not sure if the layout
>>> code give us this kind of magic though.
>>
>>
>> I'm not really familiar with the implementation of this API, so I'm not sure
>> if I can give you a good answer here. Perhaps try asking on dev-b2g if
>> nobody else here seems to have an answer? If you can't find out the answer,
>> let me know and I'll try to dig through the code to find it for you. :-)
>>
>
> Yeah I will ask around. Thanks.
>
>>
>>> Bug 868914 (deny lock orientation when not visible) sounds like a
>>> meaningful patch for the current API, however that means we
>>> unfortunately overload setVisible() again. We badly need technical
>>> ownership and WebAPI leadership to pick what's left behind by :jlebar
>>> here and come up with a overall plan on mozbrowser APIs.
>>
>>
>> Can you please clarify what you're asking for here? I'm not familiar with
>> the problem you're describing.
>>
>
> Nothing specific here actually... I was just reiterate the fact we
> overload setVisible() again in bug 868914 and the fact there isn't a
> go-to person for mozbrowser API (and impl) anymore.

For API related issues, please reach out to me, and I'll do my best to
help. I may need some historical background on some of the decisions
made previously in some cases since this API mostly predates me. I'm
looking for an owner on the implementation side of things, stay tuned!

Cheers,
Ehsan

[Mike Habicher]

This is a slight tangent but it's an issue we're having with the Camera app so I'm bringing it up here.

We need a way to override the rotation of dialog boxes. To provide a smooth user experience, the Camera app locks rotation to portrait and adjusts its own UI in response to rotation events. As a result, whenever the phone is being held in landscape mode and a prompt appears (e.g. "Allow GPS?") the dialog appears sideways. It would be very nice to either (a) have the Camera app be able to tell the system to display these prompts with a different orientation; or (b) have the system detect the orientation itself and display the prompts properly, independent of the lock.

--m.

On 14-05-22 12:28 AM, Alive wrote:

Hi,

Recently we found we have a poor control to this API in FirefoxOS:
1. mozLockOrientation/mozUnlockOrientation.
2. orientation property in app manifest.

The story is: this API is open to any web content, so any web page could lock the device orientation. This is good if the web content is foreground and there’s no app transition is happening. But if we are transitioning the app and it’s using this API - something weird will happen.

My question is: could we have better control from OS side to prevent this happen? I found background app could lock orientation long time ago and submitted a patch to fix it [1], but that seems to break again now. Moreover, even we could prevent a background page to lock screen orientation, we cannot prevent during app transition. And since we are introducing widgets now, I think OS should have the ability to ignore the lockOrientation request from a widget. Also haida is introducing a mechanism to switch between landscape and portrait apps but I am afraid this could not be easily implemented for now.[3] I am using Android phone and what I observed is android didn’t have a good UX when it’s switching between landscape and portrait apps. The screen usually goes black when you launches a landscape app from portrait home screen. iOS does better than android.

Screen orientation lock API is more like an UI-facing API but it’s now gecko controls we can or cannot lock. Some cases apply to requestFullScreen API. OS - gaia system app - should be able to take decision.

My proposal would be: let gaia system app has the ability to "return false" to any screen orientation lock/unlock request from any app/web content if now we are in a condition no body could lock the orientation.

Thanks for reading.

[1]https://bugzilla.mozilla.org/show_bug.cgi?id=868914
[2]https://bugzilla.mozilla.org/show_bug.cgi?id=868914#c26
[3]https://bugzilla.mozilla.org/show_bug.cgi?id=992085

-- 
Alive C. Kuo, Firefox OS, Senior Software Engineer at Mozilla Taiwan, Taipei office.
_______________________________________________
dev-webapi mailing list
dev-w...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-webapi

[Ehsan Akhgari]

On 2014-05-23, 12:15 PM, Mike Habicher wrote:
> This is a slight tangent but it's an issue we're having with the Camera
> app so I'm bringing it up here.
>
> We need a way to override the rotation of dialog boxes. To provide a
> smooth user experience, the Camera app locks rotation to portrait and
> adjusts its own UI in response to rotation events. As a result, whenever
> the phone is being held in landscape mode and a prompt appears (e.g.
> "Allow GPS?") the dialog appears sideways. It would be very nice to
> either (a) have the Camera app be able to tell the system to display
> these prompts with a different orientation; or (b) have the system
> detect the orientation itself and display the prompts properly,
> independent of the lock.

That's an interesting case. Can we detect that the orientation is
locked in the system app and have it respected by the dialogs that it
renders automatically?

Cheers,
Ehsan

[Alive]

開啟 2014年5月24日 at 上午3:08:53, Ehsan Akhgari (ehsan....@gmail.com) 寫:

I’d think this is a case not in the scope of this topic.

The problem is camera app changes its UI by using device motion API to detect ‘current device position’

but in fact it told system app to lock its orientation to default in its manifest.[1]

The only way to resolve this is to create some API for app to tell system app to display another orientation dialog.



Think about this:

App A tells system app to lock the orientation in the manifest, and system app does so when launching App A.

This is fine. But when it called window.alert(‘..'), I don’t think system app needs to display a ‘rotating-with-device’ dialog.

The dialog should just be at the same orientation with the app specified, unless we have a new way to call window.alert().



[1] https://github.com/mozilla-b2g/gaia/blob/master/apps/camera/manifest.webapp#L6



Cheers, 
Ehsan 

[John Hu]

----- Original Message -----
> From: "Ehsan Akhgari" <ehsan....@gmail.com>
> To: "Mike Habicher" <mi...@mozilla.com>, "Alive" <al...@mozilla.com>, "dev-webapi" <dev-w...@lists.mozilla.org>
> Cc: tch...@mozilla.com, j...@mozilla.com
> Sent: Saturday, May 24, 2014 3:08:53 AM
> Subject: Re: Better orientation management in FirefoxOS
>
> On 2014-05-23, 12:15 PM, Mike Habicher wrote:
> > This is a slight tangent but it's an issue we're having with the Camera
> > app so I'm bringing it up here.
> >
> > We need a way to override the rotation of dialog boxes. To provide a
> > smooth user experience, the Camera app locks rotation to portrait and
> > adjusts its own UI in response to rotation events. As a result, whenever
> > the phone is being held in landscape mode and a prompt appears (e.g.
> > "Allow GPS?") the dialog appears sideways. It would be very nice to
> > either (a) have the Camera app be able to tell the system to display
> > these prompts with a different orientation; or (b) have the system
> > detect the orientation itself and display the prompts properly,
> > independent of the lock.
>
> That's an interesting case. Can we detect that the orientation is
> locked in the system app and have it respected by the dialogs that it
> renders automatically?

Right, it is an interesting case. When an app locks the screen orientation, we don't have any chance to get the real device orientation through any API, including screen.mozOrientation, unless calculate by ourselves. It's the lesson learned from camera app.

>
> Cheers,
> Ehsan

[Jonas Sicking]

Hi All,

Alive and I had some discussions today about the problem at hand. I
haven't yet gone through this whole thread, so apologies if I'm
repeating something.

The basic need that we have is to enable the system app to participate
more in the orientation control of apps. For example we don't want
background apps to spend time on adjusting to orientation changes.
That includes avoiding reflowing them, but also avoiding changing
their screen.orientation or firing the event that goes along with
that.

Additionally we need to handle the fact that during a transition
between two apps, there are apps that are marked as visible, but we
don't want to allow the screen orientation to change.

Below is the proposal that we came up with. Note that this proposal
relies on implementing the updated screen orientation API. The
important difference there is that the new API makes lockOrientation
an asynchronous API, rather than the synchronous API that it is now.

New functions:

On the top level mozbrowser API, add the ability to call
setOrientation("landscape-primary"). This would set the orientation of
all pages of all pages inside the embedded iframe and its descendants.

We could also add the ability to call unsetOrientation() (or
setOrientation("")) to avoid the explicit set and instead let gecko
propagate the "real" orientation. Though this isn't strictly needed
since the system app could simply listen to orientation changes and
forward the orientation as needed.

(Come to think of it, we could also add
setOrientation/unsetOrientation to nested mozbrowser iframes. However
we don't have any real use cases for that for now. So for now we
should probably just do what's easiest).

New event:

We also need to add a "orientationlockrequest" event to the mozbrowser
API. This event is fired any time a page inside the mozbrowser iframe
calls "screen.lockOrientation". The event would have a
`setResult(bool)` function. When setResult(false) is called, we reject
the promise returned from lockOrientation with a "NotSupportedError"
error. When setResult(true) is called, we resolve the promise returned
from lockOrientation. Note that the exact timing of when the promise
is resolved, and the value it is resolved with, is defined by the
screen orientation spec, and Gecko should honor that once
setResult(true) has been called.

The default action of the "orientationlockrequest" event is to call
setResult(false). Note that this allows the system app to make a
true/false decision asynchronously by always cancelling the event and
then calling setResult once it knows the answer.

Doing this should enable us to fix most of the edge cases that we have today.

Fix fullscreen API:

Additionally we might need to fix the mozFullScreenElement attribute
to make it work even when the target element is an mozbrowser iframe.
Right now that appears to be broken (possibly only in OOP cases).

It'd also be nice to update our fullscreen API to follow the latest
spec and be unprefixed, but that's orthogonal.

/ Jonas

[John Hu]

Hi Jonas

Few questions embedded inline:

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Alive" <al...@mozilla.com>
> Cc: "Tim Guan-tin Chien" <tch...@mozilla.com>, "dev-webapi" <dev-w...@lists.mozilla.org>, j...@mozilla.com
> Sent: Friday, June 20, 2014 6:27:48 PM
> Subject: Re: Better orientation management in FirefoxOS
>

> Hi All,
>
> Alive and I had some discussions today about the problem at hand. I
> haven't yet gone through this whole thread, so apologies if I'm
> repeating something.
>
> The basic need that we have is to enable the system app to participate
> more in the orientation control of apps. For example we don't want
> background apps to spend time on adjusting to orientation changes.
> That includes avoiding reflowing them, but also avoiding changing
> their screen.orientation or firing the event that goes along with
> that.
>
> Additionally we need to handle the fact that during a transition
> between two apps, there are apps that are marked as visible, but we
> don't want to allow the screen orientation to change.
>
> Below is the proposal that we came up with. Note that this proposal
> relies on implementing the updated screen orientation API. The
> important difference there is that the new API makes lockOrientation
> an asynchronous API, rather than the synchronous API that it is now.
>
> New functions:
>
> On the top level mozbrowser API, add the ability to call
> setOrientation("landscape-primary"). This would set the orientation of
> all pages of all pages inside the embedded iframe and its descendants.

*. What's the difference between "top level mozbrowser API" and "mozbrowser API"?
According to the offline discussion with alive, we all think "top level" means System app. So, only System app has the setOrientation API. That means a homescreen which embeds widgets should not and can not to handle it.

> We could also add the ability to call unsetOrientation() (or
> setOrientation("")) to avoid the explicit set and instead let gecko
> propagate the "real" orientation. Though this isn't strictly needed
> since the system app could simply listen to orientation changes and
> forward the orientation as needed.
>
> (Come to think of it, we could also add
> setOrientation/unsetOrientation to nested mozbrowser iframes. However
> we don't have any real use cases for that for now. So for now we
> should probably just do what's easiest).
>
> New event:
>
> We also need to add a "orientationlockrequest" event to the mozbrowser
> API. This event is fired any time a page inside the mozbrowser iframe
> calls "screen.lockOrientation". The event would have a
> `setResult(bool)` function. When setResult(false) is called, we reject
> the promise returned from lockOrientation with a "NotSupportedError"
> error. When setResult(true) is called, we resolve the promise returned
> from lockOrientation. Note that the exact timing of when the promise
> is resolved, and the value it is resolved with, is defined by the
> screen orientation spec, and Gecko should honor that once
> setResult(true) has been called.

*. Following the same thought of "top level", the "orientationlockrequest" should only be fired to "top level", shouldn't it? If it is not, orientationlockrequest events who are fired to intermediate iframe should not have "setResult(bool)" in it.

> The default action of the "orientationlockrequest" event is to call
> setResult(false). Note that this allows the system app to make a
> true/false decision asynchronously by always cancelling the event and
> then calling setResult once it knows the answer.
>
> Doing this should enable us to fix most of the edge cases that we have today.
>
> Fix fullscreen API:
>
> Additionally we might need to fix the mozFullScreenElement attribute
> to make it work even when the target element is an mozbrowser iframe.
> Right now that appears to be broken (possibly only in OOP cases).
>
> It'd also be nice to update our fullscreen API to follow the latest
> spec and be unprefixed, but that's orthogonal.
>
> / Jonas

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

John Hu

[Jonas Sicking]

Correct.

>> We could also add the ability to call unsetOrientation() (or
>> setOrientation("")) to avoid the explicit set and instead let gecko
>> propagate the "real" orientation. Though this isn't strictly needed
>> since the system app could simply listen to orientation changes and
>> forward the orientation as needed.
>>
>> (Come to think of it, we could also add
>> setOrientation/unsetOrientation to nested mozbrowser iframes. However
>> we don't have any real use cases for that for now. So for now we
>> should probably just do what's easiest).
>>
>> New event:
>>
>> We also need to add a "orientationlockrequest" event to the mozbrowser
>> API. This event is fired any time a page inside the mozbrowser iframe
>> calls "screen.lockOrientation". The event would have a
>> `setResult(bool)` function. When setResult(false) is called, we reject
>> the promise returned from lockOrientation with a "NotSupportedError"
>> error. When setResult(true) is called, we resolve the promise returned
>> from lockOrientation. Note that the exact timing of when the promise
>> is resolved, and the value it is resolved with, is defined by the
>> screen orientation spec, and Gecko should honor that once
>> setResult(true) has been called.
>
> *. Following the same thought of "top level", the "orientationlockrequest" should only be fired to "top level", shouldn't it? If it is not, orientationlockrequest events who are fired to intermediate iframe should not have "setResult(bool)" in it.

Correct.

/ Jonas