BrowserComponent: mouse events handled differently in Android device than simulator

[P5music]

P5music

12:16 (ora) 

a CodenameOne Discussions

I am testing my CodenameApp on a real Android device.

The main screen has a table layout with a container on the left.

The container has a container inside, that has a BrowserComponent inside in BorderLayout.CENTER.

The issue is not about layout.

I have some callbacks for the BC that on the simulator handle mouseup/down on the iFrames in the HTML.

That callback work also for handing long-press on the same HTML elements.

Everything is fine. If I add a LongPressListener it gets not called but it is not needed at all.

When using the app on the real Android device the clicks are handled as in the simulator

while

the long press is not.

I debugged and it seems that the events are not fired at all.

I mean,

a click is mousedown and mouseup within a short time range.

a longpress is the same where the mouseup event is after 500ms. The app uses a timer.

But the initial mousedown event itself is not called, and also the LongPressListener is not called.

So what is happening? This is very important because the long-press is for displaying a pop-up  menu. But only clicks work.

Thanks in advance

[Shai Almog]

Are you long clicking on the browser component or somewhere else? It's unclear from the question.  Do you have a screenshot of the place you're clicking on?

[P5music]

The click is on the iFrames inside the HTML, so it is on the BC.

It works on the simulator:

click and long-press gestures are handled with a timer and comparing the timestamps.

The issue is that

when I long-press on the BC when the app runs on the real Android device,

even the first mouse event (mousedown) is not fired,

it is like the BC has a timer itself and does not fire the mousedown event if the mouseup event is too far in time,

while in the simulator the mousedown event is issued every time, regardless of the timing of the subsequent mouseup event.

Regards

[Shai Almog]

That sounds like a bug in the simulator not on the device. If you click in the browser component the event should be delivered there. Not to Codename One. You need to intercept it in JavaScript.

[P5music]

I am talking about are handled in javascript

indeed I explicitly wrote about callbacks.

(the longpress listener also is not called, it is in Java just for testing purposes as a fallback but it is not useful because it is not called either)

This is the problem:

The first event should be "mousedown"

and it is in the simulator, whatever kind of click type the user does, i.e. click or longpress.

The second event should be "mouseup"

and it is in the simuator, whatever kind of click type the user does, i.e. click or longpress, just the difference is timing.

The Javascript callbacks work and a suitable part of the code is called for each of the above mentioned events.

(It is not a simulator bug, it works as it should)

So In case of click those events are fired correctly both in simulator and also in the Android device.

Instead

in case of long press on the real Android device only click is handled correctly and I see the correct sequence of mouse events,

but with long press those events are not fired, like they are gobbled down.

So both the down event and the up event are not handled, they do not trigger the callbacks, they are not fired ore they are gobbled.

It seems that the WebView has a timer itself and decides whether to fire the first event according to the timing of the second.

Regards

[Shai Almog]

If it's in JavaScript then it sounds like a bug. Can you extract a test case and file an issue?

[P5music]

I created a test case but before filing an issue I have to point out what follows. (You find the relevant code below)

The test app features a CEF BrowserComponent with some text inside.

Text lines are not so long horizontally to fill the view, so the right part is empty.

Also the bottom part is empty because only a few text lines are in the HTML.

On the Android device:

I can click on text and I have the mousedown/mouseup sequence (events are fired and callbacks are issued).

I can click on the empty part on the right and I have the  mousedown/mouseup sequence (events are fired and callbacks are issued). 

If I click on the empty part on the bottom I have no callbacks.

If I long-press on the text the Android text-selection interface appears.

If I long-press on the empty part on the right no callbacks are issued (this is the issue this thread is about but it is entwined with something else, read on).

I remind that the empty part on the right is a DIV with the callbacks on it.

And the bottom part is really empty.

Sofor my app to work

-the text should not be selectable in the BrowserComponent because in my app the long-press on any iFrame in the list is just for launching the pop-up menu.

-the long-press events should be issued also on the text, if present in the iFrames.

How to have those two specs satisfied?

It seems that there are two questions, one is maybe the simplest to solve, the other is maybe a bug to file an issue about. 

And the two are together, it seems.

(They are not seen in the simulator)

Regards

public void start() {
if(current != null){
current.show();
return;
}
Form hi = new Form("Hi World");
hi.setLayout(new BorderLayout());
hi.add(BorderLayout.NORTH,new Label("Hi World"));
BrowserComponent bc=new BrowserComponent();

bc.addWebEventListener("onLoad", new ActionListener() {
@Override
public void actionPerformed(ActionEvent evt) {
System.out.println("onload");
bc.addJSCallback("var div=document.getElementById('divId');" +
"callMouseDown =function(){callback.onSuccess('mousedown');};callMouseUp =function(){callback.onSuccess('mouseup');};"+
"div.addEventListener('mousedown', function () {"+
"callMouseDown();"+
"});"+
"div.addEventListener('mouseup', function () {"+
"callMouseUp();"+
"});"
, new SuccessCallback<BrowserComponent.JSRef>() {
@Override
public void onSucess(BrowserComponent.JSRef value) {
System.out.println("webview callback value="+value.getValue());
}});
}
});

bc.setPage("<HTML><BODY><DIV id='divId' ><DIV>TEST</BR>TEST</BR>TEST</BR>TEST</BR>TEST</BR>TEST</BR>TEST</BR></DIV></DIV></BODY></HTML>","");

hi.add(BorderLayout.CENTER,bc);
hi.show();
}

[Shai Almog]

OK, is this filed as an issue so I can assign it to Steve?

[P5music]

CEF BrowserComponent: mouse events handled differently in Android device than simulator · Issue #3378 · codenameone/CodenameOne (github.com)   

[P5music]

I need the information whether this issue could impact iOS or not.

Is it possible to know this while the issue is not fixed yet on Android? 

If it is not on iOS too, for example thanks to more consistent APIs, I could start testing on those devices, but some kind of action has to be taken on my side, so it would be useful to know in advance.

Thanks in advance

[Shai Almog]

I don't know, it's hard to tell from a review of the code.

[P5music]

Please can you point me to some source code so I can inspect it about the issue I filed?

It's about CEF so it boils down to some interface for mouse events.

Thanks

[Shai Almog]

It's generally the touch events in the glviewcontroller file. From there you can just follow the flow all the way out to the Java code in IOSImplementation.

[P5music]

Well, I would like to compare to the relevant  Android code, because I meant that code, however you pointed me to the iOS code so I looked at that.

If the Android and iOS are similar, I can see that an array of touches is managed and some touches could be lost.

Some code fragments that hint at that direction:

things like

pointerDraggedC(xArray, yArray, [touches count]);

or

 pointerDraggedC(xArray, yArray, [touchesArray count]);

in

-(void)touchesBegan

-(void)touchesEnded

-(void)touchesCancelled

or also

skipNextTouch

all seem to prune the events.

But I think that also the text-selection interface that pops up on Android could lead to a cancellation of events if this code is also in Android:

if(touchesArray != nil) {

        [touchesArray removeAllObjects];

    }

in

(void)updateCanvas:(BOOL)animated 

Regards

[Shai Almog]

That's an array for the multi-touch behavior not a buffer of touches. That means that if you have 2, 3+ fingers on the screen we'll get the events for all of them.

[P5music]

And what about the possibility that the long-press event not being fired is a byproduct of the text-selection-interface

so this code

if(touchesArray != nil) {

        [touchesArray removeAllObjects];

    }

in

(void)updateCanvas:(BOOL)animated 

effectively remove the touches when the interface appears.

So the bug would be: 

why the text-selection-interface appears also when the iFrames are set not to process pointer events?

Fixing this, the text-selection-interface would not be triggered and long-press events would be fired again and go through the event bubble chain.

Regards

[Shai Almog]

A long press is triggered in the Codename One layer not in the native layer. We detect it when receiving a press and not receiving a release after a while.

[P5music]

So, as I said, the LongPressListener method is not called, so it has to be fixed, but I recall that you said that I should handle those events with Javascript.

Said that,

whatever method will work when the issue is fixed (I hope that on iOS the issue is not present but I have yet to check)

it seems that the code

if(touchesArray != nil) {

        [touchesArray removeAllObjects];

    }

in

(void)updateCanvas:(BOOL)animated 

could remove the events

when the text-selection-interface is triggered.

It seems that you used the gestures, not the touches so it could lead to this behaviour.

I can manage the non-wanted mouse events from long-pressing the  iFrames

but

the interface for text-selection has not to appear.

Regards

[Shai Almog]

The long press isn'5 the problem. It's the press/release that aren't sent since the browser component grabs them. That's the correct behavior.

[P5music]

Ok so it is something that is on iOS too probably, according to your analysis.

Now I think it is importat to fix the filed issue

because it is very simple, a matter of just looking at the code and understanding that wrong timing.

Even if it is a non-enterprise issue, it's blocking of a very important component, crucial for modern app development in many cases.

It is also completely blocking my app, so it is different from other non-blocking issues.

It's just a matter of time that an enterprise customer stumbles upon it. Better it does not happen and they find the component correctly working in the first place.

Regards

[Shai Almog]

We'll get to it when we do. You can debug it on Android using Android Studio pretty easily.

[shannah]

A couple of comments:

1. BrowserComponent is a heavyweight/native component and its implementation will be different on different platforms.  

2. Debugging hybrid apps (that use both Javascript/HTML and CN1 UI) is necessarily more complex than debugging an app where all the UI is in Codename One.  You should use Safari's debugging tools to debug the webview on device.  You should use the Chrome debugging tools to debug the webview in the simulator and on Android device.  

3. Most important:  When developing complex JS your life will be easier if you can structure it so that you can test it independently from the Codename One app.  Create your HTML and javascript, host it on a server (even localhost - but not file:///), and test it in the web browsers of the platforms you want to support.  In your case you're looking for specific behaviour related to events (whether text should be selected in response to a mouse event, etc..), so it may take a little tweaking to get it perfect in an HTML only sandbox.  Once you have it working, then you can plug in the CodenameOne piece.

An intermediate step before fully integrating your JS with your Codename One app, is to *just* load the HTML page in a Browser component without any communication between Java and Javascript. See if any functionality that works in the platform browser app breaks when run inside a BrowserComponent.

After that is working perfectly, you're ready to add the java <-> javascript communication piece.

Steve

[P5music]

Hi Steve,

thanks for chiming in,

as you can read in the github issue I filed, the correct behaviour is already found on the simulator. 

No need to further debug, because debug has been already done on Android device too.

The issue is on Android and probably on iOS. I do not mind doing a test on iOS device myself but it cannot be done today because I must create certificates yet.

Furthermore there is no such complex case as you seem to be thinking. The issue is very simple:

text is selectable where it should not be (pointerEvents='none' on iFrame), and this leads to mouse events being captured by the system and not the BC or the CN layer, or viceversa, I do not know.

Regards

[Steve Hannah]

> Furthermore there is no such complex case as you seem to be thinking. The issue is very simple:

> text is selectable where it should not be (pointerEvents='none' on iFrame), and this leads to mouse events being captured by the system and not the BC or the CN layer, or viceversa, I do not know.

That is not a simple case.  You're mixing event models between web and CN1.  It's incredibly complex.

I recommend you isolate your Javascript web portion so that you can test it on each platform you target without CN1.  

--
You received this message because you are subscribed to the Google Groups "CodenameOne Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codenameone-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codenameone-discussions/bc6eb9a2-32d0-4285-93e3-f9b6bea8060an%40googlegroups.com.

--

Steve Hannah

Web Lite Solutions Corp.

[P5music]

There is not any difficult case, because on the simulator it works very well as expected.

I am not mixing anything, please read this issue to be fixed:

https://github.com/codenameone/CodenameOne/issues/3378

I just put a callback for mousedown and mouseup, and it gets called on a DIV, with a iFrame inside. 

The iFrame does not respond to mouse events, the DIV does.

Simple click works and the two events are fired and the callbacks get called.

Long press also works on the simulator, I just handle the different timing.

But on Android it happens as I described, not working: I can select text on the iFrame but its style settings should not allow (pointerEvents='none') so it's not OK.

It causes or is related to the DIV not respoding to long press, while it still responds to simple click.

I do not mind whether I have to handle it with JS or with the longPressListener, it is enough that it works, just tell me what is the right way.

I was said that JS is the right way. Ok, my source code is handling that way, it is ready, no modification is needed.

Please ask for other details if necessary.

[Shai Almog]

We can't fix different behaviors of Androids webkit browser, if a specific div from the webkit implementation on Android doesn't send an event (or does) for us the webkit is a black box.We can only fix bugs in the interconnect.

[P5music]

"We can only fix bugs in the interconnect"

Yes, please have a look at that.

Thanks

[P5music]

I found a workaround:

there are special events on touch devices that are not involved in the issue.

I have to use those events instead of mousedown, mouseup, which still are useful in the IDE simulator in fact.

String touchStartEventString;
String touchEndEventString;

if (Utils.isSimulator())
{
touchStartEventString="mousedown";
touchEndEventString="mouseup";
}
else
{
touchStartEventString="touchstart";
touchEndEventString="touchend";
}

That special events are always correctly fired also on Android device.

For the IDE simulator the normal events are the only ones possible, because it is not a real touch device.

It is also necessary to have this:

user-select: none;-webkit-user-select: none;

added to the style attribute of the BODY tag in the HTML so that the text is forced not to be selectable everywhere.

This workaround should be right on iOS too, even if the issue is not present there.

Of course it is not applicable to the general case where the issue is present but it I enough for my needs.

Regards