Android : Tapping a character from soft keyboard : Finding keyboard using TAG_NAME with the contex

[Umar Ali Khan]

Windows 7
Android 4.3

I am trying to tap on character keys on soft keyboard on Google Nexus 7 device (not an emulator) as following:

driver.findElement(By.tagName("keyboard"));

but Appium is unable to find it and gives following error:

info: [ANDROID] [info] Got command of type ACTION
info: [ANDROID] [debug] Got command action: find
info: [ANDROID] [debug] Finding keyboard using TAG_NAME with the contextId:
info: [ANDROID] [info] Returning result: {"value":"No element found","status":7}

info: [ADB] Received command result from bootstrap

I am using latest appium for windows (0.8.3). Please help.

[Al Villaflor]

Why not just use the key event mobile method?
https://github.com/appium/appium/blob/master/docs/finding-elements.md

[Umar Ali Khan]

thanks Al. The method you told works fine. Thanks for that. Also sendKeys work fine too.

But according to git check-ins, it seems that

driver.findElement(By.tagName("keyboard"));

statement should be working fine. The check-in was done like 20 days ago.
 
What I want to achieve is that while executing the script, the keyboard keys should be shown pressed and released and gives real time end user input.

[Jonathan Lipps]

That checkin dealt with iOS, I believe.

On Android you don't see the keyboard keys being actively pressed the way you do on iOS

--
http://appium.io
---
You received this message because you are subscribed to the Google Groups "Appium-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appium-discus...@googlegroups.com.
Visit this group at http://groups.google.com/group/appium-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Umar Ali Khan]

Thanks for the clarification guys.

I have two more queries:

1. Any idea by when Appium on Windows will support iOS automation
2. I want to pick the model number from Android device, any idea how?

Thanks

Umar

[Dan Cuellar]

1. I don't think this is far off, we've determined it's possible, just getting the time to implement it. That being said, no work has been started on this yet (that I'm aware of)

2. Check the adb docs on google. I'm sure there has to be a way to do this.

[jwallis]

Using Appium 11.1 (Android 17), I do not seem to be able to get ahold of the keyboard using tagName("keyboard") or finding any keys, elements, etc.  I have tried page_source() which has no mention of any keys/keyboards, the GUI appium inspector, which does not ever show a red rectangle around the keyboard, and tried other location strategies to no avail.  I am trying to tap the Send button on my (non-custom according to my dev) keyboard.  Any help would be appreciated…

[bootstrap online]

Try using mobile keyevent. Look up the keycode from the Android docs.

[jwallis]

I did, and I appreciate that response.  I tried KEYCODE_ENTER, which works on the emulator, but not on a physical device : (

I have since been trying to use getKeyCharacterMap but don't know the syntax, and keep getting "Message: u'Not yet implemented..."

[bootstrap online]

Are you using Selendroid or uiautomtator? Selendroid has their own API
which may be an alternative.

[jwallis]

uiautomator I believe.  Appium 11.1, Android 17.  I inherited this project a month ago so I'm not as familiar as I'm sure I could be...

[bootstrap online]

keyevent uses pressKeyCode when using uiautomator.
https://github.com/appium/appium/blob/fa1f063018c5c173eddfbf8fbdb7c58d860c4db2/lib/devices/android/bootstrap/src/io/appium/android/bootstrap/handler/PressKeyCode.java#L41

If that doesn't work, then I think you're out of luck. There are some
more methods here:
http://developer.android.com/tools/help/uiautomator/UiDevice.html

However I don't think they'll help if pressKeyCode fails.

[jwallis]

Thank you so much.  Ops took my device away so I'll have to try tomorrow : (

[jwallis]

Well sending '\n' or using KEYCODE_ENTER now work for my app under test (it's early in development).  I determined the checkin that made it work and it certainly doesn't make sense why it would change this behavior.  I will ask a dev tomorrow.

I apologize if I'm being dense, but I cannot figure out how to use the methods on http://developer.android.com/tools/help/uiautomator/UiDevice.html

I assume they'd be passed to wd.execute_script or wd.execute, but since I get "Not yet implemented. Please help us: http://appium.io/get-involved.html" for so many things, they're apparently getting massaged a little on a case-by-case basis by the appium/selenium layers before getting passed along to UIAutomator…?

[Jonathan Lipps]

android stuff is precompiled so we can't pass arbitrary uiautomator commands via appium.

is there something implemented in uiautomator that's not implemented in appium?

[jwallis]

Ha, of course.  It's been so long since I've worked directly in a compiled language.

I was just confused as to how I can call the Android methods in

http://developer.android.com/tools/help/uiautomator/UiDevice.html

with a python webdriver instance.  I'm not sure if it should look something like this (which works fine)

my_webdriver.execute_script("mobile: keyevent", {"keycode":66})

or if it's completely different.  Hence my confusion as to whether I'm just missing the syntax or the whole concept.

[Jonathan Lipps]

There's no 1-1 mapping between appium commands and UiDevice commands. We have a whole intermediate layer called the android bootstrap whose purpose is to proxy commands back and forth. this is what defines the appium protocol. it's all in the codebase under lib/devices/android if you want to take a look.

So my question is, what do you want to do? There's probably a way to do it using appium's methods (as you've figured out below with keyevent)--looking at the UiDevice docs won't help you figure out what that is, though it will help you understand what appium is doing under the hood.

[jwallis]

I've dug in some, but not having an ton of time, and having no knowledge of js, I've come up with what I think the technology stack looks like.  This has helped a little, but my main disconnect is how the remoteWebDriverServer interacts with the bootstrap.  Maybe, hopefully, it doesn't really matter for me getting my job done... 

https://docs.google.com/drawings/d/1AuVf_y7-zKLQ38eHbrIFuYpu3lEBszFd1FMSUZMbIog/edit?usp=sharing

I think when this started I was trying to figure out how to call the android UiDevice commands because I was still unable to tap the ENTER button or have a send_keys('\n') on a hardware device (worked on emulator).  I though it was my issue but a dev figured out that the app was not accounting for "older style" keyboards, and made a commit with the following comment:

"In onEditorAction, added a check for actionId being equal to EditorInfo.IME_NULL, to send messages, for the benefit of older devices that use non-standard keyboards."

Thank you for your help Jonathan.  I don't know if it'd help for people to see a full and correct appium stack diagram but maybe it would answer some questions so you wouldn't have to...?  Diagrams at this level are fairly unhelpful if you're trying to dig in:

http://www.3pillarglobal.com/sites/default/files/appium1.png

[Jonathan Lipps]

thanks for putting together that diagram. it's pretty close to correct, but not entirely. let me give some more detail.

here's what happens on android (the story is different for ios):

- when we want a new session, we push AndroidBootstrap.jar to the device and run it using uiautomator

- this process fires up, and since the main class extends UiAutomatorTestCase, it has access to UiDevice and so on.

- after the process starts, it spins up a socket server. this socket server sits and waits

- meanwhile, the main Appium process (in node), waits for the AndroidBootstrap to start on the device. once it does, it creates a connection to the socket server the bootstrap has created

- now Appium (node, host machine) and AndroidBootstrap (java, android device) can send information back and forth

- when Appium receives a command from the client, it passes that command as a JSON string to the AndroidBootstrap server.

- the server parses this string for the command, and essentially looks up the appropriate handler in a map.

- the server calls this handler, gets the response, and passes it back as JSON to the Appium server

So if you want to add behavior to appium's android automation, you need to do a few things:

1) ensure the server can handle the client's request for this new behavior (this probably means modifying the controller files in appium)

2) convert that request into a command string which can be passed from the Appium server to the AndroidBootstrap

3) teach AndroidBootstrap how to recognize that command string and turn it into the appropriate UiAutomator calls

hope this helps!

[jwallis]

That is extremely informative, thank you!  I have updated the android side of the diagram.  I now feel pretty clear on how that side is working.  If you think this diagram is of use to others, feel free to do anything you want with it.

[naddy095]

You can get all the system information using the adb command - "adb shell cat /system/build.prop | grep "any property that you want" ".

[jwallis]

Likewise with ios, you can use ideviceinfo