Proposal: new APIs for remotely controlling Android devices

[Adrian Taylor]

Hi all,

RealVNC and Sony Mobile Communications would jointly like to contribute APIs to allow remote control of Android devices. In this sense, "Remote Control" means the ability for applications to 'see' the screen image and to inject key/touch events into the platform.

Remote controlling Android devices is useful for:

• Helpdesk applications
  
• Testing tools
  
• TVs & Automotive applications
  
• Applications which wish to take screenshots for any other reason
  

It's also incredibly security-critical. To quote Dianne Hackborn[1],

 

The inject events permission effectively grants an application EVERY possible permission, because it can drive the UI and do anything the user can do.
The screen shot permission allows the application, any time the user is operating the device, to monitor them and do very dangerous things like collect every password they type in web pages and elsewhere.

In fact, the APIs we need are already exposed by the Window Manager and Surface Flinger, to other internal Android components. To make them available to applications, all that's really missing is a solid security model, and that's where we need help.

Security:

We propose to follow the example of Android's new VPN service. Specifically:

• Remote control will be a new 'dangerous' permission
  
• But because few people read the permissions prompts, there will in addition be a new runtime prompt with suitably alarmist wording:
  
• "This application is attempting to take control of your device. This application may access any of your data, impersonate you, or do anything else that you can do when using your phone. Are you sure that you want to allow remote control?"
• There will be a notification whenever remote control is active
  
• Selecting the notification will bring up a second dialog with information about the remote control and the ability to cancel it
  
• Only one application will be allowed to remotely control the device at once
  

In diagrammatic form, the arrangement is shown at the end of this mail.

Is this approach acceptable?

We would like to propose one concession to usability - the addition of an 'Always trust this application' checkbox. If checked, further prompting would not occur on subsequent remote control attempts, unless (a) something else is doing remote control, (b) the user closed remote control using the notification.

Details of the APIs:

android.os.RemoteControl will be a new Java API class. Unlike the VPN service, there is no requirement that remote control applications are a Service. This is because it can sometimes be beneficial to capture a screenshot or to inject an event in a one-off fashion from an Activity, Broadcast Receiver or other component. Of course, full remote control solutions would likely be services but we don't want to mandate that.

Broadly, the APIs provided on android.os.RemoteControl will be:

• getFrameBuffer - returns a ParcelFileDescriptor and size of some ashmem shared memory containing the framebuffer
  
• grabScreen - updates the frame buffer.
  
• setFramebufferScaledSize - requests scaling of the framebuffer
  
• injectKeyEvent(android.view.KeyEvent) - types a key
  
• injectMotionEvent(android.view.MotionEvent) - touches the screen
  
• getDeviceInfo - returns the real screen size of the device (this may differ from the area to which applications are allowed to draw)
  

These exactly correspond to internal APIs already available in the surface flinger and window manager.

There will be callbacks when:

• remote control is revoked;
  
• the device information changes.
  

Compatibility:

We have been using a remote control arrangement based on the existing APIs within the Window Manager and Surface Flinger since Android 2.3 was released. We can confirm that these APIs have worked on every Android device we have tested, from six manufacturers.

However, since android.os.RemoteControl would be a new public API, we would expect to add tests to the CTS to confirm its operation. This would require that each device supported the OpenGL framebuffer object extension. Is that acceptable? We believe many features of ICS already rely on that extension, because this screenshot code is already used to implement various transitions.

DRM:

The Surface Flinger is already aware of what surfaces contain DRM ('secure') content, and will refuse to capture screenshots if such layers are visible.

Performance:

An end-to-end remote control solution based on these APIs is able to attain high frame rates (14fps+) with low CPU impact, given sufficient bandwidth to the viewing device. This is because the approach uses shared memory and because the existing screen capture code within the Surface Flinger uses framebuffer objects.

However, in future, we would like to add follow-up submissions which:

• Enable downsampling to RGB565 on the GPU
  
• Allow for incremental captures (grabScreen doesn't return until there's a change)
  

These allow for 25+fps.

Nevertheless, these are optional enhancements and we would prefer to submit these separately because they involve changes to the Surface Flinger.

Further into the future, we'd like to explore ways to pass the screen image into hardware video encoders or alternatively run OpenGL/RenderScript transforms on the screen image before it leaves the GPU. This should enable full frame-rate HD video. We'd aim to contribute these facilities within the same security architecture.

RealVNC and Sony would be delighted to see these changes in the J release and we're expecting to work hard to achieve that. We'd really be grateful for feedback and suggested improvements, especially with regards to the security model.

Many thanks,

Adrian

[1] https://android-review.googlesource.com/#/c/8866/

PS some very preliminary code can be found at:

https://github.com/adetaylor/android-frameworks-base-with-remote-control-service/compare/remote-control-based-on-vpn-model

but it requires major changes yet. We wanted to discuss the design on this mailing list before submitting code to AOSP, so in that spirit, the code's not ready yet! Notable omissions: (a) new permission, (b) management dialog, (c) removal of old obsolete APIs, (d) simplification to support only a single client, (e) revocation API, (f) actually completing the authorisation process, (g) translations, (h) CTS.

Security design:

+-----------------+                    +------------+

| Surface flinger |--------------------|            |           +---------------+ +-----+

+-----------------+                    | New system |           | New APIs:     | | App |

               Existing Binder APIs    | service:   |-----------| android.os.   |-|     |

            to grab screen and inject  | Remote     |  Binder   | RemoteControl | |     |

               events, protected by    | Control    |           +---------------+ +-----+

              signature permissions    | Service    |                              /

+-----------------+                    |            |                         App /

| Window manager  |--------------------|            |                   launches /

+-----------------+                    +------------+                    intent /

                                             ^ Authorises app                 /

                                             |                               /

                                    +--------------------+                  /

                                    | New system dialogs:|<-----------------

                                    | Remote control     |

                                    | dialogs            |

                                    +--------------------+

[Jean-Baptiste Queru]

Android team, what do you think about this proposal?

It's be good to have some agreement on the overall direction by the
time the next major version gets released as Open Source, in order to
make potentially deep contributions easier to handle.

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Jeffrey Brown]

It gives me the willies.  This is an unprecedented amount of control to grant to an application.

I think whatever means are used should be very narrowly scoped in purpose.  At the least, the user will need to expressly authorize a remote control service to run.  This can be accomplished similar to how VPNs or IMEs or accessibility services work.  "This app wants to control your screen.... etc..."

I don't think we should encourage apps to use this dangerous API for secondary purposes like grabbing screenshots.  If an app is going to use a remote control API then it should really be for purpose of remote control.

I think the proposed API needs to be fleshed out a lot more.  We need to address how a remote control service would register itself, what happens when multiple remote control services are active at once, whether the device should locally lock itself while being remotely controlled, how to escape remote control reliably, etc.

There are a lot of issues here.  We don't want to risk being trapped in a bad design.

Jeff.

[Dianne Hackborn]

Yeah I agree with Jeff.  The VPN or IME design seems like an appropriate model for this.  This should be specifically for screen sharing, with control for instigating it going through system UI.  No point in a permission for such a thing.

I also do not like a design that revolves around polling for screen updates rather than having them pushed to it.  It may be simple for an initial implementation, but it is a really inefficient model that we'd be stuck with.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[Adrian Taylor]

Hi Dianne, Jeff,

Thanks for the feedback.

We'll rework the design to get pushed screen updates from the Surface Flinger. We'll also separate the API for screenshotting, and make some proposals about whether local screen locking can be put in place. We'll also get rid of the permission.

Apart from that, though, I think we're doing exactly what you want. We have exactly copied the VPN subsystem model, with a very similar (but scarier) runtime prompt shown by the system UI. I apologise if that wasn't clear in my initial mail - but I promise we're doing exactly the same thing!

Best wishes

Adrian

-- 
Adrian Taylor     Technical Account Manager
RealVNC Ltd.   Mobile/cell: +44 7779 723608