Design proposal: remote control of Android devices
2,056 views
Skip to first unread message
Adrian Taylor
Dec 7, 2009, 9:57:30 AM12/7/09
to android-platform
Hi all,
This is a proposal for a contribution to enable remote control of
Android
devices. We'd really appreciate all comments and feedback.
The Problem
===========
Most other mobile platforms can be remotely controlled using
technologies
like VNC. This is used by helpdesk, device management, and automated
testing tool vendors. Automotive vendors are also showing an interest
in
having touch screen phone user interfaces appearing in the car
dashboard
display.
(Obviously, all desktop platforms can be controlled remotely, too.)
Remote control needs two things:
* Read access to the screen image
* The ability to inject key and touch events
Both of these on Android are protected using signature-level
permissions,
which means that only software in the device ROM (or signed
equivalently)
can do this.
These permissions can't be exposed more widely because Android's
software
signing scheme is based only around self-signatures[1].
(There is currently *no* way of capturing the framebuffer
programmatically
without root, but that can be rectified once we work out the security
- see
Q&A later.)
The Solution
============
We propose a similar solution to Google's OMA-like device management
design[2] ("DM").
A new service provides access to the functionality needed. But it
checks
the public key of the application against a database of permitted
applications. Only if the key is authorised can the framebuffer be
read or
events injected.
How do applications' keys get into the database?
* Through the operator provisioning code. This has a special
signature-level permission which allows it to add entries to the
database.
* By exposing the database to Android's DM subsystem, such that
additional
applications can be authorized by anyone already authorized to
control the
device using DM.
This satisfies many of our users:
* Our customers who are operators, or who work with operators, can
authorize remote control applications directly.
* Our customers who are enterprises, and who already have the ability
to
configure the device using DM, can also control the device using
remote
control by authorising their favourite remote control application.
What about ad-hoc remote control?
=================================
This doesn't satisfy our entire user base - some users will want to
control
their device remotely despite being unrelated to operators or others
who
may have the right to control the device via DM.
This applies to testing tools, car dashboard viewers and also
developers
wishing to control the phone from their Eclipse-based IDEs.
So we'd like to go a little further.
Alongside the "USB Debugging" checkbox we'd like to add an extra check-
box,
"Allow remote control by anyone". This will normally be turned off,
but
users can turn it on to drop the restrictions on remote control.
If this is regarded as too risky, perhaps similar functionality could
be
enabled only via 'adb shell'.
What will we contribute?
========================
Before JBQ asks :-)
We're very happy to produce all the code for this.
But we really do need your help with the design and security aspects.
What
do the experts think about our proposed design?
Beautiful ASCII artwork
=======================
+----------------+ +--------------------+ +------------------+
| VNC open- | | RealVNC commercial | | Other remote |
| source servers | | servers | | control products |
+----------------+ +--------------------+ +------------------+
| | |
+-------------------------+--------------------------+
|
+----------------------------------------------+
| New API for remote control (event injection, |
| screen grabbing, screen change notification) |
+----------------------------------------------+
|
| Binder
|
+-------------------------------+ ------------
| New remote-control service. | ( Authorised )
| In ROM. Checks authorisation |------( application)
| of callers against database. | ( public key )
| | ( database. )
+-------------------------------+ ------------
| | | ^ ^
| | | | |
+------------+ +---------+ | +---------------+ +---------+
| Surface | | Window | | | Operator | | DM sub- |
| Flinger | | Manager | | | provisioning | | system |
| + new | | | | | code | | (adds |
| Layer | | | | | (adds entries | | entries |
| Screenshot | | | | | on 1st boot) | | later) |
+------------+ +---------+ | +---------------+ +---------+
|
+-----------------------------------+
| Optional new settings application |
| checkbox (allows arbitrary remote |
| control) |
+-----------------------------------+
Questions & answers
===================
Q: What would check the public keys of the remote control
applications?
A: A new service started from the system server.
Q: How would the remote control applications communicate with that
service?
A: Using a new, standard, Android API class. This would use
IServiceManager
and some new Binder interfaces to talk to the remote control
service.
Q: Would these new APIs be Java or native?
A: Well, we'd prefer native, since performance would be better.
Thoughts?
Q: What exactly would be in these APIs?
A: Injecting key and touch events; grabbing the whole screen; grabbing
part of the screen; registering for notifications when something
draws
to part of the screen.
Q: Would these remote control APIs be accessible to other remote
control
schemes beyond VNC?
A: Yes. There would be nothing VNC-specific. And of course these APIs
could
be used by open-source remote control mechanisms. (Those familiar
with
RealVNC will know that we published VNC's RFB protocol and the
original
open-source implementation, and we encourage third-party
implementations.)
Q: Will the new service be in ROM?
A: Yes. It needs to have signature-level permissions to talk to the
surface flinger and window manager.
Q: How much ROM space will the new service take?
A: We'd expect just a few KiB. It doesn't really do anything
complicated.
Q: How will the new service inject events on behalf of remote control
apps?
A: Using IWindowManager.injectPointerEvent/injectKeyEvent.
Q: How will the new service read the frame buffer?
A: SurfaceFlinger used to have an extra type of layer called
"LayerScreenshot" which grabbed the screen and returned it in a
Parcel.
This was guarded by android.permission.READ_FRAME_BUFFER. We'd
reinstate
this mechanism, albeit with a more formal API. We'd also modify
the
screenshot layer such that, instead of directly reading the frame
buffer,
it used OpenGL to render the frame. (We're not sure about this bit
but we
assume it's possible). That way, the mechanism should even work on
Android
devices where the rendering is done in a GPU and there isn't a
traditional
framebuffer.
Q: How will the new service get notified of screen draws?
A: SurfaceFlinger::postFrameBuffer will call a Binder callback, if any
are
registered, sending the dirty region.
Q: Won't that reduce SurfaceFlinger performance?
A: If there are no remote control agents, it will have no effect. Even
if
there are remote control agents, our experience from prototyping
the change
shows that it has negligible effects so long as they respond to
the
callback quickly. And because there's an extra service in the
middle, the
SurfaceFlinger will be shielded from misbehaving applications.
Q: How will you store the application public keys?
A: It would be nice to store them in SharedPreferences such that they
are
automatically exposed via DM[2] but we could instead keep them in
a
separate SQLite database and expose them explicitly via DM.
Thoughts?
Q: How would you submit this?
A: Through the normal Gerrit mechanisms unless you've got any better
ideas!
Obviously we'd break it down into small independent patches insofar
as
possible.
Q: Who are you?
A: RealVNC are the originators of the VNC protocol and open-source
version.
We also sell commercial VNC tools.
Many thanks for your time, and any comments you have.
Adrian.
References:
[1] http://review.source.android.com/Gerrit#change,8866
[2] http://groups.google.com/group/android-platform/browse_thread/thread/39e93b7e31270a36?hl=en
This is a proposal for a contribution to enable remote control of
Android
devices. We'd really appreciate all comments and feedback.
The Problem
===========
Most other mobile platforms can be remotely controlled using
technologies
like VNC. This is used by helpdesk, device management, and automated
testing tool vendors. Automotive vendors are also showing an interest
in
having touch screen phone user interfaces appearing in the car
dashboard
display.
(Obviously, all desktop platforms can be controlled remotely, too.)
Remote control needs two things:
* Read access to the screen image
* The ability to inject key and touch events
Both of these on Android are protected using signature-level
permissions,
which means that only software in the device ROM (or signed
equivalently)
can do this.
These permissions can't be exposed more widely because Android's
software
signing scheme is based only around self-signatures[1].
(There is currently *no* way of capturing the framebuffer
programmatically
without root, but that can be rectified once we work out the security
- see
Q&A later.)
The Solution
============
We propose a similar solution to Google's OMA-like device management
design[2] ("DM").
A new service provides access to the functionality needed. But it
checks
the public key of the application against a database of permitted
applications. Only if the key is authorised can the framebuffer be
read or
events injected.
How do applications' keys get into the database?
* Through the operator provisioning code. This has a special
signature-level permission which allows it to add entries to the
database.
* By exposing the database to Android's DM subsystem, such that
additional
applications can be authorized by anyone already authorized to
control the
device using DM.
This satisfies many of our users:
* Our customers who are operators, or who work with operators, can
authorize remote control applications directly.
* Our customers who are enterprises, and who already have the ability
to
configure the device using DM, can also control the device using
remote
control by authorising their favourite remote control application.
What about ad-hoc remote control?
=================================
This doesn't satisfy our entire user base - some users will want to
control
their device remotely despite being unrelated to operators or others
who
may have the right to control the device via DM.
This applies to testing tools, car dashboard viewers and also
developers
wishing to control the phone from their Eclipse-based IDEs.
So we'd like to go a little further.
Alongside the "USB Debugging" checkbox we'd like to add an extra check-
box,
"Allow remote control by anyone". This will normally be turned off,
but
users can turn it on to drop the restrictions on remote control.
If this is regarded as too risky, perhaps similar functionality could
be
enabled only via 'adb shell'.
What will we contribute?
========================
Before JBQ asks :-)
We're very happy to produce all the code for this.
But we really do need your help with the design and security aspects.
What
do the experts think about our proposed design?
Beautiful ASCII artwork
=======================
+----------------+ +--------------------+ +------------------+
| VNC open- | | RealVNC commercial | | Other remote |
| source servers | | servers | | control products |
+----------------+ +--------------------+ +------------------+
| | |
+-------------------------+--------------------------+
|
+----------------------------------------------+
| New API for remote control (event injection, |
| screen grabbing, screen change notification) |
+----------------------------------------------+
|
| Binder
|
+-------------------------------+ ------------
| New remote-control service. | ( Authorised )
| In ROM. Checks authorisation |------( application)
| of callers against database. | ( public key )
| | ( database. )
+-------------------------------+ ------------
| | | ^ ^
| | | | |
+------------+ +---------+ | +---------------+ +---------+
| Surface | | Window | | | Operator | | DM sub- |
| Flinger | | Manager | | | provisioning | | system |
| + new | | | | | code | | (adds |
| Layer | | | | | (adds entries | | entries |
| Screenshot | | | | | on 1st boot) | | later) |
+------------+ +---------+ | +---------------+ +---------+
|
+-----------------------------------+
| Optional new settings application |
| checkbox (allows arbitrary remote |
| control) |
+-----------------------------------+
Questions & answers
===================
Q: What would check the public keys of the remote control
applications?
A: A new service started from the system server.
Q: How would the remote control applications communicate with that
service?
A: Using a new, standard, Android API class. This would use
IServiceManager
and some new Binder interfaces to talk to the remote control
service.
Q: Would these new APIs be Java or native?
A: Well, we'd prefer native, since performance would be better.
Thoughts?
Q: What exactly would be in these APIs?
A: Injecting key and touch events; grabbing the whole screen; grabbing
part of the screen; registering for notifications when something
draws
to part of the screen.
Q: Would these remote control APIs be accessible to other remote
control
schemes beyond VNC?
A: Yes. There would be nothing VNC-specific. And of course these APIs
could
be used by open-source remote control mechanisms. (Those familiar
with
RealVNC will know that we published VNC's RFB protocol and the
original
open-source implementation, and we encourage third-party
implementations.)
Q: Will the new service be in ROM?
A: Yes. It needs to have signature-level permissions to talk to the
surface flinger and window manager.
Q: How much ROM space will the new service take?
A: We'd expect just a few KiB. It doesn't really do anything
complicated.
Q: How will the new service inject events on behalf of remote control
apps?
A: Using IWindowManager.injectPointerEvent/injectKeyEvent.
Q: How will the new service read the frame buffer?
A: SurfaceFlinger used to have an extra type of layer called
"LayerScreenshot" which grabbed the screen and returned it in a
Parcel.
This was guarded by android.permission.READ_FRAME_BUFFER. We'd
reinstate
this mechanism, albeit with a more formal API. We'd also modify
the
screenshot layer such that, instead of directly reading the frame
buffer,
it used OpenGL to render the frame. (We're not sure about this bit
but we
assume it's possible). That way, the mechanism should even work on
Android
devices where the rendering is done in a GPU and there isn't a
traditional
framebuffer.
Q: How will the new service get notified of screen draws?
A: SurfaceFlinger::postFrameBuffer will call a Binder callback, if any
are
registered, sending the dirty region.
Q: Won't that reduce SurfaceFlinger performance?
A: If there are no remote control agents, it will have no effect. Even
if
there are remote control agents, our experience from prototyping
the change
shows that it has negligible effects so long as they respond to
the
callback quickly. And because there's an extra service in the
middle, the
SurfaceFlinger will be shielded from misbehaving applications.
Q: How will you store the application public keys?
A: It would be nice to store them in SharedPreferences such that they
are
automatically exposed via DM[2] but we could instead keep them in
a
separate SQLite database and expose them explicitly via DM.
Thoughts?
Q: How would you submit this?
A: Through the normal Gerrit mechanisms unless you've got any better
ideas!
Obviously we'd break it down into small independent patches insofar
as
possible.
Q: Who are you?
A: RealVNC are the originators of the VNC protocol and open-source
version.
We also sell commercial VNC tools.
Many thanks for your time, and any comments you have.
Adrian.
References:
[1] http://review.source.android.com/Gerrit#change,8866
[2] http://groups.google.com/group/android-platform/browse_thread/thread/39e93b7e31270a36?hl=en
Reply all
Reply to author
Forward
0 new messages