Screen capture problems

[alex2206]

Hi all,

I use screen capture API of ddmlib to capture G1 screen (Android 1.5).

Sometime the captured result is the previous screen, not the current.
I think it is due to double buffer mechanism.
So that I decided to fix it in ddmlib code to get correct screen-shot
every-time.
I investigated in ddms/libs/ddmlib/src/com/android/ddmlib/
AdbHelper.java at:
AdbHelper.getFrameBuffer().
Is it a right way? However, I can not fix this problem.
Can you suggest me someway to solve this problem? modify
AdbHelper.getFrameBuffer(), invoke some system command to refresh
buffer, etc... ?

[Adrian Taylor]

Hi,

On Thu, 2010-02-11 at 01:16 -0800, alex2206 wrote:

> I use screen capture API of ddmlib to capture G1 screen (Android 1.5).
>
> Sometime the captured result is the previous screen, not the current.
> I think it is due to double buffer mechanism.

That's roughly correct.

DDMS ultimately just reads the contents of /dev/graphics/fb0. However,
that doesn't necessarily show the genuine screen image.

We believe this is due to these lines in SurfaceFlinger's
DisplayHardware.cpp:

if (mFlags & PARTIAL_UPDATES) {
mNativeWindow->setUpdateRectangle(dirty.getBounds());
}

Without these lines, the device screen shows exactly the same corruption
visible via DDMS.

> So that I decided to fix it in ddmlib code to get correct screen-shot
> every-time.
> I investigated in ddms/libs/ddmlib/src/com/android/ddmlib/
> AdbHelper.java at:
> AdbHelper.getFrameBuffer().
> Is it a right way? However, I can not fix this problem.
> Can you suggest me someway to solve this problem? modify
> AdbHelper.getFrameBuffer(), invoke some system command to refresh
> buffer, etc... ?

We (at RealVNC) would like to fix this, since ultimately we hope to
persuade the Android platform to add the APIs necessary to allow remote
control of Android phones. (One of those APIs we need is a reliable
screen shot API).

It would be great to hear from the Android platform architects about how
to fix this.

Our proposal is:

* SurfaceFlinger regains its old LayerScreenshot.cpp which it had
back in Android 1.0.
* But, it's modified to use glReadPixels to retrieve the genuine
screen image. (This works reliably, though it's quite slow).
* SurfaceFlinger exposes a new Binder interface to request a
screenshot.
* SurfaceFlinger checks that callers of this interface have
android.permission.READ_FRAME_BUFFER.
* A new native library is provided which exposes a simple C
interface, yet internally uses C++ to talk to the new Surface
Flinger Binder interface.
* The device-side part of adb is modified to link to this new C
library rather than reading /dev/graphics/fb0.
* The device-side part of adb gains
android.permission.READ_FRAME_BUFFER.

Android platform maintainers: Does that sound reasonable?

If so we'll get started! We have an engineer ready and waiting to do
this work; in fact we know the first three steps above work fine.

Later we'll want to provide a version of this screenshot API accessible
from on-device software, secured appropriately [1].

Regards

Adrian

[1]
http://groups.google.com/group/android-platform/browse_thread/thread/77a2add4384202dc

[Dianne Hackborn]

(1) I wouldn't want such a thing that only requires the app declaring a permission.  The user must explicitly opt in (and be able to opt out) of the app doing this, with appropriate warning, like IMEs and other things.

(2) Having an operation here that you admit is quite slow doesn't seem like a good idea, does it?

--
You received this message because you are subscribed to the Google Groups "android-platform" group.
To post to this group, send email to android-...@googlegroups.com.
To unsubscribe from this group, send email to android-platfo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-platform?hl=en.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[Adrian Taylor]

Hi Dianne, thanks for the feedback.

(1) READ_FRAME_BUFFER will remain a 'signature' permission. So it will only be accessible to 'adb' and other parts of the ROM. If (when?) you allow remote control applications like VNC servers, then we will have to do something more sophisticated, as per reference [1] in the e-mail, but let's worry about that later.

Hopefully this solves that concern.

(2) No! But we don't know any other reliable way to get the genuine screen image except for glReadPixels within SurfaceFlinger. Can you suggest a better way? Or point us to the Surface Flinger maintainer - is it still Mathis Agopian?

(And it's not that slow - 5-10fps is possible - which isn't great for a VNC-like solution but is fine for other users of adb/DDMS who just want to capture screenshots).

Please do let us know if there's a different way you'd prefer this to be done. We're eager to get started.

Adrian

[Joshua Park]

(1)

Hi,
I am also interested in Screenshot/remote control API.
I have an idea on the security for Screenshot/remote control API
and thought I should leave a comment.

Android has special checkboxes for 'Unkown sources' & 'USB debugging'
'Unkown sources' : Settings -> Applications
'USB debugging' : Settings -> Applications -> Development

Like so, add a checkbox for 'Remote Control'.
This setting will not just have a checkbox but add password field as
well.

So, the application not only requires to add
android.permission.READ_FRAME_BUFFER
and others, but also 'Remote Control' checkbox needs to be checked
(manually by the user) and the password has to be matched to access
remote control API.

It will be the application developer's task to prompt some kind of GUI
for the app users
to type in the correct password.

To avoid the security flaw of 'brute force attack', it may be a good
idea to
disable Remote Control checkbox if an app tried wrong password 3
times.
(then it has to be manually checked again by the user)

[brian]

I've used the DroidEx application that also works in conjuction with
ddms to capture android screens. Ideally it would be great to use a
facility like this to capture video playback from the phone.

see http://groups.google.com/group/android-discuss/browse_thread/thread/939b320cea3c81d4

[Joshua Park]

Thank you Brian for sharing information.
I already know about the screen capturing capability through ddms.
But it is limited to USB connection and Key Injection capability is
not available.
I am really hoping Google will provide a Secure API to do a remote
control.
Else people may just root their devices to achieve their needs.

On Feb 12, 12:21 pm, brian <walsh.brian.patr...@gmail.com> wrote:
> I've used the  DroidEx  application that also works in conjuction with

> ddms tocaptureandroid screens.  Ideally it would be great to use a

> facility like this tocapturevideo playback from the phone.
>

> seehttp://groups.google.com/group/android-discuss/browse_thread/thread/9...

[alex2206]

Thanks all your reply!

I am still investigated in this problem.
Anyway, captured screen-shot will become correct after some seconds,
but we have to wait, sometime only 2-3 seconds, sometime 30 seconds!
Could you tell me the maximum time we should spend to wait until
captured screen-shot becomes correct ?

[Mike Playle]

We've encountered this problem too in our attempts to build a VNC
server for Android.

I've noticed that the frame buffer contents sometimes become correct
at the turn of the minute, when the current time displayed in the top
right corner is updated. So you could have to wait for up to a minute
to get a correct screen image! However this behaviour is very hardware
dependent and we don't consider this to be reliable - I can easily
believe that there might be situations where /dev/graphics/fb0 never
becomes correct, no matter how long you wait.

We're working on a way to fix this by grabbing the screen using
glReadPixels() from within the SurfaceFlinger. This should be a much
more reliable way of getting an accurate screen image. Hopefully it
won't be long before we've got a proposed patch to post for
discussion.

Eventually we hope to build our VNC server on top of this mechanism,
but of course this depends on coming up with an acceptable security
model for protecting access to this interface. In the meantime though
it should be possible to improve the DDMS screenshot mechanism without
exposing any risky APIs to applications.

Regards,

Mike Playle
RealVNC

[Mike Playle]

On Mon, 2010-03-08 at 12:55 +0000, Mike Playle wrote:
> We're working on a way to fix this by grabbing the screen using
> glReadPixels() from within the SurfaceFlinger. This should be a much
> more reliable way of getting an accurate screen image. Hopefully it
> won't be long before we've got a proposed patch to post for
> discussion.

Our proposed screenshot fix is now on Gerrit here:

https://review.source.android.com/13953
https://review.source.android.com/13954
https://review.source.android.com/13955

Any comments are more than welcome.

Mike Playle
RealVNC