suggestions on synchronizing ioio with camera frames

[Qianqian Fang]

hi

I want to get some suggestions from you guys, particularly
Ytai, on a ioio+camera project I am working on.

What I am trying to do is to let ioio control a list of LEDs,
and take a few image frames after turning on each
LED.

In my current code, I have a program structure like following:

public class Master extends IOIOActivity {

boolean [] ledstatus;

class Looper extends BaseIOIOLooper {
//write pins based on ledstatus[i]
}

class Preview extends SurfaceView implements SurfaceHolder.Callback,
Camera.PreviewCallback {
// set ledstatus[i] based on the onPreviewFrame() callback
}
}

I want to understand what are the delays that
I need to be aware of once I set the ledstatus[]
inside the onPreviewFrame callback.

My understanding to the sample code, ioio has
its own thread. In the loop() function, there is
a Thread.sleep() to set an interval to check the
status. This is certainly one of the sources for
asynchronism.

From the point pin.write(true) is called to
the full intensity of the LED, what other latencies
that I should be expecting? I saw from the Wiki
that OpenAssessory mode has a latency of 1ms,
and bluetooth has much longer latency, were you
referring to this this delay?

My camera frame speed is about 20fps~30fps
depending on size of the preview, so it will
be really nice to reduce all the latency to be
less than 30ms, so I only need to toss out one
frame.

Your feedback is very much appreciated!

Qianqian

[Ytai Ben-Tsvi]

On Fri, Mar 30, 2012 at 8:49 AM, Qianqian Fang <fan...@gmail.com> wrote:

hi

I want to get some suggestions from you guys, particularly
Ytai, on a ioio+camera project I am working on.

What I am trying to do is to let ioio control a list of LEDs,
and take a few image frames after turning on each
LED.

In my current code, I have a program structure like following:

public class Master extends IOIOActivity {

  boolean [] ledstatus;

  class Looper extends BaseIOIOLooper {
     //write pins based on ledstatus[i]
  }

  class Preview extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback {
      // set ledstatus[i] based on the onPreviewFrame() callback
  }
}

I want to understand what are the delays that
I need to be aware of once I set the ledstatus[]
inside the onPreviewFrame callback.

My understanding to the sample code, ioio has
its own thread. In the loop() function, there is
a Thread.sleep() to set an interval to check the
status. This is certainly one of the sources for
asynchronism.

Calling sleep() from within loop() is not a requirement.

 

From the point pin.write(true) is called to
the full intensity of the LED, what other latencies
that I should be expecting? I saw from the Wiki
that OpenAssessory mode has a latency of 1ms,
and bluetooth has much longer latency, were you
referring to this this delay?

Yes. This refers to the time it takes from DigitalOutput.write() is called, and until the pin's voltage changes.

Please note: if you're latency-sensitive and want to control many LEDs all at once, take a look at IOIO.beginBatch() et al. It will give you a lot of performance improvement.

 

My camera frame speed is about 20fps~30fps
depending on size of the preview, so it will
be really nice to reduce all the latency to be
less than 30ms, so I only need to toss out one
frame.

Your feedback is very much appreciated!

Qianqian


--
You received this message because you are subscribed to the Google Groups "ioio-users" group.
To post to this group, send email to ioio-...@googlegroups.com.
To unsubscribe from this group, send email to ioio-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ioio-users?hl=en.

[Qianqian Fang]

On 03/30/2012 11:55 AM, Ytai Ben-Tsvi wrote:

Calling sleep() from within loop() is not a requirement.

hi Ytai

I removed the sleep call, and my program suddenly
became very sluggish. The timing between switching
on/off each LED became longer and uneven. It appears
like that thread is hogging the CPU and delays responses
from other threads. I put the sleep back afterward.

Yes. This refers to the time it takes from DigitalOutput.write() is called, and until the pin's voltage changes.

good, then OpenAccessory mode is definitely good
enough for me. If I can control ioio via a USB cable
from my galaxy Epic, does that mean the
OpenAssessory mode is working? should I turn off
USB debugging?

Please note: if you're latency-sensitive and want to control many LEDs all at once, take a look at IOIO.beginBatch() et al. It will give you a lot of performance improvement.

I now enclose the pin writing calls with the batch
commands, I do feel that the cycling is more uniform.
thanks for pointing this out!


Qianqian

 

My camera frame speed is about 20fps~30fps
depending on size of the preview, so it will
be really nice to reduce all the latency to be
less than 30ms, so I only need to toss out one
frame.

Your feedback is very much appreciated!

Qianqian


--
You received this message because you are subscribed to the Google Groups "ioio-users" group.
To post to this group, send email to ioio-...@googlegroups.com.

To unsubscribe from this group, send email to ioio-users+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/ioio-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "ioio-users" group.
To post to this group, send email to ioio-...@googlegroups.com.

To unsubscribe from this group, send email to ioio-users+...@googlegroups.com.

[Ytai Ben-Tsvi]

On Fri, Mar 30, 2012 at 9:31 AM, Qianqian Fang <fan...@gmail.com> wrote:

On 03/30/2012 11:55 AM, Ytai Ben-Tsvi wrote:

Calling sleep() from within loop() is not a requirement.

hi Ytai

I removed the sleep call, and my program suddenly
became very sluggish. The timing between switching
on/off each LED became longer and uneven. It appears
like that thread is hogging the CPU and delays responses
from other threads. I put the sleep back afterward.

I wasn't clear enough. What I meant is that having a sleep() isn't required. But surely you must need some kind of blocking in your loop() if you don't want it to hog the CPU. This blocking can be wait() until a new camera frame is available for example.

 

Yes. This refers to the time it takes from DigitalOutput.write() is called, and until the pin's voltage changes.

good, then OpenAccessory mode is definitely good
enough for me. If I can control ioio via a USB cable
from my galaxy Epic, does that mean the
OpenAssessory mode is working? should I turn off
USB debugging?

In order to use OpenAccessory:

1. Your Android must support it. Not sure whether yours does.
2. Your IOIO has to be running app firmware V3.23.
3. Your IOIO should preferably be running bootloader V3.03. or otherwise you'd have to disconnect / reconnect USB after IOIO power-up in order to get the connection established.
4. Your USB debugging needs to be off.
5. You need to build your app in a way that supports it (see here: https://github.com/ytai/ioio/wiki/IOIO-Over-OpenAccessory).

Having that said - my suggestion is: get it working over ADB or BT first... Then see if you need to optimize.

[Qianqian Fang]

On 03/30/2012 02:28 PM, Ytai Ben-Tsvi wrote:

I wasn't clear enough. What I meant is that having a sleep() isn't required. But surely you must need some kind of blocking in your loop() if you don't want it to hog the CPU. This blocking can be wait() until a new camera frame is available for example.

hi Ytai

thanks again for the comments. To be honest, I wasn't
aware of the wait/notify mechanism in Java. After
working on this last weekend, I managed to migrate
the thread communication using wait/notify.

In the new code, after pin.write(), I set a delay
by Thread.sleep(X) to account for the latency, then
I synchronize a shared object to inform the camera
to collect images. I found that when the delay "X"
is greater than 250ms to 300ms, my images are
always sync with the LED status. However, for shorter
X, I can see the random intermediate states where
multiple LEDs are half-way on/off. I believe this is due
to the jitter and latency of BT-based communication.

In order to acquire data in faster speed, I would
like to set up the OpenAccessory mode.

In order to use OpenAccessory:

1. Your Android must support it. Not sure whether yours does.
2. Your IOIO has to be running app firmware V3.23.
3. Your IOIO should preferably be running bootloader V3.03. or otherwise you'd have to disconnect / reconnect USB after IOIO power-up in order to get the connection established.
4. Your USB debugging needs to be off.
5. You need to build your app in a way that supports it (see here: https://github.com/ytai/ioio/wiki/IOIO-Over-OpenAccessory).

Having that said - my suggestion is: get it working over ADB or BT first... Then see if you need to optimize.

I will definitely read more from your wiki, but here I
have a few quick questions and maybe you can help
me to confirm:

1. my IOIOs were purchased a month ago from
sparkfun, do I still need to upgrade the firmware
and bootloader?

2. I imported the libraries from App-IOIO0311.zip,
I saw IOIOLibAccessory in the project list, it appears
there is no error under my Eclipse+ADT14.0.0. Is
this good enough to run OpenAccessory mode?

If an upgrade is needed, I hope the project can still
be backward compatible with BT mode, as it is much
easier to debug.

thanks

Qianqian

[Ytai Ben-Tsvi]

On Mon, Apr 2, 2012 at 9:19 AM, Qianqian Fang <fan...@gmail.com> wrote:

On 03/30/2012 02:28 PM, Ytai Ben-Tsvi wrote:

I wasn't clear enough. What I meant is that having a sleep() isn't required. But surely you must need some kind of blocking in your loop() if you don't want it to hog the CPU. This blocking can be wait() until a new camera frame is available for example.

hi Ytai

thanks again for the comments. To be honest, I wasn't
aware of the wait/notify mechanism in Java. After
working on this last weekend, I managed to migrate
the thread communication using wait/notify.

In the new code, after pin.write(), I set a delay
by Thread.sleep(X) to account for the latency, then
I synchronize a shared object to inform the camera
to collect images. I found that when the delay "X"
is greater than 250ms to 300ms, my images are
always sync with the LED status.

This doesn't make sense. The latency shouldn't be as high as that, even over Bluetooth. Something else is going on here.

 

However, for shorter
X, I can see the random intermediate states where
multiple LEDs are half-way on/off. I believe this is due
to the jitter and latency of BT-based communication.

In order to acquire data in faster speed, I would
like to set up the OpenAccessory mode.

You probably don't need that. ADB will give you 3-4ms latency. OpenAccessory will improve this to be about 1ms. Shouldn't matter.

 

In order to use OpenAccessory:

1. Your Android must support it. Not sure whether yours does.
2. Your IOIO has to be running app firmware V3.23.
3. Your IOIO should preferably be running bootloader V3.03. or otherwise you'd have to disconnect / reconnect USB after IOIO power-up in order to get the connection established.
4. Your USB debugging needs to be off.
5. You need to build your app in a way that supports it (see here: https://github.com/ytai/ioio/wiki/IOIO-Over-OpenAccessory).

Having that said - my suggestion is: get it working over ADB or BT first... Then see if you need to optimize.

I will definitely read more from your wiki, but here I
have a few quick questions and maybe you can help
me to confirm:

1. my IOIOs were purchased a month ago from
sparkfun, do I still need to upgrade the firmware
and bootloader?

Yes. The OpenAccessory version has not yet been officially released, thus SparkFun have not yet started using it for new boards.

 

2. I imported the libraries from App-IOIO0311.zip,
I saw IOIOLibAccessory in the project list, it appears
there is no error under my Eclipse+ADT14.0.0. Is
this good enough to run OpenAccessory mode?

App-IOIO0311 shouldn't have open-accessory. Are you sure???

 

If an upgrade is needed, I hope the project can still
be backward compatible with BT mode, as it is much
easier to debug.

Yes. The latest version supports ADB, Bluetooth and OpenAccessory and allow you to write applications that will seamlessly work with either one.

[Qianqian Fang]

On 04/03/2012 01:01 AM, Ytai Ben-Tsvi wrote:

This doesn't make sense. The latency shouldn't be as high as that, even over Bluetooth. Something else is going on here.

hi Ytai

here is my loop() function with some comments:

        @Override
        public void loop() throws ConnectionLostException {
            int i;
            // wait status change from Cam thread
            waituntil(CamStatus.Status.csRequestNext);
            ioio_.beginBatch();
            try{
                for(i=0;i<PIN_NUM;i++){ // change pin status
                     pins[i].write(pinstatus[statusid%STATUS_COUNT][i] != 0);
                }
                statusid=(statusid+1)%STATUS_COUNT; // cycle status
            }finally{
                ioio_.endBatch();
            }
            try {
                Thread.sleep(300); // set delay
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            // tell camera thread to take images
            camstatus.setStatus(CamStatus.Status.csInMotion);
        }
    }

camstatus is a shared object by ioio and camera
threads, the setStatus() method is synchronized and
issue a notify() if status is changed. When the camera
finishes capturing the current status, it will call

synchronized(camstatus){
    camstatus.setStatus(CamStatus.Status.csRequestNext);
}

to tell ioio to move on to the next pin status.

Does this look right?

You probably don't need that. ADB will give you 3-4ms latency. OpenAccessory will improve this to be about 1ms. Shouldn't matter.

if it is not caused by ioio communication latency, I guess
the only other possibility is the latency for the
onPreviewFrame() callback: i.e. although I am
getting 30 fps preview buffers, the images are delayed
versions from what is actually happening.

Qianqian

[Kong Kevin]

Does this camera project succeed? I'm interested in the same thing taking a video remotely and then sending back to the phone. May I know what type of camera are u using? Thanks!

thanks

Qianqian

 

To unsubscribe from this group, send email to ioio-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/ioio-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "ioio-users" group.
To post to this group, send email to ioio-...@googlegroups.com.

To unsubscribe from this group, send email to ioio-users+unsubscribe@googlegroups.com.

[Qianqian Fang]

On 04/03/2012 05:37 PM, Kong Kevin wrote:

Does this camera project succeed? I'm interested in the same thing taking a video remotely and then sending back to the phone. May I know what type of camera are u using? Thanks!

hi Kevin

I am not taking photos remotely. I used the phone camera, and took pictures of an IOIO-controlled circuit.

for what you want, I am sure you can find more than one tutorials online using an arduino+ethernet/cellular modules+camera module to make a remote security camera.

Qianqian

To view this discussion on the web visit https://groups.google.com/d/msg/ioio-users/-/d5gcw3Enxj8J.

To post to this group, send email to ioio-...@googlegroups.com.

To unsubscribe from this group, send email to ioio-users+...@googlegroups.com.

[Ytai Ben-Tsvi]

The code makes sense, but I'm not sure what you trying to achieve exactly.

Do you want to turn an LED on so that it turns on just when a frame gets taken? If so, using your approach, the LEDs will turn on AFTER the picture got taken, and you'll be synchronizing them with the next frame.

I don't think there should be much delay between the frame and the callback, since you can see e.g. in the camera app that the preview view is pretty low-latency from real life. You might as well put some System.nanoTime() logging into your code and see what's going on.

[Qianqian Fang]

On 04/03/2012 08:37 PM, Ytai Ben-Tsvi wrote:

The code makes sense, but I'm not sure what you trying to achieve exactly.

Do you want to turn an LED on so that it turns on just when a frame gets taken? If so, using your approach, the LEDs will turn on AFTER the picture got taken, and you'll be synchronizing them with the next frame.

no, I want to turn on an LED, wait and "make sure" it
is on, then take a picture, then next. The difficulty is the
"make sure" part. Currently, the only way for me to be
sure is to wait >250ms, which is not good enough. I was
trying to figure out if the IOIO or the camera is responsible.

I don't think there should be much delay between the frame and the callback, since you can see e.g. in the camera app that the preview view is pretty low-latency from real life.

It seems so, but the truth is that I don't know
if the preview buffer in the callback is indeed
synchronized with the image shown on the screen!
I did try mapping the preview buffer to a 2D
OpenGL texture and display it real-time in a
GLSurfaceView, I definitely notice a delay (1/5~1/4 s)
for the OpenGL layer compared to the camera
preview surface. But I don't know if this delay
is due to the processing time of the GL texture
mapping, or due to the arrival delay of the
preview buffer.

In your earlier email, you mentioned ADB latency
is ~4ms, and OpenAccessory is 1ms. How is ADB
related to Bluetooth? Sometimes, when I power on
IOIO and run the HelloIOIO example, I do see a
noticeable delay between pressing the button and
the LED turning on, does this tell me anything?

One last question, I did try connecting my cellphone
with IOIO by a USB cable, I was able to run the
HelloIOIO example and my code. But you were
certain that OpenAccessory was not on. Is there
another protocol that enables this communication?

Qianqian

[Ytai Ben-Tsvi]

On Tue, Apr 3, 2012 at 9:30 PM, Qianqian Fang <fan...@gmail.com> wrote:

On 04/03/2012 08:37 PM, Ytai Ben-Tsvi wrote:

The code makes sense, but I'm not sure what you trying to achieve exactly.

Do you want to turn an LED on so that it turns on just when a frame gets taken? If so, using your approach, the LEDs will turn on AFTER the picture got taken, and you'll be synchronizing them with the next frame.

no, I want to turn on an LED, wait and "make sure" it
is on, then take a picture, then next. The difficulty is the
"make sure" part. Currently, the only way for me to be
sure is to wait >250ms, which is not good enough. I was
trying to figure out if the IOIO or the camera is responsible.

You can wire your digital output pin to another pin open as digital input, then waitForValue() on it. It will double your latency, but should still be way smaller than the values you're talking about.

I haven't seen in your code the "take a picture code". Moreover, I'm not sure I understand yet what then you're doing in response to a preview frame.

 

I don't think there should be much delay between the frame and the callback, since you can see e.g. in the camera app that the preview view is pretty low-latency from real life.

It seems so, but the truth is that I don't know
if the preview buffer in the callback is indeed
synchronized with the image shown on the screen!
I did try mapping the preview buffer to a 2D
OpenGL texture and display it real-time in a
GLSurfaceView, I definitely notice a delay (1/5~1/4 s)
for the OpenGL layer compared to the camera
preview surface. But I don't know if this delay
is due to the processing time of the GL texture
mapping, or due to the arrival delay of the
preview buffer.

I don't know either. Can it perhaps be related to the preview buffer encoding / decoding? Have you tried different formats / resolutions?

 

In your earlier email, you mentioned ADB latency
is ~4ms, and OpenAccessory is 1ms. How is ADB
related to Bluetooth?

ADB is not related to Bluetooth. ADB works over USB. Bluetooth will have latency in the order of 20ms or so IIRC.

 

Sometimes, when I power on
IOIO and run the HelloIOIO example, I do see a
noticeable delay between pressing the button and
the LED turning on, does this tell me anything?

This is probably the time it takes to establish a connection to the IOIO. Does it happen also after the initial connection has been established?

 

One last question, I did try connecting my cellphone
with IOIO by a USB cable, I was able to run the
HelloIOIO example and my code. But you were
certain that OpenAccessory was not on. Is there
another protocol that enables this communication?

Yes. ADB.

[Qianqian Fang]

hi Ytai

I designed a simple test and estimated that the onPreviewFrame
latency is about 210ms. You can find my report here

http://groups.google.com/group/android-developers/browse_frm/thread/b7a2bf1ac96d6099/9b99de707383c4ba?hl=en&lnk=gst&q=qianqian+fang#9b99de707383c4ba

This roughly matches what I found earlier with the
working IOIO delay time. As a result, I belive that
it was the frame buffer callback that is responsible for
the asynchronousness.

On 04/04/2012 11:51 AM, Ytai Ben-Tsvi wrote:

ADB is not related to Bluetooth. ADB works over USB. Bluetooth will have latency in the order of 20ms or so IIRC.

I see.

This is probably the time it takes to establish a connection to the IOIO. Does it happen also after the initial connection has been established?

it happens mostly at initial connection.

 Yes. ADB.

thanks, that is helpful.

Qianqian

[Ytai Ben-Tsvi]

OK, so IIUC, all your IOIO questions are answered, right?

[Qianqian Fang]

On 04/05/2012 12:00 AM, Ytai Ben-Tsvi wrote:

OK, so IIUC, all your IOIO questions are answered, right?

yes, they were. I really appreciate your timely and
informative replies (needless to say, thanks for creating
IOIO in the 1st place). I am sure there will be new
questions coming up. Hopefully I can learn more from
you and the list down the path.

Qianqian