Data Preparation

[iouri...@gmail.com]

Our Android application collect the data based on the Android events. Therefore we can catch the event from some IMU sensor (for example, from gyroscope or accelerometer), put the time stamp and write data to file.
The results looks something like this:

5160004;TYPE_ACCELEROMETER;6.1498566;4.6300507;5.636917
5160005;TYPE_ACCELEROMETER;6.1546173;4.7133636;5.4988556
5160007;TYPE_GYROSCOPE;0.14741516;-0.64793396;0.054626465
5160011;TYPE_ACCELEROMETER;6.23555;4.722885;5.603592
5160017;TYPE_GYROSCOPE;0.17678833;-0.6965332;0.04234314
5160019;TYPE_ACCELEROMETER;6.273636;4.832382;5.6916656
5160019;TYPE_GYROSCOPE;0.19708252;-0.7392578;0.029525757
5160021;TYPE_ACCELEROMETER;6.987747;3.8968964;5.2227325
5160021;TYPE_ACCELEROMETER;7.0282135;3.9207;5.258438
5160022;TYPE_GYROSCOPE;0.26651;0.12698364;0.22392273

As you can see these events are asynchronous, and we can't predict what'll come next, and can't directly combine them together, because of the different time stamps.
In your SDK the .bag file require the data to be prepared like this:

timestamp,omega_x,omega_y,omega_z,alpha_x,alpha_y,alpha_z
1385030208736607488,0.5,-0.2,-0.1,8.1,-1.9,-3.3
...
1386030208736607488,0.5,-0.1,-0.1,8.1,-1.9,-3.3

The question is: how better to merge our data to prepare them for the SDK. Is it Ok to merge them together with some kind of interpolation, or it might break SDK calculations.
The 2nd question is: Should the the time stamp be in nanosec and in UNIX absolute
time format, or it can be different (for example, see the timestamp in the example
above)
Thank you

[Paul Furgale]

Hi Iouri,

Interesting problem. It is okay to have different timestamps for the gyroscope and accelerometer measurements in the actual estimator. However, we are really tied to the ROS Imu message so we expect the gyro and accelerometer to be synchronized in the interface.

I just looked through the interface code and there is no easy way to separate this or to flag messages as "accelerometer only". 

Your options are probably the following:

1. Use some kind of interpolation for preprocessed measurements. 

2. Build your bag file with some kind of "magic value" that specifies that a gyro or accel measurement is bad (say [-1e10, -1e10, -1e10]). Then hack in to the kalibr source code, find these values with an if statement, and don't add the measurements (gyro source, accel source) 

The timestamps should be in nanoseconds. It doesn't matter if it is POSIX time.

Does your device have a rolling shutter camera?

[Paul Furgale]

No code hacking from my side. I'd better to interpolate the data. But may be in the SDK you could add some invalid
data marker and skip, for example, 1 type of sensor (with invalid data) and keep data from another sensor with valid numbers.
I'm not sure if our camera is the rolling shatter type. It's very low resolution RGB 640x480.

The best thing to do would be to separate the gyro and accelerometer but I'm not sure when we will have time to do this.

Let us know how it goes. If it is a rolling shutter camera, things might not work that well.

[iouri...@gmail.com]

Hi Paul,

I've tried to convert our files with the bagcreator utility, and got this errors:

Traceback (most recent call last):
File "/var/kalibr-build/devel/bin/kalibr_bagcreater", line 5, in <module>
exec(fh.read())
File "<string>", line 110, in <module>
File "<string>", line 73, in loadImageToRosMsg
ValueError: invalid literal for int() with base 10: ''
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 64, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
ImportError: No module named apport.fileutils

Original exception was:
Traceback (most recent call last):
File "/var/kalibr-build/devel/bin/kalibr_bagcreater", line 5, in <module>
exec(fh.read())
File "<string>", line 110, in <module>
File "<string>", line 73, in loadImageToRosMsg
ValueError: invalid literal for int() with base 10: ''

The files where prepared like this:
-cam0
|- 414423.png
|- 414438.png
...
|- 417228.png
-imu0.csv

the imu0.csv has the following format:

timestamp,omega_x,omega_y,omega_z,alpha_x,alpha_y,alpha_z
415575,-0.204351,-0.0873515,-0.387217,7.71489,2.09307,5.68067,
415579,-0.183167,-0.0983887,-0.391312,7.75074,2.10571,5.62694,
415581,-0.174622,-0.0987447,-0.395584,7.7723,2.11792,5.59251,
....

The question is: Does your utility sensitive to the time stamp format?
Some devices (vendor specific) provide system timestamps not in the absolute
time. Or the reason for failure could be somewhere else?

Thank you

[iouri...@gmail.com]

It looks like I've found the problem why bagcreator doesn't work.
All images have to be in the grayscale format. It doesn't work with rgb images

[iouri...@gmail.com]

And it looks like it also sensitive to the time stamp format

[Paul Furgale]

What is the error on the time stamp format?

[iouri...@gmail.com]

In your example the time is absolute, something like:
1402065930450759000
In our case the time stamps have numbers like:
415574

In this case the utility is failing.
When I artificially scale our timestamp to the absolute time (19 digits), the utility is working and creates .bag file

And the 2nd issue, the utility doesn't work with RGB images (has to be converted to grayscale)

[iouri...@gmail.com]

I've managed to get the calibration results, which are look like:

Calibration results
===================
Reprojection error squarred (cam0): mean 1.32204429049, median 0.693250189377, std: 2.08354871694
Gyro error squarred (imu0): mean 13.0906885399, median 5.11788296825, std: 23.1408840571
Accelerometer error squarred (imu0): mean 24.5099671798, median 6.94171501898, std: 77.0610019325

Transformation (cam0):
-----------------------
T_ci: (imu to cam0): [m]
[[ 0.99008744 0.13056635 0.05176192 0.07960034]
[-0.12470078 0.64760111 0.75170641 0.73479587]
[ 0.06462649 -0.75070982 0.65746344 -1.03456271]
[ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu): [m]
[[ 0.99008744 -0.12470078 0.06462649 0.07967847]
[ 0.13056635 0.64760111 -0.75070982 -1.26290413]
[ 0.05176192 0.75170641 0.65746344 0.12371614]
[ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
0.0

Gravity vector in target coords: : [m/s^2]
[-1.61853573 4.06865922 -8.77852233]

The translation parameters in the matrices are too big, non-real.
The timeshift cam0 to imu0 is always 0 (I've tried different data sets)

Any advise, why the translation is so big?

[iouri...@gmail.com]

Hi,
I've found that our IMU data have different x,y,z coordinate system then yours
omega_x omega_y omega_z alpha_x alpha_y alpha_z
For example, for accelerometer you've got:
9.733100125,-0.62108783333333339,-0.46581587499999999
and our numbers:
0.068534,10.0881,1.67145

Probably in the gyro case we've got the similar situation.

The question is, how it'll affect the SDK results?

Thank you
Iouri

[Paul Furgale]

HI Iouri,

My guess is that there are a few things all going wrong together that stop you from having a good time with the toolkit.

1. The big one would be that I bet this device uses a rolling-shutter camera. The toolbox doesn't handle this case and the unmodeled effects of the rolling shutter will turn into calibration errors
2. The inaccurate timing of the IMU measurements will have a similar, but less pronounced effect.
3. The toolkit needs good models for the IMU noise parameters. There is a good introduction here: https://github.com/ethz-asl/kalibr/wiki/IMU-Noise-Model-and-Intrinsics

All of those things taken together mean that the calibration will be inaccurate. It may be possible to tune the noise models to get a better result. 

When I look at these values: 

Reprojection error squarred (cam0):  mean 1.32204429049, median 0.693250189377, std: 2.08354871694
Gyro error squarred (imu0):          mean 13.0906885399, median 5.11788296825, std: 23.1408840571
Accelerometer error squarred (imu0): mean 24.5099671798, median 6.94171501898, std: 77.0610019325

I can see that the nose models are not good. The expected mean value for each should be 2, 3, and 3 (reprojection error, gyro, accelerometer). It is not straightforward to tune the noise models to get these correct. Again, the best method we know how is this wiki page.

Okay, I hope this helps!

Paul

[Paul Furgale]

This shouldn't be a problem. The toolkit uses this data to make a good initial guess for the orientation of the IMU.