Research: Want to use PSMoveAPI on RaspPi 3 with 2+ webcams, 2 controllers, data out to iOS

[Justin Hawkwood]

As title states I am researching the feasibility of a project and have some questions regarding it. While I am a seasoned programmer, I am absolutely new to the Raspberry Pi.

1) does PSMoveAPI work using Raspberry Pi 3's included Bluetooth LE and USB?

2) is there a tutorial for installing it on RPi?

3) I am hoping to use 2 to 4 webcams to increase the FOV that the controllers are tracked in, is this supported in PSMoveAPI, or does each camera need to be switched to manually for processing?

3b) Alternately, I could just use 1 camera and a fisheye lens on it. My assumption is that the distance from the camera is calculated by the size of the PSMove light ball.  If this is the case, is there a way to adjust the calculation to take the fisheye in to account?

5) I am looking at this link as to how to do the communication between the iOS app and the RPi using Bluetooth LE, but some other sources, RetroPie, suggest that there would be issues trying to connect different device types to the RPi via Bluetooth LE.  Do you have any information on whether this would or wouldn't be an issue, or (not right group, I know) other suggested sites for the connecting the RPi to iOS (and eventually Android)?

Thank you for your assistance,

~ Justin

[Thomas Perl]

Hi,

2017-01-05 2:39 GMT+01:00 Justin Hawkwood <justinh...@gmail.com>:
> As title states I am researching the feasibility of a project and have some
> questions regarding it. While I am a seasoned programmer, I am absolutely
> new to the Raspberry Pi.
>
> 1) does PSMoveAPI work using Raspberry Pi 3's included Bluetooth LE and USB?

We recently got it working nicely on a Pocket C.H.I.P with USB and
Bluetooth working. Not sure which chipset the RPi3 uses. It looks like
some version of the RPi has problems with Bluetooth that can be worked
around (see https://github.com/Jonty/Oust#things-you-should-know).

> 2) is there a tutorial for installing it on RPi?

Basically you can follow the Linux tutorial here, this should work
just as well with Raspbian (if not, let me know and I can adjust the
instructions):

http://psmoveapi.readthedocs.io/en/latest/build.html#building-on-ubuntu-14-04

You might also be able to adapt this script:

http://psmoveapi.readthedocs.io/en/latest/build.html#building-for-the-pocket-c-h-i-p

> 3) I am hoping to use 2 to 4 webcams to increase the FOV that the
> controllers are tracked in, is this supported in PSMoveAPI, or does each
> camera need to be switched to manually for processing?

IIRC there was some talk about multi-camera support in PS Move
Service, although I don't know if this works on a RPi:

https://github.com/cboulay/PSMoveService

> 3b) Alternately, I could just use 1 camera and a fisheye lens on it. My
> assumption is that the distance from the camera is calculated by the size of
> the PSMove light ball. If this is the case, is there a way to adjust the
> calculation to take the fisheye in to account?

Yes, you can get the raw (x/y/radius) values and calculate the
distance yourself.

> 5) I am looking at this link as to how to do the communication between the
> iOS app and the RPi using Bluetooth LE, but some other sources, RetroPie,
> suggest that there would be issues trying to connect different device types
> to the RPi via Bluetooth LE. Do you have any information on whether this
> would or wouldn't be an issue, or (not right group, I know) other suggested
> sites for the connecting the RPi to iOS (and eventually Android)?

Possibly WIFI.

HTH :)
Thomas

[Chadwick Boulay]

> 3) I am hoping to use 2 to 4 webcams to increase the FOV that the
> controllers are tracked in, is this supported in PSMoveAPI, or does each
> camera need to be switched to manually for processing?

IIRC there was some talk about multi-camera support in PS Move
Service, although I don't know if this works on a RPi:

https://github.com/cboulay/PSMoveService

Yes, PSMoveService has multi-cam support.

But, PSMoveService does not currently build on Linux. We use cross-platform libraries so it should be possible, I just haven't spent the required time to add Linux support to our cmake scripts.

 

> 5) I am looking at this link as to how to do the communication between the
> iOS app and the RPi using Bluetooth LE, but some other sources, RetroPie,
> suggest that there would be issues trying to connect different device types
> to the RPi via Bluetooth LE.  Do you have any information on whether this
> would or wouldn't be an issue, or (not right group, I know) other suggested
> sites for the connecting the RPi to iOS (and eventually Android)?

Possibly WIFI.

Thomas, I never fully grok'd how psmoved works. Would it be possible to run a psmoved host/server on the RPi and then run a psmoveapi-wrapping client on iOS?

 

[Justin Hawkwood]

Thanks for the responses Thomas.  I am not fixed on it being a RPi, so I will look in to PocketCHIP to see if it can do the BluetoothLE to iOS portion.  Mostly I'm hoping for a headless/remote admin'd system once everything is set up (hence the mobile device connection). I can see some benefits (ease of implementation) and some drawbacks (tediousness of connecting if there are multiple setups on one network) of using Wifi instead of Bluetooth, I will have to research/consider further.

Alternative camera option: what changes would be needed in code (or is there already an option) if I used a singular camera (PS Eye) that was ceiling mounted pointing down instead of in front of the user?  

As mentioned above, there is a possibility of multiple setups in one space, does the tracking algorithm have error correction should a neighbors controller become briefly visible in your camera?  

[Justin Hawkwood]

I opted for the RPi 3 and after some finagling (manually installing some Linux and some PockeyCHIP libraries) I am able to successfully connect via Bluetooth and run some of the examples.  My current notable problems on the RPi are:

A) If it has been more than a minute or two, frequently the controller will disconnect/power off right after I start a program, like "./build/example_new_api", so I have to quite the program, reconnect the controller, then relaunch the program.

B) the exposure on the PSEye camera is really high to the point that it has difficulty seeing the controller in a semi lit room, and thus cannot track it. When I run "./build/test_tracker" I get a lot of "Calibrating controller 0... ERROR - retrying" and the ball blinks about 10 times, then just stops. When the camera view comes up, I can see the environment pretty clearly and there is a large circle in the upper left hand corner.  In comparison, if I run the same program on my Mac with either the iSight camera or the PSEye, the ball blinks only once or twice and I rarely get the error, and when the camera view comes up it is mostly black with only the ball showing, and is being tracked properly.

C) When I can get the tracker functioning, the OpenGL tests are (understandably) very slow and the placement of the wireframe is rarely in the correct place and correct size. Additionally the rotation drifts a lot. 

D) I used a combination of the tracker examples and the move-daemon to make my own UDP server, with the Unity client either running on my Mac or iPhone.  When the server is running on my Mac, data to the Mac client is excellent (because it's all local), and to the iOS client is pretty good, and I have rumble feedback working pretty well too.  When the server is running on the RPi, the controller will randomly rumble and within a few seconds the ball turns off and the rumbling feedback stops entirely, but the button data is still getting sent to the client.  I don't think the random rumbles are due to old packets as I have a basic level sequence checking to ignore old packets, but if the server is really lagging, then it might be possible.

E) On the positive side, I was also able to connect the controllers via Bluetooth and set the RPi to send BLE signals (piBeacon), so I think it might be possible to set it as a BLE server instead of a UDP server, which might alleviate issue D if it is due to dropped packets.

[Chadwick Boulay]

B) the exposure on the PSEye camera is really high to the point that it has difficulty seeing the controller in a semi lit room, and thus cannot track it. When I run "./build/test_tracker" I get a lot of "Calibrating controller 0... ERROR - retrying" and the ball blinks about 10 times, then just stops. When the camera view comes up, I can see the environment pretty clearly and there is a large circle in the upper left hand corner.  In comparison, if I run the same program on my Mac with either the iSight camera or the PSEye, the ball blinks only once or twice and I rarely get the error, and when the camera view comes up it is mostly black with only the ball showing, and is being tracked properly.

This might be related to this issue. You can try the solution I liked to or you can probably come up with a better auto-calibration algorithm on your own. I'm sure thp and the community would appreciate a pull request on that.

I've since given up on auto-calibration. In PSMoveService you have to calibrate manually.

 

Additionally the rotation drifts a lot. 

Did you calibrate the magnetometer?

[Justin Hawkwood]

I'll take a look at that algorithm, also because my next step with that is to switch to a different camera.

Yes, I did calibrate the magnetometer.

[Justin Hawkwood]

D) I used a combination of the tracker examples and the move-daemon to make my own UDP server, with the Unity client either running on my Mac or iPhone.  When the server is running on my Mac, data to the Mac client is excellent (because it's all local), and to the iOS client is pretty good, and I have rumble feedback working pretty well too.  When the server is running on the RPi, the controller will randomly rumble and within a few seconds the ball turns off and the rumbling feedback stops entirely, but the button data is still getting sent to the client.  I don't think the random rumbles are due to old packets as I have a basic level sequence checking to ignore old packets, but if the server is really lagging, then it might be possible.

On further testing, it looked like the server was sending trigger data, which in turn was being sent back (on purpose) as rumble data.

To test more, I added the following code to the while loop of test_tracker.c:

    while ((cvWaitKey(1) & 0xFF) != 27) {
        ...

        for (i=0; i<count; i++) {

            unsigned char trig = 0;

                while (psmove_poll(controllers[i])) {

                trig = psmove_get_trigger(controllers[i]);

                psmove_set_rumble(controllers[i], trig);

            }

            float x, y, r;

            psmove_tracker_get_position(tracker, controllers[i], &x, &y, &r);

            printf("x: %10.2f, y: %10.2f, r: %10.2f, t: %i\n", x, y, r, trig);
        }

    }

 

What I saw was that on RPi the controller was in fact sending false trigger data, and this was not happening when running on macOS.  If I commented out the psmove_set_rumble line, the false trigger data was still coming it.  Any ideas as to why the RPi and macOS version are behaving differently?

Interestingly, example_new_api.cpp did not seem to throw false trigger data, so it will take even more investigation to try to determine the difference.