SIgyn and the can_do

[Michael Wimble]



Just an update on progress towards the can-do challenge.

I’m not planning on exactly doing the challenge, and won’t be able to be there in person. I’m trying to do something more like the “fetch me a beer” challenge. I started by getting a prototype behavior tree working in simulation, and now I’m moving from simulation to the real robot. I’m getting pretty close with just a few modules and mechanical changes needed.

From the simulation, I just began at the start to get my action nodes to work with real components. Some I may just ignore for the moment, like the nodes that sense “fallen over”, “about to fall over”, “predicted time to battery discharge”. They are easy enough to implement, but I’m focused on “must work” modules first.

My latest stumbling block was the chain of logic to approach the can. My code looks up in a table for locations where the can might be, like inside the refrigerator, on a side table, on the kitchen table, etc. It picks the first location to explore and uses nav2 to get in that general area. Then it uses the OAK-D camera up 4 feet on the robot, looking down towards the front, to run a YOLO detector looking for the can. If not found, it will rotate up to 360 degrees until it sees it. Then it will orient (rotate) towards the can and try to drive so it’s about 10cm from the can. 

Then the Pi camera, mounted just behind and above the gripper, takes over. The robot was driving with the 2-axis gripper retracted and lowered to the body. Now the gripper arm is slowly raised until the Pi camera sees that the gripper is at the perfect height to grab the can and then centers the can in the gripper. 

All of this involves capturing training images, annotating them, and building a YOLO recognizer that will work on the OAK and Pi cameras, which have two very different model requirements. Getting it to fit on the OAK-D was hard because of the limited model choices. Getting it to fit on the Pi was hard because the AI Hat can only handle teeny, tiny models. But it was straightforward. That wasn’t the issue.

The issue was when I tell the robot to move, I need to:

• Make sure a movement is complete (e.g., rotate a few degrees).
• Make sure the OAK-D has captured a new RGB camera image after the movement. 
• Make sure the AI has processed that image, has found a can, and has selected one of the can detections.
• Make sure the stereo cameras have captured a new stereo image.
• Make sure the depth cloud has been produced.
• Compute the centroid of the bounding box from the AI.
• Find the centroid in the rectified depth cloud image.
• Compute the 3D pose of the can from the depth cloud.
• Transform the pose to the frame of the testicle twister gripper. 

Getting the synchronization was a bitch. Getting it to run at speed was bitchier. Once the pose was coming out, correcting the URDF for small error measurements was a pain as there wasn’t any easy way to put a tape measure to the components as there was always some weird blobby thing in the way of making a measurement. 

The current hurdle is stepper motor timing, which is affected by the behavior tree tick timing. 

It’s getting close. On Facebook, you can read about my descent into rage trying to talk to Gemini. 

[Michael Wimble]