Hawk Stereo Camera

[Suyay Escarsega]

FREMONT, Calif., Oct. 21, 2021 /PRNewswire/ -- Leopard Imaging Inc. (Leopard Imaging), a global leader in high-definition embedded camera design and manufacturing, today announced its new 3D Depth Hawk camera supporting the NVIDIA Jetson edge AI and Isaac robotics platforms.

In addition, the camera's high-speed GMSL2 interface enables reliable operation at distances up to 10m away from NVIDIA Jetson edge AI platform to further satisfy the challenging mechanical design needs of today's AI-enabled robots. The Hawk Camera provides accurate depth perception from 15cm to 5m and has the ability to detect 10cmx10cm objects with superior resolution. The camera solution works with NVIDIA Vision Programming Interface (VPI) algorithms and integrates to the NVIDIA Isaac platform to ensure optimum performance on the Jetson AGX Xavier and Jetson Xavier NX system on modules (SOM). The camera is supported out of the box by the camera and stereo depth Isaac ROS GEMs, enabling developers to quickly integrate in their products.

NVIDIA Jetson is the leading AI-at-the-edge computing platform with over 850,000 developers. With pretrained AI models, developer SDKs and support for cloud-native technologies across the full Jetson lineup, manufacturers of intelligent machines and AI developers can build and deploy high-quality, software-defined features on embedded and edge devices targeting robotics, AIoT, smart cities, healthcare, industrial applications, and more.

Leopard Imaging, an Elite Partner in the NVIDIA Jetson ecosystem, enhances the imaging quality with active alignment (AA) technology through its powerful manufacturing capabilities. Leopard Imaging's full integration services will help customers incorporate the Hawk Camera into a full range of robotics projects and deployments.

"By leveraging our full stack of GPU-accelerated computing technologies and enabling ease of deployment of cost-effective 3D camera solutions, Leopard Imaging is accelerating time-to-market of vision-based edge AI and robotics applications," said Murali Gopalakrishna, Head of Product Management, Autonomous Machines and General Manager for Robotics at NVIDIA.

SETUP
We have modified the Devicetree and created Kernel drivers to include two IMX264 image sensors which are each connected to a 4x CSI port on the Orin NX, respectively. The two image sensors are synchronized in hardware.

You can take a look at the Argus docs to consider writing your own driver. You might find it helpful to write your own ROS node that manages the two streams and publishes the image pairs yourself than going through Argus too.

Ok, this explains why ArgusStereoNode is not working. However, since we can run nvarguscamerasrc with sensor-id 0 and 1, I thought it would be possible to run two ArgusMonoNodes with the different sensor-ids and appropriate camera_info_urls for the calibration data?

A stereo camera driver that knows that the two imagers are part of the same camera, for example, could also provide stereo rectification parameters and extrinsics of the right imager relative to the left.

Can you let me know if our hardware setup (see picture below) would even allow us to modify the sensor kernel driver for stereo-compatability (non-monocular) so that we can run Isaac ROS ArgusStereoNode?

As a workaround I created some ROS nodes which publish [left right]/[image_raw camera_info] as you suggested. Then I launched two Isaac ROS rectify nodes + Isaac ROS Ess and Isaac ROS pointcloudnode like so:

For now I ended up creating a custom ROS node. The Node internally creates a Gstreamer Pipeline with two muxed nvarguscamerasrc elements and two instances of a custom Gstreamer plugin for rectification (based on OpenCV and VPI). The Rectification plugins read some calibration files which are created using stereoRectify from openCV. The ROS Node receives the rectified images via Gstreamer appsink callback and then publishes both rectified images and both calibration messages at once. Subsequently, this data is directly fed into the isaac_ros_ess node etc.

@hemals Are there any plans to support this configuration in isaac_ros_argus_camera? Namely hardware synchronized mipi csi cameras? It would be awesome if it could also support software synchronized cameras too so very low cost sensors like the imx219 could be used.

In machine vision applications, stereo cameras can be used to perform tasks such as object recognition, distance measurement, and 3D mapping. For example, a stereo camera system could be used to guide a robot arm to pick up an object or to measure the dimensions of a physical object for quality control purposes.

The Stereo Hawkeye was a folding stereoscopic camera for 118 size rollfilm, it takes 3 x 3 inch image pairs. It had previously been sold as the Stereo Weno until c. 1904. Stereo cameras have two or more lenses enabling them to take three-dimensional images.

W.r.t. the camera, realsense is still a solid option if you want a stereo camera as librealsense and realsense_ros is quite mature, but orbbec has A LOT of very appealing looking hardware options imo and offers stereo, structured light, and time of flight cameras to fit many applications.

In our experience mono AI depth looks visually impressive with great relative accuracy, however absolute depth accuracy is insufficient for planning functions, which is why we use stereo for AI depth.

Correct. I was able to resolve the issue, so this package works perfectly. This is probably the most valid way to use kinect v2 with ros2.The cause was that the rviz QoS settings were not set to best effort.

The r2b dataset 2023 is a collection of sequences stored in rosbags, a ROS 2 native format used for message data including time synchronized sensor captures from a robot. The dataset can be loaded and played back into an application running a ROS graph of nodes for testing and performance benchmarking of image processing, perception and localization. Each sequence consists of a single sequence in a scene, providing a small diversity of data across the collection of sequences in this dataset.

Several sequences contain sensor multi-second captures including LIDAR, stereo camera, and IMU data from HAWK stereo camera, RealSense D455, and XT32 with highly precise time synchronization. Other sequences contain data recorded from a simulated scene in Isaac Sim. The r2b_hope sequence is distinct as it contains D415 RGB data.

A MD5 checksum of each rosbag is provided to verify the dataset sequence used in testing or benchmarking matches, to detect inconsistencies in input data, and to provide a method to confirm that the same input dataset sequence is used when comparing independent results.

The Leopard Imaging LI-AR0234CS-STEREO-GMSL2-30 2D/3D stereoscopic Hawk Camera integrates two AR0234CS global shutter image sensors from onsemi with an active pixel array of 1920 1200 delivering 120 frames per second and offers a wide horizontal field of view of 120 degrees. The AR0234CS combines global shutter efficiency technology with optimized MTF and SNR, suitable for sensitive applications such as 3D imaging. Additionally, the 2D/3D camera also couples with a 6-axis IMU for highly accurate (sub-microsecond) timestamping of the camera frames relative to the IMU also needed highly accurate robotics perception. Also available with the same AR0234CS CMOS image sensor is the LI-AR0234CS-GMSL2-OWL 2D Owl robotic vision camera.

The kit contains a near-ready-to-fly carbon-fiber quadcopter equipped with a Pixhawk 4 or Pixhawk 6C (on V1.5) flight controller, a UP Core companion computer (4GB memory & 64GB eMMC), and a Occipital Structure Core depth camera sensor.

This vehicle comes with no pre-installed software. A USB stick is included in the kit with an example of an obstacle avoidance feature implementation, based on the PX4 Avoidance project. This example is intended as a reference only and serves to demonstrate the capabilities of the platform. The software is not compatible with the latest version of PX4, nor is it actively maintained or supported.

The guide explains the minimal additional setup required to get the vehicle ready to fly (installing an RC system and battery). It also covers the first flight, and how to get started with modifying the computer vision code.

The kit is intended for computer vision projects that use a forward-facing camera (it does not have downward or rear-facing depth cameras). Consequently it can't be used (without modification) for testing Safe Landing, or other features that require a downward-facing camera.

Obstacle avoidance in missions can only be tested when GPS is available (missions use GPS coordinates). Collision prevention can be tested in position mode provided there is a good position lock from either GPS or optical flow.

For PX4 Vision v1 with ECN below 010/carrier board below RC04, the UP Core should only be powered using the battery (do not remove the UP Core power socket safety cover). This does not apply to PX4 Vision v1.5

The vehicle should arrive pre-calibrated (e.g. with firmware, airframe, battery, and sensors all setup). You will however need to calibrate the radio system (that you just connected) and it is often worth re-doing the compass calibration.

To test collision prevention, enable Position Mode and fly manually towards an obstacle. The vehicle should slow down and then stop within 6m of the obstacle (the distance can be changed using the CP_DIST parameter).

To test obstacle avoidance, create a mission where the path is blocked by an obstacle. Then switch to Mission Mode to run the mission, and observe the vehicle moving around the obstacle and then returning to the planned course.

The PX4 Avoidance system consists of computer vision software running on a companion computer (with attached depth camera) that provides obstacle and/or route information to the PX4 flight stack running on a flight controller.

Documentation about the companion computer vision/planning software can be found on github here: PX4/PX4-Avoidance. The project provides a number of different planner implementations (packaged as ROS nodes):

c80f0f1006