Dassault Stimulus

[Nikita Desjardins]

Withdomestic air travel in India resuming from the 25th of May, 2020, there is a clear opportunity to bolster business and offset any challenges arising from other areas. Two things are critical to achieving this:

Looking at the stimulus package, it is clear that the Indian government recognizes the integral role of aerospace and defense in a post-COVID-19 economy. To maximize the possibilities, it is essential for the industry to embrace modernization and leverage the latest technology to maintain continuity.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

In some of these closed-loop control systems, computer vision was utilized to measure animal behaviour in real-time and sustain a feedback loop between animals and robots. For instance, computer vision techniques were used to modulate the tail-beat frequency of a robotic fish with respect to the position of a zebrafish32; track the centre of a shoal of guppy fish (Poecilia reticulata) to offer feedback to a replica actuated by a mobile robot34; and acquire the position of a school of golden shiner to create feedback for a robotic predator31.

In this study, we propose a first, interactive robotics-based platform to study the behaviour of zebrafish (Danio rerio, Hamilton, order Cypriniformes, family Cyprinidae) in three dimensions (3D). Zebrafish, our target species, is a freshwater fish which is recently emerging as a species of choice for hypothesis-driven studies in several research fields, such as developmental genetics44, translational neuroscience45, and toxicology46. This animal model presents several unique features, such as a high level of sociality45,47, elevated number of genetic and neural homologies with humans48,49, and ease of husbandry and maintenance50. Zebrafish has a uniquely complex repertoire of 3D swimming patterns51, including fast horizontal and vertical movements, in a so-called burst-and-coast style52. Over the years, we have demonstrated the feasibility of using robotic stimuli in zebrafish research to study the determinants of social behaviour7,53,54,55 and fear response56,57, along with the effect of pharmacological manipulations57,58,59. The potential of robotics to support zebrafish research in the study of complex brain disorders has already been acknowledged by a number of researchers60,61,62.

The platform is grounded in recent advancements by our group on the design of biologically-inspired robotic stimuli for the study of fish behaviour in open-loop experiments53,54. We integrate 3D printing and real-time computer vision tracking to establish a closed-loop control system, in which a 3D-printed replica of a live fish is manoeuvred in 3D by a robotic arm, based on real-time measurements of fish position. The platform allows for actuating the replica through four independent degrees of freedom, including 3D translational movements and single axis rotational motion to proxy fish tail-beating.

In contrast with our previous efforts53,54 where predetermined trajectories were implemented based on independent observations of live fish motion, a custom-made tracking software was developed to allow for real-time tracking of the position of the focal subjects in the experimental tank. Differently from existing studies using 2D position tracking of focal subject to sustain closed-loop control31,32,34, this system affords, for the first time, real-time tracking and closed-loop control in 3D. The effectiveness of the new robotic platform was tested on zebrafish in a set of binary choice experiments, in which fish were systematically presented to a biologically-inspired replica with different levels of interactivity (ranging from full open-loop to closed-loop control in 3D). While the platform enables the implementation of a wide range of control schemes, we focused on a feedback loop in which the replica would seek to mirror the behaviour of the focal subject, by following some of its movements.

We hypothesized that the behavioural response of the focal subject would vary as a function of the degree of the interactivity of the replica. Specifically, we predicted an increased preference, measured as time spent close to the replica, for closed-loop controlled systems, where the replica would respond to the focal subject and mirror its behaviour. This increased preference was expected to be associated with a higher degree of biological mimicry, thereby accompanied by the absence of any stress-related responses, measured in the form of freezing or geotaxis52,63.

To detail the interaction between the replicas and live fish, we applied the information-theoretic framework of transfer entropy64. Transfer entropy measures the information flow between two coupled dynamical systems from their raw time series, offering a quantitative insight into potential cause-and-effect relationships between them. This approach has been demonstrated in several other efforts to evaluate study cause-and-effect relationships between animals and robots65, leadership-followership relations in pairs of bats66, and predator-prey interactions67, offering a model-free data-driven perspective on behavioural interactions.

Within this information-theoretic approach, we predicted that transfer entropy would help in detecting the cause-and-effect relationships underlying the behaviour of the replica and quantifying the role of the feedback from the replica. Specifically, we expected that transfer entropy would detect a net information flow from the replica to the focal subject in open-loop control, where the replica was unresponsive to the motion of the live fish. Instead, in closed-loop control, we anticipated a net information flow from the focal subject to the replica, whose movement was actuated in real-time to follow the live fish. Finally, we predicted that closed-loop control would create a positive feedback mechanism in the behaviour of the live subject, leading to an increased information flow from the live fish to the replica, and vice versa.

Experiments were performed in accordance with relevant guidelines and regulations, and were approved by the University Animal Welfare Committee (UAWC) of New York University under protocol number 13-1424.

Sketch of the experimental apparatus. The drawing shows the experimental tank, robotic platform, lightings, cameras, and holding frame. For clarity, the black curtain on the front of the frame is omitted and the focal fish and the robotic stimulus are magnified.

The motion of the robotic platform was controlled by master and slave microcontrollers (Arduino Uno, Arduino, Italy), an Ethernet shield (Arduino, Italy), and two motor shields (Adafruit, New York, NY, USA). The two microcontrollers were required for simultaneously actuating the two stepper motors. The master microcontroller was utilized to: (i) receive the command for the desired position from the computer; (ii) control the position of the two servo motors (X-axis); (iii) adjust the voltage delivered to the DC motor with positional feedback from a magnetic rotary encoder (Y-axis); and (iv) regulate the position of the stepper motor regulating the oscillating motion of the replica. The slave microcontroller was commanded by the master microcontroller through serial communication to actuate the position of the stepper motor for its vertical motion (Z-axis).

A computer vision-based real-time tracking software was developed using Visual Studio 2015 (Microsoft, Redmond, WA, USA). The software was designed to automatically track the 3D position of the focal zebrafish, and follow it throughout the trial at the full acquisition rate of 30 frames per second, while interactively matching the motion of the replica with the trajectory of the fish. The software was written in C++ language, and was based on the open source computer vision library OpenCV v3.169 (Intel Corp., Santa Clara, CA, USA). The software was programmed to simultaneously activate the two orthogonal cameras (Fig. 3) and read frames for the real-time tracking.

Schematics of the interactive robotic platform. The computerized base station for control feedback received an input (images) from the top and frontal camera. The images were processed and the fish was tracked in real-time to generate a feedback control signal that actuated the robotic platform.

Real-time tracking software process. The picture shows the series of computer vision filters for acquiring the target (a), and a sample image of zebrafish tracking in the (b) top (left) and front (right) views. The fish trajectory corresponding to 100 consecutive frames is shown in yellow. Corners of the near and far side of the water tank are marked with green squares and circles, respectively.

To compensate for the distortion associated with the perspective view from each camera, the 2D coordinates from the front and top views (Fig. 4b) were linearly interpolated based on known physical measurements of the water tank and water column and their 2D inferences71. Further details on the interpolation process are available in the supplementary material.

After the interpolation process to obtain the 3D coordinates of the focal fish, the motors of the robotic platform were independently actuated on the X-, Y-, and Z-axes, in either an open-loop (OL) or closed-loop (CL) control (see Supplementary Videos S1 and S2, respectively), to manoeuvre the replica to a desired position within the 3D space. We implemented independent OL or CL control on each axis. In OL control, the robotic platform was actuated in a predetermined trajectory based on observations of a live stimulus fish, without any feedback from the focal fish. In CL control, feedback from the focal fish was utilized in the form of real-time spatial information.

3a8082e126