Ethics statements

Experiments were performed in accordance with relevant guidelines and regulations, and were approved by the University Animal Welfare Committee (UAWC) of New York University under protocol number 13-1424.

Animals and housing

A total of 70 adult wild-type zebrafish were used in the experiment. The fish were purchased from an online aquarium retailer (LiveAquaria.com, Rhinelander, WI, USA), with a one to one nominal ratio in males and females. Fish measured approximately 3 cm in body length (BL) at the beginning of the experiment. They were housed in 37.8 l (10 gallon) water tanks for an acclimatization period of 15 days prior to the experiment, with a maximum density of 20 fish per tank. Water parameters were regularly checked, and temperature and pH were maintained at 26 °C and 7.2 pH, respectively. Animals were kept under a 12 h light/12 h dark photoperiod68 and fed once a day around 7 pm with commercial flake food (Hagen Corp. Nutrafin max, Mansfield, MA, USA). After acclimatization, a total of 10 fish were randomly selected as live stimuli, and separated from the others in a 10-gallon tank. The remaining 60 fish were used as focal subjects for the 60 trials comprising our study.

Apparatus

The experimental apparatus consisted of a glass water tank measuring 74 × 30 × 30 cm (length, width, and height), partitioned into three compartments using transparent acrylic panels (McMaster Corp., Waltham, MA, USA), as shown in Fig. 1. The two lateral compartments measured 10 × 30 × 30 cm (length, width, and height), and the water level was kept at 15 cm. The tank was placed within a holding frame of dimensions 120 × 120 × 20 cm (length, width, and height), made of aluminum T-slotted bars and acrylic plates.

Figure 1 Sketch of the experimental apparatus. The drawing shows the experimental tank, robotic platform, lightings, cameras, and holding frame. For clarity, the black curtain on the front of the frame is omitted and the focal fish and the robotic stimulus are magnified. Full size image

Two 25 W white fluorescent lights (All-glass aquarium, Franklin, WI, USA) were mounted at 78 cm from the water surface and orientated toward the tank to illuminate the apparatus. Two cameras (Logitech C920 webcam, Lausanne, Switzerland) were used to record videos of the focal fish and the stimulus at 30 frames per second with a resolution of 640 × 480 pixels. One camera was placed at the same height of the lights, pointing at the center of the water tank, to capture the horizontal motion of the zebrafish and the stimulus. The other camera was mounted on a tripod at a distance of 58 cm in front of the water tank and centred at the water level to capture their vertical motion.

The bottom walls of the tank and holding frame, the two short side walls of the tank, the lateral walls of the holding frame were all covered with white contact paper. To ease the tracking of the fish and stimulus from the frontal view, a sheet of white contact paper was used as background. This sheet was placed 25 cm away from the tank to reduce asymmetries in the perception of the environment by live fish. Black curtains surrounded the experimental apparatus to reduce external disturbance.

Robotic platform

An interactive 3D robotic platform (Fig. 2a) was designed to control the motion of a 3D-printed zebrafish replica. Similar to our previous work53, the platform was designed to actuate the replica in 3D, and execute small body oscillations at a frequency of 2 Hz, proxying the tail beating motion of a live fish. The platform frame was constructed from aluminium T-slotted bars (McMaster Corp., Waltham, MA, USA) and 3D-printed parts. The 3D-printed parts were designed using the computer-aided design software SolidWorks (Dassault Systems SolidWorks Corp., Waltham, MA, USA) and fabricated with a Dimension Elite 3D printer (Stratasys, Eden Prairie, MN, USA) in acrylonitrile butadiene styrene (ABS) material.

Figure 2 Elements of the robotics-based experimental approach. This composite picture presents: a general overview of the platform used for manoeuvring a 3D-printed zebrafish replica (a); detailed movements of the platform along the X-axis (b), Y-axis (c), and Z-axis (d); rotation of the replica (e); a live zebrafish (f); and biologically inspired 3D-printed replica of the zebrafish (g). Full size image

Two servo motors (HS-755HB, Hitec RCD, Poway, CA, USA), located at the top of the platform, were connected to a rack-and-pinions gear (Robotzone, Winfield, KS, USA) and used to actuate the platform along the X-axis, as shown in Fig. 2b. A DC motor with an encoder (Robotzone, Winfield, Kansas, USA), mounted to a rack-and-pinions gear (Robotzone, Winfield, KS, USA), was used to control the movement of the replica along the Y-axis (Fig. 2c). A stepper motor (NEMA-17, Adafruit, New York, NY, USA) was utilized to actuate the vertical motion of the replica by using a threaded rod (McMaster Carr, Elmhurst, Illinois, USA), which transformed rotational motion into translational movement along the Z-axis (Fig. 2d). Another stepper motor (NEMA-17, Adafruit, New York, NY, USA) was used to actuate the oscillations of the replica (Fig. 2e), and it was mounted on a cantilever traveling along the threaded rod.

Following our previous work53, the morphology and colour pattern of live zebrafish (Fig. 2f) served as the inspiration for the design of the replica (Fig. 2g). Specifically, the replica was designed in SolidWorks, and 3D-printed in yellow ABS thermoplastic using the Dimension Elite 3D. The replica measured 3 cm in length, and it was painted with non-toxic colours (Krylon, Krylon Products Group, Cleveland, OH, USA) to mimic the blue and silver striped pattern of live zebrafish. In addition, glass eyes (Van Dyke Supply Co., Granite Quarry, NC, USA) were attached to the replica to accurately mimic fish morphology.

The replica was connected to a 6.35 mm diameter acrylic rod (McMaster Carr, Elmhurst, IL, USA), which was fixed to a 24 mm diameter pulley printed in a transparent material (Objet Vero clear RGD 810, Stratasys, Eden Prairie, MN, USA) using a dedicated 3D printer (Connex500, Stratasys, Eden Prairie, MN, USA). Another pulley with the same diameter was connected to the stepper motor employed to elicit the oscillating motion. Two acrylic plates connected to the cantilever were used to hold the two pulleys. The two pulleys were coupled with a clear nylon line (McMaster Carr, Elmhurst, IL, USA), such that the rotational motion of the stepper motor could be transmitted to the rod. The nylon line was tightly coiled around both pulleys to prevent slippage. The platform was concealed using a black fabric curtain and white contact paper to match the colours of the background. The curtain was fixed to the platform using an aluminium support, and two acrylic rods were attached to the side of the experimental tank to hold the contact paper, as shown in Fig. 1.

The motion of the robotic platform was controlled by master and slave microcontrollers (Arduino Uno, Arduino, Italy), an Ethernet shield (Arduino, Italy), and two motor shields (Adafruit, New York, NY, USA). The two microcontrollers were required for simultaneously actuating the two stepper motors. The master microcontroller was utilized to: (i) receive the command for the desired position from the computer; (ii) control the position of the two servo motors (X-axis); (iii) adjust the voltage delivered to the DC motor with positional feedback from a magnetic rotary encoder (Y-axis); and (iv) regulate the position of the stepper motor regulating the oscillating motion of the replica. The slave microcontroller was commanded by the master microcontroller through serial communication to actuate the position of the stepper motor for its vertical motion (Z-axis).

The platform wirelessly received a command, every 0.5 s, via user datagram protocol (UDP) to actuate the motors based on real-time feedback from the 3D position of the focal fish, when closed-loop control was implemented, or on a pre-programmed trajectory. The workspace of the robotic platform, defined as the intended operational spatial limits, was smaller than the stimulus compartment of about 1 BL from each boundary, to avoid collisions of the replica on the walls. Technical details on the control of the robotic platform are available in the supplementary material.

Real-time tracking software

A computer vision-based real-time tracking software was developed using Visual Studio 2015 (Microsoft, Redmond, WA, USA). The software was designed to automatically track the 3D position of the focal zebrafish, and follow it throughout the trial at the full acquisition rate of 30 frames per second, while interactively matching the motion of the replica with the trajectory of the fish. The software was written in C++ language, and was based on the open source computer vision library OpenCV v3.169 (Intel Corp., Santa Clara, CA, USA). The software was programmed to simultaneously activate the two orthogonal cameras (Fig. 3) and read frames for the real-time tracking.

Figure 3 Schematics of the interactive robotic platform. The computerized base station for control feedback received an input (images) from the top and frontal camera. The images were processed and the fish was tracked in real-time to generate a feedback control signal that actuated the robotic platform. Full size image

The zebrafish tracking software was built on a blob (binary large object) detection algorithm69 and a series of computer vision filters to robustly detect the location of the focal fish or replica (Fig. 4a). Specifically, for each camera view, the acquired image frame was first converted into grayscale and then blurred with an averaging window of 7 × 7 pixels (corresponding to approximately 8 mm). Successive frames were then subtracted to identify potentially moving targets, observed as differences between the frames. To reduce noise from the image subtraction, a binary filter was utilized on the frame, with a threshold of 8 and 12 (on a 255 grayscale range) for the front and top views, respectively. These thresholds accounted for differences in the brightness and contrast from the two views. Pixels with intensities below the threshold indicated that there was no fish motion detected and were thus treated as part of the background, while pixels with intensities above the threshold were associated with a potentially moving target.

Figure 4 Real-time tracking software process. The picture shows the series of computer vision filters for acquiring the target (a), and a sample image of zebrafish tracking in the (b) top (left) and front (right) views. The fish trajectory corresponding to 100 consecutive frames is shown in yellow. Corners of the near and far side of the water tank are marked with green squares and circles, respectively. Full size image

Given the relatively large acquisition frequency compared with the typical speed of zebrafish, the difference between two frames typically resulted in the target being split into two disconnected regions for the head and tail (see Fig. 4a). To create a connected body, a dilation filter was applied, thereby increasing the size of each detected pixel into a 15 × 15 pixel square. A 7 × 7 pixel square eroding filter was then applied to resize the artificially dilated region. Next, to facilitate the identification of target shapes prior to the application of the blob detection algorithm, the software identified the contour of each object and generated a convex hull shape for it. A blob detection algorithm was then applied to locate the centre position, size, and aspect ratio of each object (blob) in the frame.

Although unintended blobs appeared intermittently (such as from surface ripples or reflections of the fish), a single blob of the target was identified in the majority of the images, and this was used to track the position of the target (focal fish or replica). To tackle the problem of unintended blobs and reliably track the target, we implemented three additional methods after 100 frames (or 3.3 s) from the initialization. First, when the tracking of the target was lost due to abnormal behaviour or freezing of the fish, the tracking software searched for other possible blobs within a circular region with a radius of 30 pixels. Second, if the target was not found within the circle, the radius of the search region was expanded by 1.5 pixels for successive frames, until the target was retrieved (and the radius was reset to 30 pixels). Meanwhile, a Kalman filter70 was implemented to anticipate the position of the target from the previously tracked positions. Finally, in the event that multiple blobs were detected in the search region, the new target was selected as the blob with the minimum distance from the last tracked position. During tracking, the system automatically marked a success for each frame if a blob was identified. The average success rate of the tracking software over a trial was 91.8% and 84.8% for the front and top views, respectively.

To compensate for the distortion associated with the perspective view from each camera, the 2D coordinates from the front and top views (Fig. 4b) were linearly interpolated based on known physical measurements of the water tank and water column and their 2D inferences71. Further details on the interpolation process are available in the supplementary material.

Experimental conditions

After the interpolation process to obtain the 3D coordinates of the focal fish, the motors of the robotic platform were independently actuated on the X-, Y-, and Z-axes, in either an open-loop (OL) or closed-loop (CL) control (see Supplementary Videos S1 and S2, respectively), to manoeuvre the replica to a desired position within the 3D space. We implemented independent OL or CL control on each axis. In OL control, the robotic platform was actuated in a predetermined trajectory based on observations of a live stimulus fish, without any feedback from the focal fish. In CL control, feedback from the focal fish was utilized in the form of real-time spatial information.

To acquire realistic trajectories for use in OL control, we conducted experiments on the interaction between a live stimulus, placed in a lateral compartment, and the focal fish in the centre (see Supplementary Video S3). Not only was this condition (2-Fish) used for informing the control design, but also it served as a baseline on which to test the efficacy of the robotic stimulus on the focal subject. To mitigate potential confounds associated with the robotic platform, we sought to create an equivalent acoustic signature and visual background to those generated by the motion of the robotic platform. Thus, the robotic platform was placed in the same lateral compartment of the live stimulus and actuated along a 3D trajectory utilized in our previous work53. To avoid physical contact with the live stimulus, a shorter rod, without the replica, was used. The swimming pattern of the stimulus fish was tracked and used as the basis for predetermined trajectories in OL control.

In OL control, the robotic platform actuated the replica in the predetermined trajectories acquired from condition 2-Fish. Based on the camera frame rate and the rate of commands sent to the platform, the trajectory of the stimulus fish was downsampled for use in OL control. Specifically, since the observation of the stimulus fish was recorded at 30 frames per second, and position data was sent twice per second, we downsampled the data by a factor of 15. The actuation along each axis was individually implemented to afford the composition between OL and CL controls, such that the motion along an axis could be from the 2-Fish condition while closed loop would be created on another axis.

In CL control, the position of the focal fish, obtained through the real-time tracking software, was utilized as feedback for actuating the replica to follow the focal subject. Specifically, in CL control, we sought to minimize the distance between the focal fish and the replica, with respect to any of the X-, Y-, and Z-axes. Minimizing the distance along the X-axis caused the CL control to position the replica to the limit of the workspace, in front of the transparent partition. With respect to either the Y- or Z-axis, instead, the distance between the focal fish and the replica was minimized by dynamically actuating the replica toward the Y- or Z- coordinate of the focal fish, respectively. The signal to the motor was generated so that the position of the replica was a scaled down version of the focal fish with respect to the Y-axis. Specifically, the geometric centre was placed in the middle of the width of the compartment and the desired position was linearly scaled to match the workspace of the platform. Closing the loop in the Z-axis was analogous to the Y-axis closed-loop, with a different scaling factor.

While the motion along the Z-axis was controlled through a stepper motor that enabled fine position control, an overshoot on the order of 1 body length could be generated by the DC motor for the Y-axis control. This overshoot was estimated from the target refresh rate of 0.5 s and the maximum speed of 10.4 cm/s from technical specifications of the DC motor. Typical results of the focal fish and replica positions in OL and CL controls in the X-, Y-, and Z-axes are illustrated in (Fig. 5a and b). Sporadically, slippage between the Y-axis motor and its holding case of about 1 mm (clockwise or counter-clockwise) occurred during the trial with a frequency of few events per minute. Finally, the translation along the Y-axis might have caused the replica to potentially move backwards few times per trial, for short bouts of approximately one second in duration.

Figure 5 Sample trajectories of focal fish and replica. The graphs illustrate two-minute sample trajectory of the focal fish and the replica along each of the axes in XYZ-OL condition (a, open-loop) and XYZ-CL condition (b, closed-loop). The trajectory of the focal fish is shown as a dashed-grey curve, and the trajectory of the replica is depicted as a solid-black curve. For X, Y, and Z, respectively, the root mean square errors computed over the entire trial are: 34.85, 11.29, and 6.5 cm (a); and 34.78, 4.72, and 3.36 cm (b). Full size image

The OL and CL control strategies were utilized to generate five conditions, namely, full open-loop in all three axes (XYZ-OL), closed-loop along the X-axis (X-CL), closed-loop along the Y-axis (Y-CL), closed-loop along the Z-axis (Z-CL), and closed-loop along all three axes (XYZ-CL). In the XYZ-OL condition, the replica was controlled in open-loop long all three axes. We refer to these five conditions as “replica” conditions, in contrast with condition 2-Fish that used a live stimulus.

Specifically, in the X-CL condition, the replica was actuated in CL control along the X-axis while the motion along the Y and Z-axes was controlled in OL. Similarly, in the Y-CL and Z-CL conditions, the replica was actuated in CL along the Y- and Z-axes, respectively, while the movements along the two remaining axes were controlled in OL. In the XYZ-CL condition, the replica was actuated in CL along all three axes. The control strategies for the five replica conditions are shown in Table 1.

Table 1 Control strategies for the robotic platform. Full size table

Experimental procedure

The experiment was conducted from November to December 2016. Immediately after transferring the focal fish to the central compartment of the experimental tank using a hand net, the real-time tracking software was started, and the robotic platform was actuated. Video of the trials were recorded using the real-time tracking software for 20 min, which includes 10 min for habituation and 10 min for observation. Each focal fish and live stimulus was used only once.

Up to ten trials were performed per day, for a maximum of five trials in the morning from 9:00 to 1:30 PM, and five in the afternoon from 2:00 to 6:30 PM. Each condition consisted of 10 trials, in which the stimulus alternated between the left and right lateral compartments between trials. Further, the order of the replica conditions was randomized with an even distribution of trials in the left and right lateral compartments. In each trial, the replica was initially positioned in the lower-left midpoint of the respective lateral compartment.

The 2-Fish condition was conducted before the replica conditions. The trajectory of the live stimulus was acquired from the stimulus of the 2-Fish condition to generate predetermined trajectories. The order of the trajectories used in any of the replica conditions implementing OL control (XYZ-OL, X-CL, Y-CL, and Z-CL) was identical to the 2-Fish condition. That is, the first trial conducted for XYZ-OL, X-CL, Y-CL, Z-CL conditions utilized the predetermined trajectory generated from the first trial of the 2-Fish, and so on.

Data analysis

Prior to data analysis, the trajectories of the fish and replica were repaired with an ad hoc script to correct for tracking errors in the trials. Specifically, errors were identified by examining when the speed of the focal fish or the replica was higher than a threshold value. The threshold value for focal fish speed was set at approximately three times its average speed (20 cm/s in all axes). Similarly, the threshold value of the replica speed was chosen based on estimated maximum speeds calculated from technical specs (X-axis: 5 cm/s, Y-axis: 13 cm/s, and Z-axis: 6 cm/s).

In the event of an error, the ad hoc script was allowed to repair the trajectories of a target (either the focal fish or the replica) by facilitating its manual tracking. Specifically, the script slowed down the playback of the trial frames, down to 10 frames per second, such that the experimenter could manually control the mouse pointer to hover it over the target in the frame and replace the erroneous coordinates. After correcting the errors, the trajectories of the focal fish and the stimuli were filtered using the Daubechies-wavelet filter72 to reduce the noise and smoothen the trajectories (Fig. 5a and b).

We investigated the behavioural response of the focal fish, including spatial preference, shoaling tendency, and activity. With respect to the spatial preference of the focal fish, we computed the time budgeting along the water column by virtually dividing the water column into two equal partitions of 7.5 cm and counting the time the fish spent in the top and the bottom of the tank. Shoaling tendency was evaluated as the time spent by the fish within four body lengths from the stimulus73. With respect to the activity of the focal fish, we evaluated the time spent freezing, computed as the percentage of time in which the fish was not moving in a circle of 2 cm in radius during 2 s7,32. Further analysis on the activity of the focal fish and of the stimuli is available in the supplementary material.

Transfer entropy64, an information theoretic quantity to infer causal relationships between processes, was used for quantifying the interactions between the fish and the stimulus. Let Q and R be two discrete stationary stochastic processes, transfer entropy from R to Q is defined as

$${{\rm{T}}{\rm{E}}}_{R\to Q}=\sum _{{q}_{t+1},{q}_{t},{r}_{t}}p({q}_{{\rm{t}}+1},\,{q}_{t},\,{r}_{t}){{\rm{l}}{\rm{o}}{\rm{g}}}_{2}\frac{p({q}_{{\rm{t}}+1}|{q}_{t},\,{r}_{t})}{p({q}_{{\rm{t}}+1}|{q}_{t})},$$ (1)

where q t and q t+1 denote the values of Q at time t and at time t + 1, respectively; r t and r t+1 the values of R at time t and time t + 1; p(q t+1 |q t , r t ) is the conditional probability of Q at t + 1, given its previous state and that of R; and Σ represents summation over all possible realizations q t , q t+1 , and r t . Transfer entropy is a non-symmetric quantity that measures the reduction in the uncertainty of Q given R. Net transfer entropy, measured as the difference between transfer entropy from R to Q and transfer entropy from Q to R, quantifies the extent of asymmetry in the information flow between Q and R.

As demonstrated in our previous work65, this quantity could be used to infer cause-and-effect relationships in behaviour, such that a positive value of net transfer entropy could be interpreted as Q influencing R, and vice versa, a negative value of net transfer entropy would identify an influence of R on Q. We computed transfer entropy between the focal fish and the stimulus in each trial for all experimental conditions using the raw positions along the Y- and Z-axis. The position in the Y-axis was discretized into 10 bins, such that a bin would approximately match 1 BL. The position in the Z-axis was discretized into 5 bins, to match the bin size along the Y-axis. We evaluated net transfer entropy as the difference between the transfer entropy from the fish to the stimulus and that from the stimulus to the fish. Transfer entropy was computed using Process_Network_v1.474 software.

For each of the experimental conditions, a one-tail one-sample t-test with reference mean at 50% was used to assess whether the fraction of time spent by the focal fish at the bottom of the tank was higher than chance. A one-tail one-sample t-test with reference mean set to zero was also performed on net transfer entropy for each condition to study cause-and-effect relationships. One-way ANOVA was used to compare the observables (time spent in the bottom of the tank, shoaling tendency, time spent freezing, transfer entropy from focal fish to replica, and transfer entropy from replica to focal fish) with the replica conditions as the independent variable. Upon significance of the one-way ANOVA, Tukey’s honest significant difference (HSD) post-hoc comparisons were conducted. To assess whether the appraisal of the robotic stimulus and the live counterpart by the zebrafish was comparable, we confronted the response of the focal fish in the condition 2-Fish with respect to all the replica conditions using a one-tail two-sample t-tests assuming equal variances of all the parameters. All the analyses were conducted with p < 0.05, except for the pairwise comparison, in which the statistical significance was determined based on a corrected p-value, which was set to 0.01 based on Bonferroni correction75.

Data and materials availability

Datasets and codes used in the analyses are stored at the authors’ home institution and will be provided on request.