Abstract Immersive virtual reality (IVR) typically generates the illusion in participants that they are in the displayed virtual scene where they can experience and interact in events as if they were really happening. Teleoperator (TO) systems place people at a remote physical destination embodied as a robotic device, and where typically participants have the sensation of being at the destination, with the ability to interact with entities there. In this paper, we show how to combine IVR and TO to allow a new class of application. The participant in the IVR is represented in the destination by a physical robot (TO) and simultaneously the remote place and entities within it are represented to the participant in the IVR. Hence, the IVR participant has a normal virtual reality experience, but where his or her actions and behaviour control the remote robot and can therefore have physical consequences. Here, we show how such a system can be deployed to allow a human and a rat to operate together, but the human interacting with the rat on a human scale, and the rat interacting with the human on the rat scale. The human is represented in a rat arena by a small robot that is slaved to the human’s movements, whereas the tracked rat is represented to the human in the virtual reality by a humanoid avatar. We describe the system and also a study that was designed to test whether humans can successfully play a game with the rat. The results show that the system functioned well and that the humans were able to interact with the rat to fulfil the tasks of the game. This system opens up the possibility of new applications in the life sciences involving participant observation of and interaction with animals but at human scale.

Citation: Normand J-M, Sanchez-Vives MV, Waechter C, Giannopoulos E, Grosswindhager B, Spanlang B, et al. (2012) Beaming into the Rat World: Enabling Real-Time Interaction between Rat and Human Each at Their Own Scale. PLoS ONE 7(10): e48331. https://doi.org/10.1371/journal.pone.0048331 Editor: Gonzalo G. de Polavieja, Cajal Institute, Consejo Superior de Investigaciones Científicas, Spain Received: June 15, 2012; Accepted: September 24, 2012; Published: October 31, 2012 Copyright: © 2012 Normand et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: This study was funded by the European Commission through the European Union projects PRESENCCIA FP6-027731, IMMERSENCE FP6-027141 BEAMING FP7-248620, MicroNanoTeleHaptics (ERC 247401) and TRAVERSE (ERC 227985). European FP6 and FP7 projects' URL is http://cordis.europa.eu/home_en.html and the European Research Council's is http://erc.europa.eu/. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors in the paper who are employed by the company Guger Technologies are Bernhard Grosswindhager and Christoph Guger. The main business of that company is brain-computer interfaces (www.gtec.at). In the work described in this paper, these authors were responsible for implementing the robot controller. Dr Guger, the director of Guger Technologies, has sent the corresponding author an email stating that there is “no conflict of interest with the publication as it was done for research purposes.” There is a small commercial relationship between Guger Technologies and the University of Barcelona (UB). UB licenses to Guger Technologies a system that controls a virtual character that can be moved by the company’s brain-computer interface system. This has nothing to do with the work described in the present paper. Taking into account all of the above this does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.

Introduction The potential for immersive virtual reality remains largely untapped, and although the promise and excitement that it generated in the early 1990s has waned, it is an extremely powerful technology with applications that range far beyond those that have hitherto been developed. These have included simulation and training [1], therapy and rehabilitation [2], simulation of social situations in experimental studies [3], [4] and many others of a similar type. The vast majority of applications operate at human scale, except when virtual reality has been used for data visualisation, for example of data obtained from a confocal microscope [5] or for manipulation at the nanoscale [6]. Virtual reality still requires significant technical and conceptual advances [7] but such advances will come through novel applications that spur further technical and scientific research. In particular when combined with teleoperation it can open up a new class of applications such as the one considered in this paper. Immersive virtual reality (IVR) and teleoperator (TO) systems provide the technical means for instantaneously transferring a person into a different place. An IVR system places people into a computer-generated environment where they can use their body normally for perception and interact with virtual objects, and with representations of other humans. Such virtual reality systems can be used to give people the illusion of being in the place depicted by the environment where they tend to behave as if what they were experiencing were real [8]. With TO an operator can have the sense of being physically in a remote real place, embodied there as a robot – seeing through the eyes of the robot whose actions are slaved to the motor actions of the operator. There the operator can, for example, operate remote machinery, collect samples, and so on. When we combine IVR with TO we open up a new class of application where the human participant operates in a virtual (possibly transformed) representation of a remote physical space in which there are other live beings that may exist and act on an entirely different scale to humans. In particular here we show how to use IVR and TO to create a system that allows humans, and in principle, the smallest of animals or insects to interact together at the same scale. The fundamental idea is that the human participant is in an IVR system interacting with a virtual character (avatar) representing a remote animal. The animal is tracked in its physical space. The tracking information from the animal is relayed to the IVR and controls the actions of the avatar that represents it. The VR is scaled so that movements of the animals are mapped into appropriate changes in position of their avatar representations on a human scale. From the point of view of the humans there is a VR in which other live beings are represented with which they can interact. We have so far described the setup from the human point of view - but how do the animals interact with the human, since the animals themselves are not in a virtual environment but in their own habitat without any special displays? The answer is that just as the animals are tracked and this information controls the movements of their virtual representations, so the humans are tracked and this controls the movements of a robotic device that is located within the animal habitat. Hence when the human, for example, moves close to the representation of the animal in the virtual environment, so the robot moves close to the corresponding animal in the physical habitat. There is a proportional mapping between the spatial relationships and orientations of the robot with respect to the animal in the physical space, and the human with respect to the animal’s avatar representation in the virtual reality. Both animals and humans experience their environment at their own scales. We call this process ‘beaming’ since the human in effect digitally beams a physical representation of him- or herself into the animal environment. We describe an example of such a system that enables people to beam into a rat arena and interact with the rat at human scale, while the rat interacts with the human on the rat scale. In our particular application, a humanoid avatar represented the rat in virtual reality, and a small robot in the rat open arena represented the human. The human and rat played a game together as an example of the type of interaction that is straightforward to achieve in such a system. The purpose was to (a) Test the overall system performance during an interactive game played between person and rat. (b) To examine how the rat reacted to the robotic device. (c) To examine how the human participants accepted the setup and played the game, indeed whether it was possible to play the game at all. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Figure 1. The rat arena and robot device. (a) Two of the pictures on the wall can be seen, and the frame on which a webcam was mounted for tracking purposes. (b) The e-puck robot protected by a purpose-made armour. For tracking purposes, a typical Augmented Reality marker was attached on top of the armour. The plastic platform in front was used to hold the food (strawberry jelly) for the rat. (c) Left hand side: View of the robot and rat for tracking. Right hand side: Result of the threshold used to detect the rat in the image. https://doi.org/10.1371/journal.pone.0048331.g001

Results System Performance A number of measures were used in order to evaluate the performance of the system, in terms of network performance, video streaming latency and robot command latency. The software architecture of the experiment was distributed on three different machines at the two different physical sites both connected via the internal network of the University of Barcelona (Figure 2). Hence, a ‘ping’ command issued between distant computers, which corresponds to measuring the time between sending and receiving back 32 bytes of data, showed an unnoticeable delay (<1 ms). The video stream required sending a 640×480 pixels RGB video between two distant computers. The latency measured revealed a delay of 120 ms (±20 ms) between a frame sent from the video streaming laptop and the IVR computer. Finally, the measured delay of the robot command stream between the computer responsible for tracking and that running the virtual reality displays was 150 ms (±20 ms). This delay corresponded to sending a command via the UDP protocol from the IVR computer, receiving this command on the tracking computer in the MATLAB software, and processing the command before finally sending it to the robot via the Bluetooth protocol. The Bluetooth protocol itself induced a delay up to 20 ms. The human participants in virtual space and the rat and robot device in the physical space of the open arena were tracked at the sampling rate of 30 Hz. Since there is no Gold Standard algorithm against which we can compare the accuracy of our system we only can provide the algorithm’s runtime, which was estimated as 10 ms for the calculation of the rat's major position and 20 ms for the estimation of the head position and viewing direction on an Intel Core2 Duo CPU with 2.50 GHz. The robot tracking which is marker-based is very efficient and is negligible compared to the rat tracking. Putting everything together the time spent in the tracking process represents roughly 30 ms, which consists of both robot and rat tracking (body position, head position and head orientation). Movement Distributions The two rats both showed typical navigational patterns, staying close to the walls for most of the time, with occasional forays towards the centre. This is a typical behaviour of rodents referred to as thigmotaxis, enhanced by illumination [13] which was the case in our experiments. Figure 4 shows movements over the whole period of an arbitrarily selected trial for both rats, and the movements of the corresponding participants. It is shown that the rat tended to gravitate towards the edges and corners. The human covered more the central area to entice the rat towards the centres of the walls (where the posters were located). Rats were trained to follow the robot in the search for reward, and thus the principal reason for the rat to move away from the thigmotactic pattern of remaining close to walls and corners was most probably the presence of the robot. This can be seen in Video S2, which shows 6 typical sequences of the movements of rat and robot. We obtained all of the (x, y) positions of each of the two rats during all the trials using the sampling unit of time as 0.2 s following [14]. The proportion of time that the tracked centre of the rat’s body (without tail) was within a radius of 20 cm of the centre of the arena was computed. The rats were approximately 18–20 cm in length and 5–6 cm in width. Hence a radius of 20 cm in the area size of 80×80 cm2 indicates a region quite distant from the edges. We counted the number of times that there was contact between the rat and robot that occurred while not by the correct poster, referred to as ‘rat points’ (since the humans only obtained a point when the collision was near the correct poster). Figure 5 shows the number of rat points by the proportion of time that the rat was in this central region, over all participants and for both rats (for the first trials only). There is a linear relationship between these (Pearson r = 0.71, P<0.001) indicating that the greater the time that the rat was in the centre the greater the number of collisions with the robot. Since the participants knew that they would lose a point in the game if a collision occurred that was not under a poster, it is likely that such encounters were due to the rat following the robot, rather than through the actions of the human. A similar result holds for a radius of 15 cm, and even with a radius of 10 cm the relationship is still significant for rat A (r = 0.89, P<0.0015). Was the game played? Corresponding to each (x, y) position was the distance between the robot and the rat at that moment (which itself was directly proportional to the distance between human participant and the avatar representing the rat in the VR). We divided the arena floor into a 5×5 grid and found the mean distance between rat and robot for each grid cell over all the participants. We were interested to see whether any pattern could be found that indicated that movements were not just random, and that indeed the game was played. Figure 6 shows the resulting graphs. The figure shows that the distance between rat and robot (human) was greatest when the rat was in its starting corner or an adjacent corner. The graphs also show minima where the posters were located indicating that the game was being successfully played. This is most pronounced in the case of Figure 6 (a) and least pronounced for Figure 6 (b) which corresponded to trials when the participants believed that they were playing against a human opponent. However, in almost all cases the mean distances near the posters are significantly less than the mean overall distance between the rat and robot taken over the whole time period. This can be shown by calculating the normal z-statistic for comparison of a sample mean with a population mean, here taking the population mean to be the mean distance over the whole time period for a particular rat and trial. These overall means are 0.37 m and 0.39 m for Rat A for trials 1 and 2 respectively, and 0.38 m and 0.40 m in the case of Rat B. For Rat A in trial 1 the four regions in the 5×5 grid corresponding to the positions of the posters have |z| >4 for all but one, and similarly for trial 2 all |z| >3.6 except for (the same) one. For Rat B all |z| >6.6 for trial 1, and all |z| >10.7 for trial 2. This does strongly suggest that the distances around the posters were usually quite different from the overall distance. The time varying distance between the rat and the robot representing the human is illustrated in Figure 7 which shows the plot of distance between the rat and the robot (human participant) over the 5 minute period of the experimental trial, following the same rat (A) and participant as in Figure 4 (a, b). This is typical of all such plots representing the dynamics of movement of both rat and human, as they approached each other and moved away again. The evidence suggests the distances between rat and human tended to be slightly greater in trial 2 than in trial 1. In trial 2 participants believed that their opponent was a human. This change in distance could be due to that belief and therefore the desire of human participants to follow rules of proxemics, that is to keep a socially acceptable distance from their opponent, or could be due to the fact that in the second trial the game was played less successfully than in the first. In fact the total number of points scored by participants in the second trial was about half that scored in the first trial. This may have been because the rats were tired or satiated, or it could have been because the humans believed that they were playing against another real human, and adjusted their behaviour accordingly. The evidence regarding this issue is considered and weighed in Text S4.

Discussion Since this is a newly developed system it is interesting to consider possible applications. Unlike existing ethological studies of animals, for example, cats [15] and horses [16], it may be interesting for life science investigators to obtain an entirely different view of animal behaviour, by seeing the animals on a human scale, even represented as humans. This would offer a possibility of participant-observational study of animal behaviour and generally of animal communities in a way never before possible. Such changes of view may offer quite new insights. It might be thought that generally rats would not behave normally when there are robots in their vicinity. However, the placing of robots in rat arenas has been carried out before, as part of the quest to develop a robot that is rat-like in its behaviour. For example, in one system [17] a robot that emulated some rat-like behaviour was placed in a open arena with a rat. An experimental study concluded that the robot influenced the rat behaviour in an appropriate way. Ultimately the authors wished to create robots that would interact with humans; however, working with rats provided an environment in which to understand the relationships that may develop between animal and robot in a simplified form. Other work has also had this motivation [18], where the rat and robot developed a symbiotic relationship over many hours, and where the robot could learn to manipulate the behaviour of the rat. Generally there is an increasing amount of work that seeks to understand animal behaviour for the engineering of robots and then testing the robots in the context of interacting with the animals that they emulate, for example, an ‘animat’, a robot that navigates like a rat [19]. The flow of understanding is two-way, where such animal-based robots can shed light on animal behaviour and cognition. To our knowledge there has never been a system where a physical device operating in a rat environment acts as a surrogate representation of a human operating in an equivalent virtual environment. Some specific computational requirements are discussed in Text S2, but in general the system components needed to do this are: (a) An IVR system that can track the movements of a human participant; (b) A device that can be slaved to the actions of the human which is located in the animal space - a teleoperation system; (c) Tracking of the animals in their space and the relaying of the tracking information to control avatars in the virtual environment; (d) A network capable of real-time distribution of data between the human and animal sites; (e) A virtual model of the remote (animal) locale. As an example, this type of system could even be used to allow interaction between humans and birds or flying insects. There exist today flying robots [20] so that (b) would be supported. Moreover, it is possible to track, for example, birds [21] so that (c) would also be supported. Also in relation to (b) another instance of this type of system could replace the robotic device by a real rat with its movements controlled remotely though brain stimulation [22]. In the paragraph above we have extended beyond a single animal - which requires the capability to track multiple animals simultaneously and thereby control multiple avatars. Moreover, the same could be extended to multiple human participants (further technical details are discussed in Text S3). Virtual reality has previously been used for communication between multiple participants where people in remote places can meet in a virtual environment shared by all. In such applications each of the participants uses their own virtual reality system, perhaps separated by thousands of kilometres, and they can see and talk to life-sized representations of one another, and carry out tasks together [23]. This is facilitated by Internet network protocols that distribute the data between the various systems, and each system is responsible for displaying the virtual environment from the viewpoint of its particular participant. This has even been achieved with haptic interaction between the remote participants [24], [25]. However, what is different in our system is that the human is represented in the animal environment through a physical surrogate. In shared virtual environments all participants are in a virtual reality system. In our case only the human is in such a system, whereas the animals are located in their own physical environment without any need for virtual displays. The conjunction of immersive virtual reality with teleoperator systems supports a class of application that would be very hard to achieve through any other means. The virtual environment acts as a unifying medium through which participants who operate at quite different scales can be brought together, and their appearance changed as appropriate to the demands of the application. Although we have applied this technology to interaction between humans and animals, primarily for use in the life sciences, the very same idea could be used for example, to realise human to remote-human interaction, with an example of such remote communication described in [26], [27].

Acknowledgments We thank Alvaro Gimeno from the animal care facility in Bellvitge and Sílvia Aliagas for the animal training.

Author Contributions Conceived and designed the experiments: MS MVSV MAS. Performed the experiments: J-MN EG. Analyzed the data: MS. Wrote the paper: MS MVSV J-MN GK CG MAS. Programming the environment: J-MN CW BG BS.