Morphology-independent sensorimotor coordination

For over 10 years11, 12, we have been developing the basic technologies that are a prerequisite for MNS robots. We have developed robotic units that can autonomously form physical connections with each other (Supplementary Fig. 1). Previously, however, each unit was always an independent robot5. While advances have been made in the development of algorithms and hardware that allow modular systems to form collective robot bodies of previously unattainable scales13, the coordination and control paradigms for such robots are constrained to a predefined set of morphologies14. Our MNS robots are the first self-assembling multirobot system able to display sensorimotor coordination equivalent to that observed in monolithic robots. To demonstrate the capability of MNS robots to display morphology-independent sensorimotor coordination, we set-up an experiment in which we manually design the behavioral rules for ten robotic units so that they form a series of MNS robots of different shapes and sizes (Fig. 2 and Supplementary Movie 2). The different MNS robots all display the same coordinated sensorimotor reaction to a provided stimulus. This reaction involves pointing at the stimulus using light emitting diodes (LEDs), and retreating from the stimulus if it is sufficiently close. When a composite MNS robot points to the stimulus, only the LEDs closest to the stimulus illuminate, independently of the robotic unit to which those LEDs belong. When moving away from the stimulus, movements of all wheel actuators on all constituent robotic units are coordinated through the robot nervous system of the MNS robot, allowing smooth motion of the composite body (see Methods section).

Fig. 2 Morphology-independent sensorimotor coordination. Independently of shape and size, MNS robots display consistent sensorimotor reactions to a stimulus, while autonomously merging their bodies and robot nervous systems. Photos are snapshots from a single experiment (Supplementary Movie 2) in which ten MNS robots respond to a moving stimulus. We design the behavioral rules for the robotic units so that when the green stimulus enters a robot’s sensor range, the robot ‘points’ at the stimulus by illuminating its three closest green LEDs (in a composite MNS robot, these are the closest LEDs on the closest constituent robotic unit). For clarity, we have added concentric green lines as overlays to highlight the pointing direction, and brain units have their LEDs illuminated in red. When the stimulus is ‘too’ close (i.e., proximity to any part of the MNS robot’s body exceeds a threshold), the robot retreats from the stimulus. a Each robotic unit is an independent MNS robot in its own right. The stimulus provokes a reaction in three of the robots. b The ten MNS robots have merged to form two larger MNS robots. These newly formed robots are composite MNS robots each consisting of several robotic units. However, the two MNS robots are independent robots in their own right—they each have a single brain unit and a single robot nervous system. Both robots point at the stimulus. c The two MNS robots retreat from the stimulus. d The two MNS robots autonomously merge to form a larger MNS robot with a single brain unit. e The newly formed 10-unit MNS robot points at the stimulus. f The MNS robot retreats from the stimulus Full size image

Body representation

The physical connection topology of an MNS robot is a rooted tree. The logical topology of an MNS robot’s nervous system follows the physical connection topology with each constituent robotic unit maintaining a recursive body representation of itself and all of its child robotic units. The representation includes the relative positions and hardware configurations of its child robotic units stored as a set of spatial relationships—the geometry of a robotic unit, and the physical arrangement of sensors and actuators (Supplementary Fig. 2). The recursive body representation allows MNS robots to react to radical morphological changes promptly. For instance, when an MNS robot splits into multiple robots with separate bodies, each root unit of the uncoupling body segments already has all the knowledge it needs to become the brain unit of the new independent robot. Given the tree structure of a mergeable nervous system, the root unit in a robot can always be identified unambiguously and serves as the brain unit. An important feature of morphology change in MNS robots is the speed at which the internal representation can be updated. When a merge between two MNS robots occurs, only a single message needs to be passed up the merged nervous system from the connecting MNS robot to the brain unit of the MNS robot to which it connects. The information contained in the message is incrementally updated by each intermediate unit with local topological information (Fig. 3), and the newly formed MNS robot incorporates all the sensing, actuation and computational capabilities of the units in the new body (Supplementary Movie 3). When an MNS robot changes shape or size, there is thus no need for time-consuming processes such as self-discovery14, trial-and-error15, or hormone-based messaging16.

Fig. 3 Propagation of internal representation during the merging of two MNS robots. a The internal representations of robotic units are shown in the insets. In each of the insets, the robot whose internal representation is illustrated is indicated by the corresponding letter. Note that units that are not the brain unit only have awareness of their descendant units. b The robot on the left has assembled to the robot on the right. The brain unit of the robot that is attaching (left robot) cedes authority to the brain unit of the robot to which it is attaching (right robot). The brain unit of the new merged robot still does not have an accurate internal representation of its new morphology. However, its child robotic unit has already updated its internal representation. c Information about the morphology has propagated to the brain Full size image

Control logic

Writing the control logic for a composite MNS robot is an entirely new challenge. It is impractical for such logic to take into account every possible morphology, i.e., relative placement of sensors, actuators, and body parts. Our solution is to divorce the control logic from the morphology and from individual sensors and actuators. The control of an MNS robot is expressed in high-level logic that is independent of the size and shape of the robot (Supplementary Table 1 for a list of the commands available to MNS robots). The brain unit issues high-level commands that are propagated through the robot nervous system. If a high-level command applies to a robotic unit, the robot nervous system locally translates the command into instructions for the unit’s actuators. Figure 4 shows how responsibility is delegated as part of the information flow in the robot nervous system of a merged robot.

Fig. 4 Sensor and actuator information flow in an MNS robot. A 5-unit MNS robot detects and then responds to a stimulus. a: The stimulus moves within sensor range of two robotic units in the MNS robot. Both robotic units perform the computationally intensive visual image processing required to analyze their camera feeds. They pass an abstraction of this information (e.g., existence and coordinates of the stimulus) up to their parent robotic unit (i.e., the unit in the center of the robot) using a Wi-Fi connection. b: The parent robotic unit fuses the information coming from two of its child robotic units, to form a more accurate estimate of the stimulus’ coordinates. The parent robotic unit then passes this single item of information up to its own parent robotic unit, which in this case is the brain unit. The brain unit decides what action to take, based on the data it has received—in this case, the decision is to point at and retreat from the stimulus. c: The brain unit issues high-level actuator commands. d: The high-level commands are translated into actuator instructions individually by each robotic unit Full size image

Spatial coordination is achieved by enforcing coordinate translation every time a sensor or actuator message is passed from robotic unit to robotic unit. As each robotic unit knows the relative location of its parent robotic unit and child robotic units, spatial references are translated into the frame of reference of the receiving unit before the messages are transmitted. In this way, units always receive messages that are meaningful within their own frame of reference. The coordination of the actuators on different robotic units takes any propagation delay associated with the robot nervous system into account. The length of the delay is a function of the path from the brain unit to the most distant leaf unit in a morphology, and the communication technology on which the MNS is based, Wi-Fi in our case. If the propagation delay is significant, actuator instructions are not executed by a robotic unit immediately upon reception, but instead postponed until the instructions have had time to propagate to all units in the body. In Supplementary Fig. 3, we provide a detailed example of how spatial and temporal actuator coordination happen in an MNS robot.

Scalability

We consider scalability with respect to the number of constituent robotic units in an MNS robot, first in terms of computational resources required to control the robot and then in terms of reaction time, that is, the time required for a robot to react to a new stimulus.

In an MNS robot, sensory data from child robotic units are fused by their parent robotic unit before being passed up the robot nervous system towards the brain unit. The computational cost of sensor data extraction and processing for any robotic unit, including the brain unit, is thus proportional to the number of immediate child robotic units, rather than a function of the total number of robotic units in an MNS robot’s body. Scalability issues would quickly arise if all sensory data (such as camera feeds) were collected and processed in the brain unit. Instead, the use of local sensor fusion allows the MNS method to scale gracefully in terms of computation.

For an MNS robot to react to a new stimulus, a message must propagate from the constituent robotic unit sensing the stimulus to the brain unit, which in turn must propagate a message detailing the response back through the body. The reaction time for an MNS robot thus depends not only on the number k of constituent robotic units, but also on the MNS robot’s shape (the connection topology of the constituent robotic units). The worst case reaction time is given by 2lp × τ, where lp is the length (in robotic units) of the path from the brain unit to the most distant robotic unit in the body, and τ is the communication delay between two adjacent robotic units. In our current implementation, adjacent robotic units can exchange messages every τ = 100 ms. The 4-unit MNS robot shown in Fig. 2b, c has a longest path of lp = 2 robotic units and its reaction time is thus 2 × 2 unit × 100 ms unit−1 = 400 ms. Similarly, the 6-unit MNS robot in Fig. 2b, c has lp = 3 units and therefore a worst case reaction time of 600 ms, while the 10-unit MNS robot in Fig. 2d–f has lp = 5 units and thus a worst case reaction time of one second. In general, the reaction time of an MNS robot of size k falls between two extremes depending on body shape: the upper-bound corresponds to a robot with a linear shape for which the reaction time is 2k × τ, while the lower-bound corresponds to a shape that is as compact as possible. Given the circular shape of our robotic units, the most compact shape possible is one in which constituent robotic units are arranged in a hexagonal lattice pattern around the brain unit, which yields a longest path of approximately lp = log 2 k units and therefore a reaction time of 2 log 2 k × τ.

Self-healing

MNS robots can, in principle, self-heal by combining their splitting and merging capabilities to substitute faulty components, including their brain unit. After a fault has been detected (see Methods section), MNS robots can reconfigure their bodies to excise the faulty robotic units and possibly substitute them with new, spare units. To demonstrate the self-healing capability, we design behavioral rules for eight robotic units so that they self-assemble into an MNS robot with an Y-shape. We then inject a fault, first in the brain unit of the MNS robot and then in a robotic unit of the MNS robot body. In the first part of the experiment, in which the brain unit fails, the three child robotic units detect the fault and respond by detaching, thereby creating three new, independent MNS robots each with a brain unit of its own. The three robots then merge with one another to form a new, larger robot with a morphology as close as possible to the original robot (Fig. 5 and first segment of Supplementary Movie 4). In the second part of the experiment, a robotic unit in the body of the MNS robot fails. In this case, the body part containing the failed unit is detached from the MNS robot and two new robotic units are recruited to recreate the original robot (Supplementary Fig. 4 and second segment of Supplementary Movie 4). In general, reconfiguration sequences in response to faults are task and morphology specific. The simplest fallback case is for all of the child robotic units to completely disassemble, and to reform another morphology from scratch, but partial disassembly may be sufficient in many cases.