Lighting is crucial to the art of photography. But lights are cumbersome and time-consuming to set up, and outside the studio, it can be prohibitively difficult to position them where, ideally, they ought to go.

Researchers at MIT and Cornell University hope to change that by providing photographers with squadrons of small, light-equipped autonomous robots that automatically assume the positions necessary to produce lighting effects specified through a simple, intuitive, camera-mounted interface.

At the International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging in August, they take the first step toward realizing this vision, presenting a prototype system that uses an autonomous helicopter to produce a difficult effect called “rim lighting,” in which only the edge of the photographer’s subject is strongly lit.

According to Manohar Srikanth, who worked on the system as a graduate student and postdoc at MIT and is now a senior researcher at Nokia, he and his coauthors —MIT professor of computer science and engineering Frédo Durand and Cornell’s Kavita Bala, who also did her PhD at MIT — chose rim lighting for their initial experiments precisely because it’s a difficult effect.

“It’s very sensitive to the position of the light,” Srikanth says. “If you move the light, say, by a foot, your appearance changes dramatically.”

Intuitive control

With the new system, the photographer indicates the direction from which the rim light should come, and the miniature helicopter flies to that side of the subject. The photographer then specifies the width of the rim as a percentage of its initial value, repeating that process until the desired effect is achieved.

Thereafter, the robot automatically maintains the specified rim width. “If somebody is facing you, the rim you would see is on the edge of the shoulder, but if the subject turns sideways, so that he’s looking 90 degrees away from you, then he’s exposing his chest to the light, which means that you’ll see a much thicker rim light,” Srikanth says. “So in order to compensate for the change in the body, the light has to change its position quite dramatically.”

In the same way, Srikanth says, the system can compensate for the photographer’s movements. In both cases, the camera itself supplies the control signal. Roughly 20 times a second, the camera produces an image that is not stored on its own memory card but transmitted to a computer running the researchers’ control algorithm. The algorithm evaluates the rim width and adjusts the robot’s position accordingly.

“The challenge was the manipulation of the very difficult dynamics of the UAV [unmanned aerial vehicle] and the feedback from the lighting estimation,” Durand says. “That’s where we put a lot of our efforts, to make sure that the control of the drone could work at the very high speed that’s needed just to keep the thing flying and deal with the information from the lidar [the UAV’s laser rangefinder] and the rim-lighting estimation.”

Quick study

As Srikanth explains, that required some algorithmic streamlining. “When we first started looking at it, we thought we’d come up with a very fancy algorithm that looks at the whole silhouette of the subject and tries to figure out the morphological properties, the curve of the edge, and so on and so forth, but it turns out that those calculations are really time-consuming,” Srikanth says.