Octopuses are well-known masters of camouflage and skillful escape artists, but they aren't exactly famous for their social skills. Scientists have long thought that this many-armed denizen of the deep was strictly solitary and didn't interact much with its fellows, reserving its color-shifting ability for intimidating predators — or hiding from them. But a new study reveals that both male and female octopuses frequently communicate with each other in challenging displays that include posturing and changing color.

And certain behavior patterns emerged that accompanied different color-shifting displays. If two octopuses approached each other and displayed dark colors, the encounter was likely to escalate aggressively and lead to physical confrontation. But an octopus displaying paler colors usually indicated that it was preparing to retreat, the researchers found.

A diver in Jervis Bay, Australia, initially spotted the unusual octopus behavior, writing about it in an online post that attracted the scientists' attention. To decode the octopuses' social "language," the scientists captured and screened 52 hours of footage of the Octopus tetricus species, a mottled, grayish-brown cephalopod known as the common Sydney octopus and also — somewhat more amusingly — the gloomy octopus. Despite its mopey-sounding name, what the scientists observed wasn't a community of isolated sulkers, but a highly engaged and interactive bunch. [Video: Watch Octopuses Argue Through Body Patterns And Postures]

"Because octopuses were known to kill each other at times and be cannibalistic, the general sense is that they wouldn't interact a lot and wouldn't use signals," David Scheel, the study's first author, told Live Science. But Scheel, who is a professor of marine biology at Alaska Pacific University, had already found a scattering of reports from prior studies that suggested "there was another story going on," he said.

An octopus (foreground) displays pale color and stretches out one arm before it withdraws from an approaching octopus (background). The approaching octopus displays a dark color, 'stands' tall, and spreads web and arms. Credit: David Scheel

"Some octopuses have been seen in displays that may have occurred to woo potential mates, and some have [been] found in aggregations. So, there have been hints in the literature that suggest this may have been possible, but no focused reports that looked just at signaling among octopuses." What Scheel and his colleagues observed was novel — a site where the gloomy octopuses would visit and confront each other, over and over again.

"One of the early bits of video that I saw showed one octopus approaching another in a fairly dramatic way — dark and standing very tall, and the other one crouched down, turned pale and then fled," Scheel told Live Science. "It just looked to me like they were signaling, so we just followed from there to try and explore that idea."

Anywhere from three to 10 octopuses appeared at the site on a given day, the researchers noted. In the 52 hours of recorded footage, over 7 hours showed octopus interactions, with 345 instances of changing colors and 512 examples of physical movements, such as grappling or reaching toward each other. Reaching was the most common interaction the scientists saw, making up 72 percent of all the physical interplay; the octopuses touched each other very infrequently, the scientists recorded.

They observed one posture repeatedly — when the octopus would "stand tall," extending its arms outward and drawing itself upward. An octopus that was standing tall would usually also display a dark color and raise its mantle, all of which, the researchers said, appeared to signify aggression toward another octopus. Other cephalopods, like cuttlefish, are known to assume a darker color during disputes, with males displaying a "dark face." If two male cuttlefish show each other dark faces, the confrontation usually turns physical, while if only one of the males puts on his dark face, the paler-faced cuttlefish typically backs down, a behavior pattern strikingly similar to the one the researchers observed in the octopuses' color displays.

"Signaling is well-documented in cuttlefish and squids," Scheel said, "so in that way it isn't really surprising to suggest that octopuses do the same thing." [Cuttlefish Cuties: Photos of Color-Changing Cephalopods]

One of the most surprising things that Scheel saw in the videos was how the octopuses used a piece of flotsam embedded in the silt at the study site, he said. It stuck up higher than the surrounding seafloor, and a displaying octopus would sometimes climb up on top of it to perform the "stand tall" posture. "That's something that's kind of interesting that makes use of the particular layout of the environment," Scheel said. The structure stood at the center of the site, so perhaps the presence of that "high ground" was what drew displaying octopuses to that area in the first place, Scheel suggested.

The next step, Scheel said, is to look more closely at the context of all the signaling, in which both octopus sexes participated. "Males interacted with females, males interacted with other males, females interacted with males, and we're pretty sure that females were interacting with other females as well," Scheel said. But whether all the signaling is part of a complex mating system or some other form of behavior entirely remains to be seen, he added.

The findings were published online today (Jan. 28) in the journal Current Biology.

Original article on Live Science.

Copyright 2016 LiveScience, a Purch company. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.