Predator animals have long been known to avoid devouring brightly coloured and patterned prey, and now an international study has revealed more about how they recognise toxic species.

University of Queensland Visual Ecology Lab member Dr Karen Cheney, of the School of Biological Sciences, said researchers examined sea slugs, or nudibranchs, which had bright colour patterns to warn predators they contained toxic defences.

The researchers theorised that there should be little variation in prey species' visual warning signals, as this could hinder how predators learnt that a particular species should not be eaten, due to being toxic or tasting terrible.

"However, the study -- led by UQ's Dr Anne Winters -- provides evidence of how variation in animal coloration can evolve and be maintained," Dr Cheney said.

"We have demonstrated that due to natural colour pattern variation, a fish only learns to recognise the consistent parts of the colour pattern that the Gonibranchus splendidus nudibranch displays, and not the parts of its pattern that change."

The Gonibranchus splendidus displays a consistent yellow rim bordering a white area that has variable red patterns. The species is found on reefs from the Southern Great Barrier Reef to NSW.

"We showed that fish predators (triggerfish Rhinecanthus aculeatus) only pay attention to the yellow border of the colour pattern when learning avoidance of the signal and they pay little attention to the red spots," Dr Cheney said.

"Down the coast of Australia, the colour and pattern of the nudibranch's red spots is highly variable between different populations. However, the yellow rim is very consistent.

"Natural selection may act on parts of the colour pattern in very different ways, allowing for the yellow rim to be stable, but the red spots to be highly variable."

Dr Cheney said many species ate sea sponges, from which they obtained and stored compounds distasteful to predators.

Divers and underwater photographers had long been fascinated with nudibranchs' diverse colour patterns, she said.