Whales are not only the largest animals that have ever lived – they are also among the most intelligent, and yet we still know very little about them. New research, however, suggests that sperm whales at least use sophisticated communication techniques to develop distinct and separate cultures. Here to discuss the latest in cetacean research to mark next month's Peninsula Arts Whale Festival, are Philip Hoare, a self-confessed "whalehead" and author of Leviathan or, The Whale, winner of the 2009 BBC Samuel Johnson Prize, and Dr Hal Whitehead from Dalhousie University in Halifax, Nova Scotia, a world expert on sperm whales.

Philip Hoare We're talking about our two disciplines, which meet on a very specific subject, the whale. There are those who work with this animal from a scientific point of view and those work with them from an artistic or literary point of view. What fascinates me is that whatever we do, the whale always remains elusive.

For us as writers and artists, it is elusive because it is beyond description. I quickly ran out of superlatives talking about whales when I was writing Leviathan or, The Whale. But for you as a scientist it is elusive in that it is very difficult to study in the environment in which it lives. What first drew you to the sperm whale?

Hal Whitehead It's much more exciting to go for something that we really don't know. I think that of all animals, the sperm whale is the one with the greatest mystery, and that I find very attractive. And I am assuming this is what attracts writers and artists to the same animal.

PH That's precisely right. For me there is a degree of romance about the subject. Writing about whales has brought me into close contact with scientists and given me a new understanding about the animal. Your book Sperm Whales: Social Evolution in the Ocean talks about sperm whale culture, but what do you mean by "culture" when you apply it to this animal?

HW When a biologist like myself starts talking about culture eyebrows are raised in other disciplines among scholars who see themselves as the "knowers" of culture. But, for a biologist, culture is primarily a flow of information – both ideas and behaviours – between the members of a population, which happens because they learn from each other.

The flow of information is what biology is about. Without information getting from one animal or plant to another, there would be no life. We as biologists have concentrated on genes as the way information gets around, but there are other ways in which this can happen, and culture is one of them.

Culture leads to a whole new range of processes, which don't happen in most animals or plants. The possibility that there are other groups of animals that have these flows of information, running through their population and interacting in extraordinary ways, is most intriguing. In the case of the sperm whale, we have a creature of great mystery and potential. We tried to explain what we found using the principles of genetic evolution but we found things we couldn't explain, so we had to think about other processes. The process that made sense was culture.

For example, I was studying a group of whales off the Galápagos Islands, looking at their social systems, and found two kinds of sperm whale who were behaving really quite differently. They had different ways of communicating with each other, different ways of using the resources around the island, etc. The initial explanation was that we had two sub-species but there was virtually no difference genetically. So something else was causing these sperm whales to form radically different societies, with radically different ways of behaving. It became obvious that the only explanation was that these whales had different cultures. They were living in a multicultural society.

PH Sperm whales communicate pre-eminently via the click system – series of low sounds that they create within their heads, their individual sonar devices. How far is this communication an expression of their culture?

HW There are markers in each sperm whale's vocal message which signify their cultural grouping. It has been suggested by anthropologists that symbolic ethnic markers are features of human culture. We don't know for sure, but we have indications that sperm whales use these sound patterns as symbols of their clans.

PH Recently, there was a report on intelligence in other species, which placed dolphins above primates and behind humans. I wonder, given that the sperm whale has such a big brain and such a highly developed neocortex, do you think we are talking about animals that are quite close to humans in their social organisations? How can we find out how whales use their brains?

HW It is incredibly difficult to get a handle on. There have been a number of studies on various cognitive tasks using dolphins but all these tests have been designed by humans, based on how we see our world, how we interact with it. Dolphins and sperm whales live in a world structured very differently to ours, where different features are important.

Whales, largely, sense and communicate acoustically, whereas we do most of our sensing visually. They live in a three-dimensional world; we live in a two-dimensional world. We are trying to relate them to what we are in metrics that correspond to how we see the world. This is likely severely to underestimate their capabilities.

But the other side of this is that the potential mysteries are even greater. I think it very likely that whales are doing things that, at the moment, we can't even conceive – we are not operating in that world. So a real challenge for scientists is to study things when you don't know what you are looking for.

I want to move into this area, and this is where people like you come in – because artists and writers aren't constrained by the scientific processes. You can speculate, imagine yourself in the world of the whale. And then open-minded scientists, by looking at what artists produce, may make hypotheses that will lead us onto paths that will begin to crack these great mysteries.

Another idea, which came from a discussion with science writer Jeff Warren, is to use a virtual world as close as we can get to the real one of the whale to explore what it might mean to be a whale. People go into this virtual world and live in it, and through their changing experiences we might get some idea of what it is to be a whale.

PH There are things we need to do with our brain – pay the mortgage, drive a car, write a book etc – that a whale obviously doesn't have to do and, as you say, the whale lives in an element in which we cannot live. Unlike us, the whale lives in a three-dimensional environment that covers 70% of the world's surface. How far do you think the whale is using its brain as a reflection of the vastness of its environment? Would it need a big brain to compute that environment?

HW Because whales live in this very large three-dimensional habitat and they don't have a physical structure that is home for them, they are always on the move. The most important part of the environment for a whale is probably other whales. Social life is vital. As they move around, everything is changing except their social lives. To me, it seems very likely that for these animals social life is even more important than it is for terrestrial animals, including humans. Relatively it has a much bigger role in their world. Therefore their social lives are absolutely vital – they need social intelligence.

Whales are cultural species. It is not only important to maintain and nourish the social relationships which they depend on, but also to make really good use of the information which is flowing among these social partners. My guess is that those social and cultural needs have been the drivers of the big and complex brain rather more than the direct need to keep track of a large and complicated environment.

PH That raises the notion that, for a whale, culture is more important than it might be for a human, also that this is an invisible process. Could you describe the technical way the sperm whale communicates? I read that when sperm whales gather, they bring their heads together and there might be some way in which this intensifies the communication process. Is this true?

HW Whales probably communicate in a lot of ways. The ones we know about are the ones that are most obvious to us, but this doesn't mean that they are necessarily the most important to sperm whaless; they are just the ones which we have a means of getting our heads around. At short range, whales are very tactile animals. They spend a lot of time in contact with each other and communicate through touch and this is, presumably, really important.

Unlike us, whales don't have obvious ways to change their bodies as a method of communication, such as our facial expressions, so vision for whales probably isn't so important. However, I suspect whales have a very detailed view of the outline of each other's bodies and this almost certainly conveys information between whales, perhaps involuntarily.

The sounds they make also appear to be communicative. Sperm whales do this in a strange way – about a quarter of their body is this huge sonar system which makes very loud clicks and which allows them to find food deep under the ocean. But they use the same system to communicate. The whales modify these clicks and put them into patterns. I, and others, have tried to figure out what these patterns mean – one thing is clear, they appear to be used as a means of reinforcing social bonds.

PH What fascinates me is the notion that these animals, like us, might start to rationalise their place in the world – what they mean, what their existence means. This is something you have hypothesised upon. I am not asking you to tell me what a whale might be thinking about, but do you think it is possible that a whale can have an existential sense of itself?

HW It is conceivable and it is one of the mysteries that attracts me to this area. Living in a situation where social relationships and culture is very important, things like the theory of mind – having a concept of what others know and therefore a concept of self – become more likely. In turn, this would lead to one to contemplating the self in the environment and how they fit together. As whales are a cultural species, ideas might be being passed between individuals and down through generations which relate to the whale's place in the world. I am not sure how we get to prove this but the possibility is certainly out there!

PH When I read about your hypothesis that whales have developed their own religion and that they have a sense of morality because of the way they interact with each other, I couldn't believe it. I think I am right in saying that their sonar, with which they communicate, is also used to hunt – I saw it myself in New Zealand when I watched a sperm whale use its sonar to stun kingfish at the surface to eat. But your notion is that these animals have the power to cause damage to one another, which is where they might develop a sense of morality as a social complex. Is that right?

HW Sperm whales have the most powerful sonar in the natural world. It is very directional and extremely powerful. To use the sonar effectively, you not only need to make a click, you need to hear it. Any ear damage would be very dangerous; as some people have said, a deaf whale is a dead whale. Whales have got to look after their ears. So it seems highly likely that if a sperm whale's sonar system were directed at another whale's ears, it would be very dangerous for the receiver.

Imagine a group of 20-30 sperm whales feeding at depth, each making these dangerous clicks once a second. They are all in the same area so they need to be really careful. To me it is like having a bunch of hunters with machine guns out in the forest, they are firing away pretty continuously and they have got to have clear rules if they are all going to come out of the forest alive. So I think there must be some conventions they abide by about how you use these sonar systems. This, by some definitions at least, is morality.

PH At the University of Plymouth, we are staging a symposium and exhibition about the whale, trying to bring together those who think about whales from the point of view of art or literature and those who work hands on. You said earlier that you thought it quite helpful for scientists to look at whales from an artist's perspective. For me as a writer, the big problem is anthropomorphism – to write about animals from a human perspective it is difficult to not be anthropomorphic as it is, really, the only way we humans can describe them. The whale, and especially the sperm whale, invites that because we know so little and because the science around these animals is so recent, only about three decades old.

Could you project, perhaps to 100 years, where we might reach with whale science? How would you want science to move forward and what tools do you need?

HW Whale science has been very tool-dominated – unlike the study of apes, but then they are much closer to humans both in terms of evolution and habitat. We have been able to find out things we couldn't have without these nice devices, but it also means that we sometimes get sidetracked thinking about the technicalities, rather than the animals.

An important recent development is that sophisticated tags can be put on whales for short periods, and these can give a detailed perspective on the behaviour of the animals. Ten years ago we had almost no idea how a sperm whale feeds at depth but because of these tags, and other technical advances, we are getting a good idea. It will be very interesting when the power of these techniques is brought to bear on the social end of things – the role of culture, the role of the brain in whales; this is where the puzzle is biggest and hardest.

We can see how heart rates change as different things happen to the animals socially – when they meet a friend, hear an unfamiliar pattern of clicks. We can look at how they interact with each other physically – how does the hearing of the clicks change their movements in subtle ways? Then this information can hopefully be related to individuals. As we build up knowledge on histories of the individuals we can see how they vary and how these differences affect society – personalities and so on. I see this as the way forward.

But I am likely to be wrong, and there could be an idea developed, perhaps by an artist, which allows us to figure out a way to show some new dimensions of the whale. The culture and brain of the whale is a vast mystery and something really hard to get at scientifically.

PH We are talking about human culture meeting whale culture. That's what I try to do when I write about whales, but it is what you physically do by being in the field with them. And there is the aspect of the way in which our culture affects whales – historically through hunting and now through the way we are changing the planet. (For instance, when I was in Maine recently, I learned from Dr John Wise there that sperm whales, because they inhale so deeply, may be breathing in heavy metals in the air.)

The dilemma is, are we changing whales by observing them? Whale-watching raises the animal's profile but are we influencing its behaviour? In the future, will that meeting of cultures be reciprocal? Can we reach a point where we understand the whales and they can communicate to us?

HW That's a tough one! Some of the more profound and interesting things we know about whales have come from animals in captivity. This, of course, is getting more and more problematic as captivity is a very unreal environment for the whale or dolphin.

But how do we go on if we don't want to keep whales in captivity yet all our methods for learning about their cognition and communication are developed for captive animals? Lori Marino and Toni Frohoff [two biologists] have proposed that we try to take these techniques into the wild using animals that seek out humans. This is controversial and very challenging. But if it can be shown to work we can study cognitive processes in whales and dolphins in their natural environment, in situations where we have enough control over what is happening to get a feel of what they are thinking, how they are making decisions.

Robert McCrum on Herman Melville, Books, Page 48

The Peninsula Arts Whale Festival runs from 18-20 Feb at the University of Plymouth. A collaboration between Hoare and the artist Angela Cockayne runs until 5 March at the Peninsula Arts Gallery