Eliza Watt: I'm Eliza. I go to St Catherine's. I'm turning 12 but 11. And I have synaesthesia.

Lynne Malcolm: Hi, it’s All in the Mind on RN. I’m Lynne Malcolm.

Eliza Watt: Some of my favourite synaesthetic experiences is with music. I see the colour shooting past my head to like a point in the distance, which is really cool. It feels like it's—especially in songs that I really like—sort of trying to take me to that point.

Lynne Malcolm: What colour is Monday for you? What about the number 4? Can you taste the colour purple, and does the saxophone sound orange or blue?

If those questions don’t sound too strange to you, you may be a synaesthete. Like Nina.

Nina Norden: Everything I hear, I see in shapes and colours. Even flavours and smells I can see. I can't even imagine how it is to not see a sound. Even silence has the colour more white that is like a void that I can see it as a shape. Always little sounds in the silence as well that I see. You know, like mini snowflakes. Every number has a colour and a shape to it and the year, the week—I can just make the list longer and longer the more I think about it.

Lynne Malcolm: More from artist Nina Norden in our next episode looking at synaesthesia and art. Today, how synaesthetes perceive the world.

Associate Professor Anina Rich is the director of the Perception and Action Research Centre at Macquarie University, where she heads up synaesthesia research.

Anina Rich: The word synaesthesia actually comes from the Greek syn, meaning together and aesthesis meaning of the senses. And so people have often described it as a mixing of the senses. In fact it's an umbrella term that can encompass a lot of different types of experiences that all fit within a sense of having an extraordinary response to a very ordinary stimulus. So, for example, people who have auditory visual synaesthesia have sounds of different types; it could be voices, it could be musical notes, it could be musical genres, evoking very specific and consistent experiences in the visual domain. So it might be moving pictures, it might be particular objects that have a colour and a form and a location that they see every time they hear that particular sound.

Perhaps the most common form is where people have vivid experiences of colour in response to letters, numbers and words—whether they see them, hear them, or even think about them.

Lynne Malcolm: So that's the most common mode?

Anina Rich: Yes. Although estimating the prevalence of any type of synaesthesia is quite tricky, because it's not a disorder. And so lots of people have synaesthesia and don't realise it. So with any of these are approximates, but certainly grapheme colour synaesthesia, where people have letters and numbers evoking colour experiences, very common. Also types of auditory visual synaesthesia seem to be relatively common.

There's also some other forms which relate to how we might perceive touch. So if you are seeing somebody being touched—people with mirror touch synaesthesia actually experience a sense of touch themselves. And there are some estimates that suggest that that may be actually the most common, but we don't know yet how closely that's related to the colour forms.

Lynne Malcolm: And some people, not only do they see colour in relation to letters and numbers, but they also see shape.

Anina Rich: That's right. So quite often we study the colour aspect because that's the bit that's maybe easiest to get a handle on, but in one of our studies we looked at the types of experiences evoked by just single tones on different instruments. And what we found was that far from just being colour, they were essentially geometric objects. So, for instance, one synaesthete a low note on the piano was a big round ball that she could see low down in space, whereas a high note on the same instrument, on the piano, was much smaller and kind of sparkly and a brighter colour.

And by studying people's experiences and getting them to really illustrate those complexities, we can start to get a sense of what the underlying patterns are. And what's interesting about this is that it actually seems that synaesthesia builds on mechanisms we all have. So for example if I play you a tone, a low tone on the piano and then a high tone on the piano and I show you two shapes—the shapes perhaps generated by my synaesthete—most non-synaesthetes will also say that the small, light-coloured, sparkly object goes with the high pitch, and the big, dark ball would be more likely to go with the low note.

So it seems to build on systematic, implicit patterns that we all have about low tones going with big, dark things; and high tones going with small, bright things.

Lynne Malcolm: So is there a sense that there is something that's in us all but with synaesthetes it's pronounced?

Anina Rich: It could be. One of the hypotheses that we've been working on is the sense in which synaesthesia might be an extension of a continuum. And so people like me who don't have synaesthesia might be kind of in the middle of the continuum, and synaesthetes who really experience these additional sensations might be at one end. And at the other end there's a phenomenon called aphantasia, where people don't have any mental imagery at all. And one of the working hypotheses that we're looking at at the moment is that perhaps what we're seeing is a spectrum of experience. And it's possible that synaesthesia is at one end and maybe aphantasia might be theoretically the other end of that same spectrum.

Lynne Malcolm: Anina Rich.

Eliza found out that there was a name for her way of perceiving the world when she was about seven. She was surprised that not everyone saw things as she did. I’m speaking here with Eliza, and later, her mum, Rachel Watt.

Eliza Watt: I also have colours for people. So everyone, even if I haven't met them, has an aura around them, and it's the colour of them. The first thing I base it on is what they look like and that gives me a colour. And then if I get to know them, what their voice sounds like. Usually doesn't change it much, but could change it just a bit.

Lynne Malcolm: Do you have an aura around me?

Eliza Watt: Yeah, you're yellow and orange. We actually talked about this before the interview. Mum showed me a picture and you were yellow, but then I heard your voice and now you're yellow and orange.

Lynne Malcolm: Really!

Eliza Watt: Everyone has an aura, even animals. And pictures and reflections of people. It's really cool 'cause it sort of gives me a first impression. I at first judge a person on their colour, not really what they look like or what they are like.

Lynne Malcolm: So are there particular colours that you like better and so you judge the person according to your favourite colour?

Eliza Watt: Not usually. But if I really like the colour then I might feel like they're going to be a nice person; or if I don't like the colour, not going to be a very nice person. But they only happen with my very favourite and worst colours.

Lynne Malcolm: Which are?

Eliza Watt: Well I like peach and rose gold but I really don't like the greenish, darkish brown—which sort of makes me want to shiver. And for those people I just feel like I sort of want to cringe. Which doesn't help if I'm meeting someone with that colour and I'm cringing for no reason and they're thinking, 'What's she doing?'

Lynne Malcolm: What colour do you have, yourself?

Eliza Watt: Actually I didn't think I had an aura until one day I was thinking about it and looked in the mirror and was like, 'Can I see…?' and then, 'Oh, there is an aura.' Which was quite a surprise, 'cause I didn't know. It's a peachy colour, sort of light peach, and sometimes a tiny bit of rose gold.

Lynne Malcolm: What about your mum?

Eliza Watt: She is a red-y-pinkish, which is like 6, which is her number. Or maybe a bit of 9, quite a mothering figure that looks after 2. And my dad is quite blue and a bit green sometimes. And he is a bit of 4 sort of character.

Lynne Malcolm: Did you know that, Rachel?

Rachel Watt: I didn't know I was a 6. I didn't know that Dad was a 4. It's interesting sometimes when Eliza's tried to explain to me about the numbers and the personalities and the colours—these things all overlap with each other. So she's explained to me that although a person's name written down can have a colour and they have an aura of colour around them, when I speak their name and she sees the colours that come from the sounds of their name, that adds another layer that can be different colours that add on to it. It's just so many layers of it it's hard to imagine, if you don't experience it.

Lynne Malcolm: Rachel Watt, and her daughter, Eliza.

The most studied variant of synaesthesia is grapheme colour synaesthesia, where a person perceives colours associated with numbers and letters.

Eliza Watt: With numbers I've noticed that usually even numbers have more red-ish, pink-ish, warmer colours, and odd numbers have colder colours. My lucky number's 2 because it's my favourite colour. It's peachy rose gold. With letters, they don't have personalities. Numbers have personalities.

Lynne Malcolm: Personalities? Really?

Eliza Watt: Yeah.

Lynne Malcolm: Give me an example of that.

Eliza Watt: Well 7 is not very nice. He's sort of a bully and 17 is also a bully but he doesn't do a lot of the bullying, he just advises7. So they're not a very nice pair, but 6 is like a mothering number, and that's sort of a red-y, pink-y colour. And 2 is really nice and little, but she wants to speak her mind to the bullies, like 7 and 17, but she doesn't feel like she can. She's not brave enough to.

Lynne Malcolm: Eliza Watt.

How often do synaesthetes complain about their multitude of perceptions that might interfere or distract into their lives—or perhaps even their learning?

Anina Rich: It's actually quite infrequent to find somebody who has a problem with their synaesthesia. If anything, most people think of it as, well, I would call it an unusual gift. And most people think it's just strange that we're even talking about it because isn't that the way everybody sees the world?

There are some people who do find it overwhelming, and that tends to be when multiple synaesthesia's going on, so there's auditory, there's olfactory synaesthesia, there might be tactile experiences and so on. But for most people it's okay.

I think there's a lot more research to be done in how it might affect learning and education, and that's something that I'd be keen to do in the future. Because I'm getting more and more emails from teenagers at high school, or teachers wanting to know, 'Is there a way that we can better utilise synaesthesia to help rather than to hinder?'

So for instance some people report things like, 'I'm not very good at maths because you can't multiply colours.' On the other hand another synaesthete said, 'I'm great at maths because I can already see what sort of realm of colour space the answer should be in, so I know if I get it wrong.'

Similarly, for things like memory, although the objective evidence is still mixed at the moment, about whether or not synaesthetes as a group are likely to be able to use synaesthesia to boost their memory, we have individual case reports where people will say, 'Well, I know that the person's name was Debbie or Paula, because it was a green name, but I don't know which one of those it is.' So they can sort of use it as a cue and then maybe it gets confusing there; whereas other people will say, 'Well I know that the address can't have been 78 because it was a blue colour, so it must have been 87.' So they kind of use their synaesthesia to scaffold recall.

Lynne Malcolm: There's also the suggestion that it can be a bit disconcerting when there's an incongruence between the synaesthete's perception of something and the reality. Just tell me about how you've studied that.

Anina Rich: In everyday life it probably just is a little bit of a 'Oh, that doesn't look right,' you know, Fire Extinguisher is written in the wrong colour, that sort of thing. But the consistency of synaesthetic experience is something that allows us to get an objective index of the experience. And it's based on that idea that incongruency might cause some change in behaviour.

So if I present in the lab a synaesthete with her letter colours, so if an A elicits red I've matched the colour for A and I present a nice red A. Then that's a congruent stimulus, and there's no mismatch between the synaesthesia that's evoked by the letter and the actual colour that she's seeing.

If I then present that same A, which evokes synaesthetic red, but in a blue colour, what we find is that if all she's doing is naming the display colours, she's slower in the incongruent case.

So what that suggests is that synaesthesia is involuntary in the sense that people can't stop it just because it's annoying or interfering in a particular way. I think in everyday life very rarely do people say that it actually affects their reaction time to driving or anything like that, but in the lab, I can use that slight difference in reaction time to give us an objective measure of the experience.

And from that we've then looked at other questions about, what's the role of attention in synaesthesia, when is it evoked, when can we stop it from being evoked—all of those sorts of things—by using that objective index as a measure of whether synaesthesia is occurring.

Lynne Malcolm: Associate Professor Anina Rich, from Macquarie University.

You’re with All in the Mind on RN; I’m Lynne Malcolm. And we’re discussing synaesthesia. It’s a condition in which one type of stimulation evokes the sensation of another. For example you may always perceive the colour blue with Monday, or the colour orange with the number four. You may even taste blackberries when you see the city skyline. It’s like a crossing of the senses.

Joshua Berger first became interested in synaesthesia when he was studying for a medical degree. He met a fellow student with grapheme colour synaesthesia—she would perceive specific colours with numbers. He was intrigued about whether she thought synaesthesia affected her learning.

Joshua Berger: She turned to me and said, 'It probably led to some of the affinities I had for subjects such as English or art, because they had extra colour. But mathematics never made sense to me. Why should a blue 3 times an orange 4 equal a green 12? All the other students would be doing black 3 by black 3 is black 9, or black 3 by black 4 is black 12. But for her, this lacked an essential component, and no-one ever addressed it.

She then told me that this led to a lifelong avoidance of maths, and she has really bad maths anxiety. It was a lack of the world reflecting her expectations or her embodied model of what she might or should be seeing.

Lynne Malcolm: Joshua Berger was inspired to take up full time research into synaesthesia at the University of Sydney. He and his friend decided to develop a coloured calculator to see if this would help her maths anxiety.

Joshua Berger: So we made a very early prototype which was based off the Microsoft '97 calculator, the very standard calculator there, and attached it to a colour wheel. So before she started the calculator program, she would click the digit 1, and then the colour wheel would pop up. And then she would select the colour.

Now initially we thought that this was just going to be pleasant, that this would be nice for her. Little did we suspect the reaction that was elicited when she finally used it. She wrote me a long, very emotive message, saying, 'This is the first time that addition and subtraction have made sense to me. It made my work about 80% easier. It was quite life-changing.' Being told her whole life that she had some sort of cognitive difficulties when it came to mathematics, for it finally to make sense, it was quite a big impact.

Lynne Malcolm: To further investigate how helpful a self coloured calculator could be, Joshua Berger set up an experiment. He devised a mathematics test for a group of people with grapheme coloured synaesthesia. Each person was allowed to choose their own colours on the calculator and he measured their speed and accuracy in the test.

Joshua Berger: So far what we've found is that in a study of 55 people with grapheme colour synaesthesia, there was a very, very large positive affective response by all the participants. Not a single participant disliked the calculator, with I think over 60% saying that they would definitely use one of these in the future, and asking me, 'When can I get this?'

However, we have to take those self-reports with a grain of salt because of effects I may have on the participants—experimenter effects—or biases they may have coming in to the experiment. We couldn't blind this experiment, the participants clearly knew that they were getting a colourful calculator which reflected their personal experience.

But it was encouraging that no-one disliked it, and we have some early evidence to suggest that yes, there are a sub-group of these people, people with synaesthesia, who are benefited by having their colours on a calculator. And as long as it's a small effect—and they enjoy it—it seems worthwhile.

Lynne Malcolm: Joshua Berger, who’s interested in exploring more applications for synaesthetes.

Anina Rich heads up research on synaesthesia at Macquarie University. They’re studying what is different about the brain of someone with synaesthesia.

Anina Rich: Traditionally people talked about synaesthetes as having hard-wired connections between areas that the rest of us don't, so that might be in auditory-visual synaesthesia the auditory cortex, the first place where you process sound, might have a connection to the visual cortex, perhaps, the area that is involved in colour.

There's now evidence that probably most of us—actually all of us—have connections between our sensory cortices in that way. So it's perhaps not so unusual to have that connection. What might be different is that synaesthetes might have stronger connections than the rest of us.

One of the theories that we've been working on is that actually synaesthesia is evoked by the concept of a letter or a number, certainly in grapheme colour synaesthesia where we've focused a lot. And in that case it may actually be more—rather than hard-wired connections between those early processing areas for sensory information—it might be the regions of the brain that are involved in holding our concepts of objects.

And perhaps it may be like knowing that a banana is typically yellow. So if I show you a grey-scale picture of a banana, you have no problem telling me that it's usually yellow. If I present you a purple banana you're actually slower to recognise it as a banana than if it's yellow. And in one of our recent studies that my PhD student Lena has been running, we've actually been looking in non-synaesthetes about that sort of colour activation when you see objects that difficult colours.

And the next stage is actually to look in synaesthetes and see, is the brain signal actually similar when synaesthetes are having these colours in response to letters. Because it may be at that conceptual level, so the regions of the brain that are involved in holding our concepts of objects may be the crucial ones for synaesthesia.

Lynne Malcolm: If that's correct, what are the implications of that?

Anina Rich: For synaesthesia I think it would be important in terms of understanding what it's most like. So there's been some studies that have made the case that synaesthesia's—it's different from colour perception but it's really low level in the way that colour perception is. But the body of the data seems to be that it's not the same as colour perception. We don't get the same sort of effects as we do when we put two colours next to each other, for example.

Other people have suggested that it might be like mental imagery, and so imagining colours, but maybe a very involuntary form of mental imaging.

A recent study with Rocco Chiou, one of my ex PhD students who's now at Harvard University, and Joel Pearson at UNSW, we looked at the nature of synaesthetic colours, and we've actually found that it's different from both of those things. So it doesn't behave the same as colour perception. It doesn't behave the same as colour imagery. It seems to be a unique type of colour experience. Now it's possible that it's somewhere between that and that knowledge about a banana being yellow, for example.

So by fitting synaesthesia into the bigger picture of how we represent perceptual features of objects, we can start to understand how the whole system works. Because of course the brain doesn't work in isolation, it doesn't have like—it has a little bit of the brain that's most interested in colour perception, for example, and we know that's important, but it works within a big system and understanding that system is crucial for understanding how the brain works, understanding how we perceive the world, and with that fundamental understanding then we can start to help people who have a deficit or an impairment in a particular type.

So, for example, semantics dementia patients lose the concept of an object. So they might look at a picture of a zebra and say, 'Oh, it's a stripy horse.' They can describe it but they've lost that concept. So by understanding how we represent concepts of objects, how we bring together features and integrate information, we can potentially have a much greater understanding of clinical conditions like that. And I think synaesthesia offers us a unique insight into that system that we don't have through other means.

Lynne Malcolm: Anina Rich.

And this is Eliza jamming on the saxophone with her teachers Benny and Lachie.

Eliza Watt: One of my favourite things about playing the saxophone is seeing the colours, and that's one of the times that I do pay quite a lot of attention to them. And they come from the back of my head and shoot off really fast to a point in the distance. Probably three metres away from me. Sometimes it'll be a bit late, like I'll play a note that's green, and then green will come a few seconds later. Or sometimes it'll be right on the dot and it'll happen straight away.

Lynne Malcolm: Do emotions affect the colours that you're seeing and the music you're playing?

Eliza Watt: Yeah, how I'm feeling definitely affects what sort of pieces I play, which affects what colours there are. But I don't think my emotions have a very big impact on my colours.

Lynne Malcolm: What colour does the saxophone connote to you?

Eliza Watt: Overall it's a yellow instrument, but the noises are often quite blue—dark blue and green. When I play it it's more yellowy-greenish. Sometimes a bit of pink because I like to play in the higher octaves. And that's quite a high colour for me.

Lynne Malcolm: Often the extremes of human experience lead to difficulty, but synaesthesia is not a disorder. Anina Rich says studying it gives a unique opportunity to learn more about human perception.

Anina Rich: Synaesthesia is important to study, because it gives us a unique insight into how perception might work and how we might represent objects. But I think more broadly it reminds us of the subjectivity of perception. So actually, you and I as non-synaesthetes kind of assume we see the world the same way. But we can never verify that.

We can look at the same thing and use the same words to describe it, and yet we can never know that we see the world in the same way. Your perception of it is interpreted through the filter of your prior experience, your knowledge about the world, and your emotions and your goals and everything at the time. And when we study synaesthesia it's a reminder that all of a sudden we use language to compare something and realise that we don't see the same thing. But that's just an extreme of actually what happens for everybody.

One of the things that synaesthesia might be able to tell us about is the way that the brain puts together information across the senses. And because we can use this unusual scenario to look at mechanisms that we all share, it pushes science along, I think, to study it from that perspective.

Lynne Malcolm: And it also raises a number of philosophical questions, as you've alluded to. How interested is the field of philosophy in synaesthesia?

Anina Rich: Yes, it's one of those nice fields where there's interest from a lot of different perspectives. So the philosophers do get a bit excited about the idea of having two qualia simultaneously, which shouldn't be possible.

So a qualian experience might be that you're seeing something that's coloured that's reflecting wavelengths of light that make you perceive colour, and that appears like colour in the world. I'm framing that carefully because actually colour doesn't exist in the world—we just perceive it, right? So there's that perceptual type of colour and then there's the synaesthetic colour.

And so philosophers have argued, well is that an instance of experiencing two qualia simultaneously. It's also of interest to artists, it's of interest to musicians, and I think broadly other types of scientists.

Lynne Malcolm: Associate Professor Anina Rich, director of the Perception and Action Research Centre at Macquarie University.

Before we go, Eliza Watt has something else to share about her perceptions.

Eliza Watt: I just realised that I've never told my Mum that colours also have personalities, 'cause it seemed so obvious that I didn't think to tell her or anyone else. Whenever I'm talking about synaesthesia, it's quite hard 'cause I'm thinking about all the most obvious things to me. I sort of have to think, 'Okay, this person knows nothing about anything that I'm seeing,' and then talk to them like they're an alien.'

Rachel Watt: That's right, actually. None of us know anything about what anyone else is perceiving, really, do we?

Lynne Malcolm: Rachel Watt and her daughter Eliza. And you can hear my full interview with them in the podcast version of this episode of All in the Mind.

Synaesthesia research depends on participants, so if you think you might be a synaesthete or want to find out if you are, head to the All in the Mind website for links to more information.

In our next episode, the creative side of synaesthesia.

Thanks to producer Diane Dean and sound engineer Isabella Tropiano.

I’m Lynne Malcolm. Bye till next time.