Norman Swan: When cancer recurs after initial treatment or has already spread, the prognosis can be poor and a cure very hard to achieve. That's because the cancer is battle hardened and harbours drug resistant cells. Of course there are individual variations, and some so-called metastatic tumours, those are tumours which have spread, respond better to therapy than others. The approach of cancer specialists is usually to go in hard with chemotherapy using repeated courses to try to obliterate the tumour. But the statistics show that for most cancers that's eventually doomed to failure. The outcomes haven't changed much in decades, which has prompted some researchers to argue for a complete rethink oriented around the concept of game theory.

Game theory is about decision-making in the face of an opponent of some kind. If you've seen the movie A Beautiful Mind about the Nobel prize-winning mathematician John Nash, you'll have been exposed to one form of game theory, which is about two players in a game who are aware of the other's strategy and thinking. With cancer though it's a different kind of game, also studied by mathematicians, where one player is the leader and the other the follower. The irony is that at the beginning of the game the cancer specialist is the leader. The tumour has no idea that outside the body there is this person in a white coat planning its destruction, and the cancer certainly can't guess the doctor's strategy. It's only able to respond. That should mean that the patient and the oncologist will win the game because they start with such a powerful advantage. But according to critics of the current ways of treating metastatic cancer, the doctor fritters away the advantage and very quickly loses the game and the patient's life to a tumour which can't think for itself but exquisitely utilises evolution to resist attack.

Katerina Stankova is a theoretician interested in evolutionary game theory. Katerina is at Maastricht University in the Netherlands, and, as she explains, the cancer specialist also doesn't exploit a second game going on within the tumour itself.

Katerina Stankova: In the context of cancer this is very relevant because actually you have different types of cancer cells which interact with each other and then they fight for resources, they have a really high mutation rate.

Norman Swan: So you've got a lump of cancer cells, and within that lump of cancer cells there's a power game going on about the most powerful cancer cells surviving, but there are millions and millions of them with different susceptibilities to drugs and environment.

Katerina Stankova: Indeed. When we talk about metastatic cancer, they have mutated in many, many possible ways. They can cooperate with each other, they fight against each other. Whatever you do to the system, they are trying to adjust. Not that they are rational, but in this context of evolutionary game theory, if the mutation is not profitable for them it will just not spread in the population. But if it is profitable for them then they will proliferate and survive.

Norman Swan: So in a sense while it's very adaptive and evolutionary, it's a dumb process, they're not thinking, they're just responding.

Katerina Stankova: Indeed. So it's not that the cancer cells think, but whatever you do to them they either adjust to it and then they survive, or they don't and then they die off. So you are basically training the system to adjust to whatever you would do to them.

Norman Swan: That's the cancer that the person has got, and if the cancer has spread it usually means that it has mutated and is a bit nastier than the original cancer from which it has spread. So that's a game going on within the cancer. But add to this game now you've got a rational person, you've got the cancer specialist, you've got the doctor.

Katerina Stankova: Yes, and then basically whatever the doctor does it will influence the system. Either this works and then basically you decrease the population of the cancer cells, or it is ineffective, and usually it is ineffective because some of the cancer cells evolve resistance to this drug.

Norman Swan: So in a sense, in this game, because the doctor is rational and can think and the cancer cell is dumb, the doctor has an enormous advantage to begin with.

Katerina Stankova: You would think so, right? And if this rational player, the doctor, took this into account, then from game theoretical point of view there has to be strategy, a rational strategy of the doctor which will lead to the win. But this is not really done in the current standard of care. When it comes to metastatic cancer, the success rate is really…the life expectancy is about the same as 30 years ago. So what I am claiming is that while you have this rational player, the physician which could use all these advantages of rationality, playing first, they don't do it and that's a problem.

Norman Swan: Katerina Stankova.

So with standard, repeated and inflexible courses of chemotherapy, the game flips and the winner becomes the loser. If oncologists were to accept that they could dominate this evolutionary game, how would they play it? The answer lies in the game within the tumour between drug resistant and drug sensitive cells. You want the sensitive cells to win, which means, rather uncomfortably, that the tumour actually needs to be allowed to survive.

Katerina Stankova: The standard protocol is that the physician chooses the drug at maximum tolerable dose, and this drug is kept being applied until there is an inevitable evidence of cancer progression, or until you get a very, very high toxicity and the patient cannot take it. So while the doctor could be strategic here and try to foresee what type of drug he or she could apply and what would be a possible reaction on the cancer cells, instead you choose one combination of drugs, you go for a maximum tolerable dose, which is actually the maximum level of the drug which the patient can take, and then you keep playing the same strategy until you see it does not work.

But if you now look at the game played among the cancer cells, this is like the ideal situation for them because you change their environment but you do not surprise them. Basically you do all the time something the same, while actually they keep adjusting to your drug because they keep evolving while I play still the same strategy. So basically I am training the best enemy I could have. So in the end I will end up with a tumour I cannot treat anymore.

But it might be actually better to start to be a bit strategic. If I know that there are some cancer cells which actually react to my treatment and some others which do not, then maybe it is not the best idea to actually kill all those cancer cells which actually do react to my treatment.

Norman Swan: You argue you could actually play the game within the cancer cell more cleverly.

Katerina Stankova: Yes. If the cancer cells evolved into resistance, they changed somehow their metabolism. For our cancers we looked at, this change which leads to the resistance has a cost, meaning that if you do not apply the treatment you would see that sensitive cells actually grow more.

Norman Swan: One example of how this would work in practice comes from a recent trial using evolutionary game theory in men with resistant metastatic prostate cancer. The researchers gave standard treatment until the tumours shrank by half. Then rather than keeping going with repeated courses every few weeks on a set basis, they actually paused the medication until they saw the tumour growing back. Then they administered another course until they saw shrinkage, at which point they paused again. So at no time was the tumour obliterated. They were playing the cancer to encourage the growth of a higher proportion of drug sensitive cells. The findings were impressive. It took twice as long eventually for the cancer to recur, and the amount of chemotherapy used was about half, with the men consequently experiencing a higher quality of life. There are also trials going on in lung cancer where the strategies based on switching medications, and in metastatic thyroid cancer using switching which involves immune targeted drugs. Katerina Stankova is at Maastricht University in the Netherlands.

So are oncologists ready to change their minds? In our Melbourne studio is Grant McArthur who is a researcher at the Peter MacCallum Cancer Centre and executive director of the Victorian Comprehensive Cancer Centre. Welcome to the Health Report Grant.

Grant McArthur: Good evening Norman.

Norman Swan: So are you ready to change, throw it all out, game theory in its place?

Grant McArthur: Well, I think that game theory is very applicable to the way we are thinking about resistance of cancer cells. Of course one won't change the routine practice until the evidence is in, but the mathematical principles backed up by some biological experiments do support this approach. So one example of that plays out in my field, which is the treatment of advanced melanoma where we've had some dramatic improvements in outcome for patients with metastatic melanoma, particularly targeting a mutated protein called BRAF. However, in the vast majority of patients it does start to grow again, using that maximum tolerated dose approach that Dr Stankova spoke about. So we are now looking at ways about tricking the cancer early and applying game theory so that the oncologist with the patient can stay ahead of the game and take control back.

Norman Swan: So how do you do that?

Grant McArthur: So what is critical is to understand what happens early after you apply the drug to the cancer. What are the changes happening in the cancer, study that, and then if you understand the biologic processes happening there, then you know how to tackle it. So I'll give you an example. So we are currently doing a clinical trial of adding immunotherapy on top of the BRAF targeted therapy.

Norman Swan: So we should just explain here, so BRAF is the gene that is going wrong in some of these people with advanced melanoma, and there's a drug which actually attacks the BRAF mutation, but that's nothing to do with the immune system…

Grant McArthur: No, that's right.

Norman Swan: So it's a targeted chemotherapy drug, but alongside it there's these new drugs which block the process in the immune system which stops the immune system attacking the cancer, so you're talking about adding that to the mix.

Grant McArthur: Yes, so getting in before there's any ability of those cells to change, adapt and grow out, bring an extra weapon in being the immune system, not when in fact progression has already occurred, because we know that there are some similarities between resistance to immune therapies and the cells that grow out. Instead getting in early, staying ahead of the game and preventing the emergence of resistance.

Norman Swan: I mean, this comes to the tests that you might do. So what Katerina Stankova was talking about there was a pretty crude measure, which is is the cancer coming back, you watch it, hold your water, and then put in another low dose of chemotherapy to control it down, and eventually you are training the cancer to be more drug sensitive. What are the tests that you are doing that predict this resistance or give you a strategy?

Grant McArthur: Yes, so what we want to do is have tests that show those mechanisms of adaptation happening early. So in the future to apply game theory most effectively in this model what we would do is do some special imaging, some biopsies of the tumour, understand all the characteristics of the cancer in an individual patient and the way it's adapting and then bring in the correct therapy to stay ahead of the game and prevent the cancer emerging. So to go back to the immunotherapy example, so when you treat with this BRAF inhibitor, some patients there is a secondary immune cells moving into the cancer and in other patients there are not. So if you found on your biopsy that the immune cells are moving in, ha ha, that's an opportunity, let's stay ahead of the game and add in immunotherapy or even switch to immunotherapy early.

Norman Swan: And you call that hot, the cancer goes from cold to hot. What about hard to treat tumours which can be dreadful, like pancreas, ovary, brain cancer, where you really do want to attack very difficult to treat tumours. And again, you tend to go with chemotherapy, and in fact traditionally the chemotherapy for pancreas has been very…they've used this drug called Gemcitabine which has been very poorly studied rather than a more perhaps intelligent approach. Does it work there too?

Grant McArthur: That's an excellent question. Not to date because we don't have so-called targeted therapies or immunotherapies for those cancers that we know how they work, because this game theory can be studied in the laboratory prior to getting into a patient, and you can start to predict through your laboratory studies what is going to happen in a patient. So if you don't have good targets, and we don't have great targets unfortunately at this point in time for pancreatic cancer and brain cancer, at the moment it's a challenge to try to understand how you would apply game theory. That's not to say that it isn't worth studying with the available drugs, but they don't work that well, they don't put a lot of evolutionary pressure on the tumour to change because upfront they don't really work very well in the first place.

Norman Swan: Because they don't get in in the first place. Many years ago I had an astrophysicist on the show from Stanford University, Stephen Schneider, who actually discovered the ozone hole, so not an insignificant brain, and he got lymphoma, mantle cell lymphoma I think it was. And he was being treated at Stanford, a pretty good place to be treated if you've got mantle cell lymphoma, you'd have to say. But he was shocked to discover then that his cancer specialists at one of the best universities in the United States did not use decision-making theory, which is part of game theory, they weren't being trained in it, and as a result were making what he thought were very silly decisions. There's so much variation…I'm getting on my hobby horse here, I haven't got much time left…there's so much variation in clinical oncology in Australia between…you know, you'll go to Melbourne and you'll have your gene sequenced, you'll go to Sydney and you won't, you'll go to Brisbane and you will. There's so much variation, are you teaching cancer specialists how to think and make decisions sensibly?

Grant McArthur: Yes, look, this takes time to change the health system, we've got to get the evidence first that this approach works. It's still very much in the research setting, so there are clinical trials underway to test out these principles, but until those clinical trials report, it won't change practice. I am specifically now thinking about talking about game theory. What though is changing very rapidly now is personalised medicine whereby there is rational decision-making with profiling of the cancer, with gene sequencing or looking at what proteins are found on one patient with a particular type of cancer versus another patient and then tailoring the treatment accordingly. So things are getting more personalised. And in many ways an application of game theory is taking that to a whole other level because there's really an understanding then not only of what the cancer looks like when you start treatment but how it evolves, and that would be the next generation. But I think it's exciting prospects, Norman, to apply these principles to help us overcome cancer.

Norman Swan: Grant McArthur, thank you for joining us.

Grant McArthur: My pleasure.

Norman Swan: Grant McArthur is a cancer and clinical trials researcher at the Peter MacCallum Cancer Centre in Melbourne, and he's also executive director of the Victorian Comprehensive Cancer Centre.