Antony Funnell: The idea of a fixed price for goods and services has long struck me as a bit of a western thing. Travel in other parts of the world, and most things are up for negotiation. In fact I once got chased down a street in Fiji by a confused and angry shopkeeper who simply refused to believe I wasn't bargaining when I said NO.

Hello, welcome to Future Tense, I'm Antony Funnell.

Sven Brodmerkel: The idea of fixed prices is historically a rather recent one. So at the turn of the 19th through 20th century the big retailers in the United States, Macy's and Wanamaker's, decided we want fixed prices to a certain degree or extent because they decided it was a kind of moral argument being made here, meaning everyone pays the same price.

Antony Funnell: Assistant Professor Sven Brodmerkel from Bond University.

Sven Brodmerkel: At the same time it was a kind of mechanism to make business at that time more efficient by not having to train clerks in the art of negotiating or haggling with consumers. Fixed prices make it much easier to advertise products, and fixed prices basically made at that time forecasting for businesses much easier. Nowadays of course with algorithms they take over all this work and basically make it much easier for companies to introduce what we would call dynamic pricing.

Antony Funnell: Dr Brodmerkel is with the Department of Advertising and Integrated Marketing Communications. He's been taking a close look at the way 'dynamic' pricing has been developing, crucially what impact it might have on your wallet or purse.

It already operates in certain online interacts, buying a plane ticket is probably the best example, or booking a holiday, the price can be different for different people depending on a whole range of variables, like time of booking for instance. And Dr Brodmerkel sees the individualisation of pricing becoming a whole lot more sophisticated.

Sven Brodmerkel: I would expect it to become much more widespread just because now technology allows for this highly sophisticated dynamic pricing mechanisms actually to work, because most of these dynamic pricing mechanisms depend on artificial intelligence, machine learning and huge datasets, and we are just probably at the beginning of this trend towards how can technology make use of all these huge datasets and train machines to perform certain tasks. At the same time it will be interesting to see from a moral argument to what extent consumers will accept dynamic pricing and in what kind of categories. At the moment, as we talked about in the airline industry and hospitality, we are kind of used to it.

Just a couple of days ago it was a news story that Uber is now moving from surge pricing, meaning just trying to match supply and demand, to what they call root pricing, basically saying we would try to find out how much individual people are willing to pay for a certain service. And if we can do this…and this would be the dark side of the story, we can increase our profits because we can basically figure out what are people's demand curves, so to speak, in economic language, and how can we get them to pay as much as they are actually willing to pay.

Antony Funnell: And what will those individual decisions about pricing, what will they be based on? Say in the case there with Uber and its ride sharing?

Sven Brodmerkel: It might just depend on a location. Basically if someone is booking a ride from a rather wealthy suburb, going to another wealthy suburb, the algorithm would probably determine, oh, this is someone who has more money than other people and might be willing to pay more. Or it might be, with technology advancing, much more sophisticated.

To take another example, Amazon would look at your purchase history and your web browsing history and try to determine, okay, how willing is this particular person to negotiate, to shop around, to look, to check for cheaper offers elsewhere. And if the algorithm determines this person usually doesn't shop around very much, it could offer you products at a higher price because it appears that you are willing to pay more or that you are not that price sensitive as other people might be. Or various other factors.

Antony Funnell: Many people mightn't have so much of a problem if it's rich people being charged more for products or services, but there are ethical considerations that come into all of this. If you are talking about things like profiling of people or, say, charging more if you are, say, a rideshare service, charging more to take somebody to a less desirable area of the city.

Sven Brodmerkel: We could easily imagine discriminatory pricing behaviour. You could, for example, have an algorithm which determines that the person who is shopping now is a single mother, she is probably time poor, and she is not really willing to go to various shops to buy the things she actually needs, so she will just go to this one store and this one store could then raise prices or at least don't offer certain discounts because basically the information says this person is time poor or literally poor. Say you have elderly people who aren't that mobile, who basically depend on very limited set of retailers. And again, we could see some sort of price discrimination which would then rely not so much on saying someone is particularly wealthy but more on someone is in a certain need.

Antony Funnell: Or a vulnerability.

Sven Brodmerkel: Less able to go to a competitor, exactly, and then this could potentially be exploited as well.

Antony Funnell: One of the interesting things that we've seen and we've covered on this program just recently is the move towards virtual assistants. You know, Google, Facebook, Amazon, they are all bringing out their own versions. What sort of dynamic do they throw into the mix when we are talking about individual pricing?

Sven Brodmerkel: This could be a really interesting dynamic because we might in this situation see not people negotiating with each other but algorithms trying to negotiate with each other and each algorithm has its own kind of predetermined set of rules and we would now basically see a kind of economy in which algorithms would do the shopping for you, try to negotiate with other algorithms on what the best offer for you might actually be.

There was a really interesting and famous case a few years ago on Amazon where we saw an unexpected outcome of two algorithms negotiating with each other. Two book retailers had independent dynamic pricing mechanisms. So one retailer said, okay, we have so many favourable good customer evaluations, we can afford to always be a little bit more expensive than our competitor, say about 10% more expensive.

Another retailer with less favourable reviews decided we always need to be a little bit cheaper than this particular competitor, so we will implement a pricing mechanism that always tries to be 5% lower in price than the other retailer. And once you get this dynamic into play, we basically saw over days the price of one particular book, and it wasn't even a bestseller, more a book on genetics, a kind of scientific book no one would expect to become so expensive, over a couple of days this book actually was listed on Amazon for a price of about $23 million. Until they detected, oh, there's something going wrong, there's a kind of certain dynamic here between algorithms trying to outdo each other, and then they fixed this particular problem. But this is an interesting and kind of funny example but we can now expect or at least imagine other scenarios, much more complicated scenarios in which multiple algorithms try to negotiate with each other what is basically the best price for an individual person.

Antony Funnell: So an extreme example but a reminder that these systems are not foolproof.

Sven Brodmerkel: They are probably not foolproof, and it's all a little bit still trial and error. And, as you said, once these dynamics get started, it's sometimes pretty hard to unravel what is actually going on.

Antony Funnell: So in that sense then we are removing the individual, the human from that negotiation price, which is going to be a very interesting dynamic.

Sven Brodmerkel: It would be, and again, people who are more positively inclined towards these technologies might go, oh, if we get these individual algorithms right, they might actually prevent us from making irrational decisions because we can program algorithms in a way that they don't fall for certain kind of fallacies we as humans might all share.

Antony Funnell: They're more rational, in a sense, about their decision making.

Sven Brodmerkel: Yes exactly, so when we think about this whole field of behavioural economics which teaches us that we fall for certain nudges or irrational thinking, heuristics so to speak. In a positive scenario these algorithms would prevent us from making these kinds of decisions. On the other hand, the fundamental problem with algorithms or artificial intelligence is obviously that we probably don't know because these are black boxes, so to speak. We usually don't know exactly how they come to make certain decisions. So it might be an interesting and maybe even scary dynamic developing.

Antony Funnell: This is Future Tense, I'm Antony Funnell, and we are speaking with Assistant Professor Sven Brodmerkel from Bond University in Queensland.

This dynamic pricing, this move towards dynamic pricing, we've largely talked about the online world but it is also expected to have an impact on the off-line world, on traditional commerce isn't it.

Sven Brodmerkel: Yes, we could at least imagine a future scenario. So, for example, late last year Amazon did a better trial of what they call Amazon Goods, basically a shop without checkouts. It would work with an Amazon app which would identify this particular consumer or customer. A customer walks into the store, picks products he wants and just walks out of the store and then he or she gets billed afterwards. This is interestingly the reverse of how it works at the moment in supermarkets. So even if you have a loyalty card, you walk into a supermarket, you put all your things in the basket, you go to the checkout, and only if you use your loyalty card and only then you are basically identified as a consumer, as person XYZ. So it's pretty difficult to actually introduce dynamic pricing based on this system.

But once you as a consumer, you walk into a shop and you are identified while walking into the shop, basically it offers lots of opportunities for the company to have a look at your profile, your purchase history, all the other behavioural data they might have, and then tailor your shopping experience or your prices based on this particular knowledge. So we might imagine that two people who walk into a shop that works like that at the same time might end up with a different bill, although they might have bought the same range of products or the same products.

Antony Funnell: And with this Amazon Go model, part of the rationale for it is that it saves you time, you don't have to worry about things like checkouts, it's just all done for you automatically, which a sceptical person might say means you are probably less likely to check how much you paid for things. So that on top of this dynamic pricing could mean that you potentially could pay significantly more than somebody else for the same basket of goods.

Sven Brodmerkel: Potentially yes. I guess as discerning consumers you can't overdo it obviously as a company, and you would usually probably still try to work more via discounts than via raising prices. Raising prices always looks bad. But if you can work with certain discounts, which is basically the same thing in reverse, it might work quite perfectly.

Antony Funnell: So there are cautions, but what are the advantages for consumers if we do see a move towards individual pricing, and how are average consumers likely to behave? Because a lot of people see being able to shop around for a price online for airfares, say, in a positive light, don't they.

Sven Brodmerkel: Yes, of course, we all feel pretty good if we think we've got a great deal. The danger though is that once dynamic pricing based on artificial intelligence comes in, the overall game is that you will lose because these algorithms will try to get the maximum out of you. So even if you think or feel that you've got a great deal, overall you probably won't.

Antony Funnell: As a consumer?

Sven Brodmerkel: As a consumer.

Antony Funnell: You're not going to know because they're not going to tell you what they've actually based the determination for their valuing on.

Sven Brodmerkel: Exactly. In marketing we always talk about the lifetime value of a customer. So you can of course offer certain deals and incentives if you are convinced that this particular customer will stay with you for a long period of time and then you will make up for the discounts later on. And the more sophisticated these kind of pricing mechanisms become, the more likely it is that you will walk out of the shop and think you've actually got a great deal, but in the end the retailer, the algorithm is convinced that it will make up for it in the long term anyway.

Antony Funnell: We know that in the off-line world there are lots of regulations around the way in which retail occurs. Those regulations are still a bit of a work in play, aren't they, in the online world. What are the implications there for regulators from this kind of change, where do they fit into this?

Sven Brodmerkel: The problem for regulators I think with almost all things digitally impersonalised these days is basically to find out what's actually going on. As far as I know so far the law hasn't found a way to deal with these particular instances. I guess one of the only ways…and some people have asked for it for quite a while) would be to introduce a kind of independent auditor of algorithms. So you would basically have to have a watchdog who would be able to inspect and determine how these algorithms used by retailers or financial companies or advertising agencies actually work. But then again, once machine learning, artificial intelligence comes into play, even people working in Silicon Valley would now argue we can't always tell why machines come to certain decisions because we don't exactly know how they make these decisions. It will be quite difficult to find a way to work with these new technologies.

Antony Funnell: Dr Sven Brodmerkel, thank you very much, it's been fascinating.

Sven Brodmerkel: A pleasure.

Antony Funnell: The systems we build just continue to get smarter. Increasing they're designed to learn from their mistakes.

A lot of nonsense is written about the current capacity of Artificial Intelligence, but it's clear to anyone who's been following developments in AI technology that, as Dr Brodmerkel just said, sometimes the decision-making processes that machines use to great effect are a mystery even to the machine's own creator.

Whether AI ever really gets to the point where it has human-like awareness or sentience is a debatable matter. Certainly there are those out there who believe it's possible. There are also those who have doubts, but who believe it's time to start preparing for that day, just in case.

Max Daniel is the Executive Director of the Berlin-based Foundational Research Institute.

Max Daniel: Sentience arguably can come in gradations. So it's not like the first artificially sentient being we will create will necessarily be highly advanced or very humanlike. And science fiction might not be the best place to look at in order to get a realistic picture for how this may look like, because if you look at science fiction you may think of for instance the android Data from Star Trek which is a highly advanced being that looks like a human and can even talk and so on and so forth, and obviously we are very far away from creating those kinds of beings.

However, even today in those kinds of machine learning algorithms that are making rapid advances we have a technical architecture that utilises things like reward signals. And the worry is that if we add a little more complexity to such machine learning algorithms that can autonomously learn and act in some kind of virtual environment, then we may create beings that are able to suffer but in other respects aren't highly advanced. And I think this amplifies the risk for two reasons. First, because we may create such beings in the near future than highly advanced AI beings. And second, because both beings may not necessarily be able to properly signal to us that they can suffer, that they are sentient.

And so we may worry about creating a situation not too dissimilar to the situation we see today with respect to nonhuman animals where we have a large number of sentient beings that presumably, there is a strong scientific consensus about this, that most of them have the ability to feel and to suffer but who lack some other thing, for instance human-like language, and therefore aren't able to communicate with us directly, protest and drive home the point that they need protection from our legal system. And so we may worry about creating a voiceless first generation, as it were, of artificially intelligent beings which could both be very dangerous given humanity's track record, and it could arguably happen in at least a few decades, if not earlier if you look at what experts predict about advances in the field of artificial intelligence.

Antony Funnell: Now, the idea of a piece of technology becoming so advanced that it might one day be able to experience suffering is certainly a difficult concept to get one's head around. But if it is a possibility, says Max Daniel, then it's time to start drawing up ethical guidelines for the development of AI technology. And what's called the Excluded Middle Policy could be one such safeguard.

Max Daniel: They excluded middle policy is that we should make sure that artificial intelligences we create are of one of two types. Either they are very simple, very unsophisticated and clearly not sentient without any reasonable doubt, or they are highly advanced sentient but in such a way that we can easily recognise this, for instance because they would also be able to use language and just tell us basically, hey, you've just created a sentient AI. And what we should avoid is something in the middle between those two extremes; either very simple and non-sentient or very advanced and sentient AIs. Because here the danger, as I've tried to outline, would be that we would create beings which would be sentient but not advanced enough in their other capabilities for us to notice. So in this middle ground arguably the risk of creating beings that would suffer without us even noticing would be greater, and therefore we shouldn't do this, we should avoid this middle and should stick to creating either very simple or very advanced AI systems.

Antony Funnell: Max Daniel from the Foundational Research Institute.

Another speculative suggestion that's been put forward is the possible future need to grant Artificial Intelligence what's called 'legal personhood'. Steve Wise is the president of a group called the Nonhuman Rights Project. For many years he's been arguing for legal personhood to be applied to animals, particularly the chimpanzee, our closest relative. But he also sees a potential case for AI.

Steve Wise: Human persons can have different sorts of rights. I may not have the same rights you have or an adult may not have the same rights as a child. There are all kinds of humans who have different sorts of rights. What personhood means is that you have the capacity to have rights at all. So if you are a legal thing, you simply lack the capacity to have any rights at all. If you are a person, it means that you are a rights container. It doesn't mean that you have any rights, it just means that you have the capacity for rights. And then once courts understand and agree that an entity like a chimpanzee or potentially an AI entity is a person, which means it has the capacity for at least one or more rights, then the argument switches to, well, what sort of being are you? What kind of being is a chimpanzee or is an elephant or is an AI entity, and what sort of interests, especially fundamental interests, might that entity have, and should those fundamental interests then be protected by a fundamental right?

But you could certainly have an entity who is a chimpanzee or an AI, an entity who has one right or two rights or five rights. You don't have to have 5,000 rights, you can just have one, two or five, it's really a case-by-case judicial decision, or then parliaments get involved and start talking about it as a matter of morality or public policy, what rights are these others nonhuman persons to have.

Antony Funnell: In the same way that a child has rights but not the right to vote within society.

Max Daniel: Yes, similarly an insane person who has certain kinds of fundamental rights…you know, you can't eat her, you can't kill her, but on the other hand she may not be able to have bodily liberties, she may not be able to go outside of an asylum. Or prisoners, they have certain rights but they don't have other rights, but they are all legal persons in that they have the capacity for rights. And the arguments are no longer do you have any rights at all, the argument is, well, which of the infinite number of rights that you could have should you have.

We do, by the way, oftentimes encounter judges who are clearly hostile to our idea. And it's very difficult sometimes for us to understand why they are. It seems to be almost be a visceral or an unconscious opposition to the idea of extending legal rights to nonhuman animals. Even chimpanzees…we humans have a large number of rights, and those entities who don't have rights are essentially our slaves, we can exploit them. We are essentially the masters of the legal things of the world. And history has shown us that masters don't like to give up their privileges. So I think that's really the main issue, is that it's a lot easier for us humans to be able to exploit any sort of an entity, whether it's a fellow human being, whether it's a nonhuman animal, whether it's AI. If that entity doesn't have any rights, then we can do with it or with her whatever we wish.

Antony Funnell: And from an historical perspective of course it's worth remembering that it wasn't so long ago that certain people because of their race and colour were also denied legal personhood. That was Steve Wise from the Nonhuman Rights Project. And the final word on the subject from Max Daniel.

Max Daniel: The general issue of ethical implications of artificial intelligence is definitely getting a lot more attention in recent years. But I would argue it's very important not to mix up two somewhat different debates here. So one issue which we have focused on is the question essentially; what might we do to those artificial intelligences? Could we inadvertently create suffering in them, for instance?

Another kind of discourse focuses not so much on the suffering of the artificial intelligences but just the power they could attain, irrespective of the question whether they would be sentient or not. And this is another issue which has received a lot of intention, for instance with people like Oxford philosopher Nick Bostrom or also a public figure such as Elon Musk or Bill Gates publicly warning about the risks from artificial intelligences becoming more powerful and intelligent but just in the sense of being able to do a lot of things, not necessarily in the sense of also having any kind of consciousness or sentience. And the worry they raise is more like could those artificial intelligences become at some point so powerful that they could do great harm either to themselves if they are sentient but also to humanity? So some people even think there is a risk of human extinction by advances in ever more powerful and ever more autonomously acting artificial intelligences.

And while those issues are conceptually separate, so, as I said, AI systems could be sentient without being highly advanced or being powerful, and also the other way around, AIs could be very powerful or autonomous without having any kind of consciousness or objective experience. But one thing that of course ties those two debates together is that they require some kind of idea of what progress in artificial intelligence is possible and how this could look over the coming decades. So I'm quite optimistic that if it's possible to get the attention of public figures such as Nick Bostrom, Elon Musk, Bill Gates and so on for this one kind of discourse about what could very, very powerful artificial intelligences do to themselves or humanity, then that it's also possible to raise this worry about sentience in artificial intelligence and the risks implicated in this. And in fact it's not like, for instance, Nick Bostrom never talks about these risks, he even coined the term 'mind crime' to warn about this risk about AIs becoming sentient and then potentially suffering on a large scale if we aren't careful.

Antony Funnell: And you thought the current world was already complex enough.

Max Daniel, and before him Steve Wise. We also heard today from Assistant Professor Sven Brodmerkel from Bond University.

Thanks to Karin Zsivanovits and Steve Fieldhouse. I'm Antony Funnell, until next time, cheers.