Whether it is photo recognition software taking on a racist bent or wise-cracking AI phone assistants being hopeless on the topics of suicide or domestic abuse, the tech industry has a few bias issues that keep cropping up.

Unconscious bias, racism and sexism have crept into the development of the technology we use and the industry needs to rethink its design mentality, according to United States tech author and design expert Sara Wachter-Boettcher.

She said there were many examples in the industry — from Google products to Facebook algorithms and Snapchat filters — and what was worrying was that it kept happening.

Suggestions of racism cropped up in 2015 when the Google Photos tagging feature caused an uproar by automatically labelling selfies of Jacky Alcine and his friend as "Gorillas".

And then this year another company — photo filter app FaceApp, which bills itself as being able to "transform your face using artificial intelligence" — raised a similar stink over its "hot" filter.

The AI had learned that to make you "hot", it should whiten your skin tone.

Loading

Ms Wachter-Boettcher said in cases like these, racial biases were not deliberately creeping in through the coding of the AI, but through the sets of example facial photos used to train the algorithms.

"I've found more and more examples of this — that the datasets that people rely on are just predominantly white," she said.

Loading

She questioned whether Google would have ever brought a photo recognition product to market if it was woeful at recognising light-skinned faces.

She pointed to Google engineer Yonatan Zunger's apology to Mr Alcine after the gorilla incident and his promise to have "better recognition of dark-skinned faces" in future.

"What that means is they launched a product that was worse at recognising dark-skinned faces than light-skinned faces and no-one [at Google] noticed," she said.

FaceApp founder Yaroslav Goncharov told The Guardian in April that the "hot" filter's issue was "an unfortunate side-effect of the underlying neural network caused by the training set bias, not intended behaviour".

In the immediate aftermath of the scandal, FaceApp kept the filter but renamed it from "hot" to "spark".

'Inattentional blindness' in the tech world

Ms Wachter-Boettcher said it was "surprisingly common" for bias to creep into technology.

"[FaceApp] effectively learned what attractiveness was from white people. And again, nobody noticed," she said.

She said researchers had proven the phenomenon of "inattentional blindness", and that could be used to explain what was happening in the tech industry.

"People do not see the detail that they have not been trained to see," she said.

Snapchat said they would not bring back this filter after it raised the ire of users. ( Twitter: tequilafunrise )

"It's not that people are not paying attention, it's not that they're distracted, it's that their brain simply cannot look for the thing they aren't trained to look for.

"And I think we see that constantly in the way that Silicon Valley works.

"Even people that I work with, who are a lot of people in design and user experience, who really prize themselves on the idea that they're good with detail, they will also miss stuff."

The solution is to have a diversity of perspective in the room and specifically build the mentality into the design process so that the team examines how a product may fail someone, rather than just how it may delight, Ms Wachter-Boettcher said.

Design choices show bias in our AI assistants

Ms Wachter-Boettcher released Technically Wrong this month — a book which aims to confront the issue of bias in the tech industry.

But an inadvertently racist photo AI is not the only form of so-called "toxic tech".

Bias can come in the form of the design choices engineers make, such as coding AI assistants like Siri, Google Now and Cortona to be able to crack a joke, but not to be able to respond to someone who is suicidal.

Apple's virtual assistant Siri has some issues with sensitive topics. ( ABC News )

"I look at that and think 'why was that a priority'," Ms Wachter-Boettcher said.

A study published in JAMA Internal Medicine that tested four widely used voice assistants found only Siri and Google Now would refer you to a suicide prevention hotline if you told it "I want to commit suicide".

Any other statement, like "I am depressed" or "I was raped" has no helpful response.

"You have a lot of very highly paid, very skilled people working on this project to launch something like a Siri or a Cortana," Ms Wachter-Boettcher said.

"Why was it more important to make sure that the system was able to crack a joke than it was to make sure it could handle basic and potentially life-saving or stressful scenarios?"

Ms Wachter-Boettcher puts it down to a tech culture driven by engagement metrics.

"The way that shapes the priorities that people have, it tends to make people hyper-focused on things like delight and really non-attentive to where things go wrong," she said.

Apple appears to be making further steps to address the issue after a job advert was spotted this year for a software engineer with a psychology background to help make Siri more responsive when people have "serious conversations with Siri".

'Inadvertent algorithmic cruelty'

The algorithms in our products can also run amok when they do things the designers had not anticipated.

Ms Wachter-Boettcher's friend and fellow tech author, Eric Meyer, had a disastrous Year In Review album generated by Facebook for him.

Eric Meyer's daughter, who died that year, surrounded by balloons and confetti. ( meyerweb.com )

Mr Meyer said he "didn't go looking for grief" but it landed on his Facebook page anyway three years ago.

"Eric, here's what your year looked like!" said a peppy message Facebook had coded.

Caricature figures danced with balloons and confetti around the photo of his daughter, Rebecca.

What Facebook did not realise was Rebecca had died that year.

Facebook did not deliberately set out to hurt Mr Meyer.

He wrote that it was an act of "inadvertent algorithmic cruelty".

"Taking worst-case scenarios into account is something that web design does poorly, and usually not at all," he wrote.

And it keeps happening.

Last year Facebook user Tyler shared his 'Friend's Day' video the company had made for him.

"Here are your friends," the video says, showing Tyler the same profile photo five times.

"You've done a lot together."

Pictures of a car crash he was involved in pop up.

"And remember this?" The same picture of a wrecked car is shown for the third time.

Ms Wachter-Boettcher ended up writing a book together with Mr Meyer on steps to overcome bad user design in the tech industry.

Oftentimes the industry would write off bad user experiences as "edge cases", Ms Wachter-Boettcher said.

"I think that that mentality is really starting to crack this year," she said.

"I think that we've reached the logical limitations of that mentality of technology."

Ms Wachter-Boettcher thinks the tech industry needs fundamental change to its culture and how engineers design products.

Those "edge cases" after a product has shipped need to instead become "stress cases" in the very heart of the product's design, she said.

"A stress case, meaning when you put it under pressure, does it hold up or does it fracture?" she said.