“Who is Peter Dutton?” The question from a young woman, sitting in an eatery 500m from the immigration minister’s office, is not an uncommon one in Dutton country. His electorate of Dickson on the northern fringe of Brisbane, which spans outer suburban “aspirational” areas such as Strathpine to the well-to-do semi-rural retreat of Samford Valley, has the distinction of boasting one of the lowest levels of voter engagement in all of Queensland, according to past internal Labor polling.

Dutton is on the national media radar after saying refugees were often not “numerate or literate in their own language, let alone English” and would “languish on unemployment queues” or take “Australian jobs”.

But despite coming under attack from quarters including the TV personality Karl Stefanovic, who branded Dutton’s comments “un-Australian”, name recognition remains low in Dickson.

“Is it Paul Smith?” a barista in Samford Valley says when asked about the local federal MP. “Peter Dutton?” Bingo.

“He comes in here sometimes. He seems like a normal guy.”

Dutton is often seen buying his flowers and coffee in Samford Valley but his electorate office is in Strathpine.

“I don’t think it’s a visible office at all,” says local barber shop owner Marty Flanegan. “Not that I want to see or hear of them anyway.

“The only time I pick up his name on the side of the building is when I slow down in traffic. What’s he minister of anyway? OK. Right. Nah, I didn’t even know that.”

Dickson was identified early in the campaign by a team of political scientists at Griffith university as a key marginal in a suite of Queensland seats which could decide the election outcome.

Despite Dutton being “at risk”, making comments which come under spirited attack from a TV host commanding some of the best ratings in the country doesn’t make him any more so.

Not when it’s on border protection, and not in Dickson, according to Sara Davies, an associate professor at the university who specialises in asylum policy, and lives in the electorate.

Davies assesses Dutton as a “weak performer” in public debates and “not the sort of person you would have door-knocking all over the place”.

But there’s little chance of Dutton’ remarks costing him votes in a conservative stronghold like Samford, in part because of that lack of voter engagement.

But the swinging centre which will decide Dickson are mostly over in the more “aspirational” suburbs around Strathpine, where border protection has previously rated highly among voter concerns and where such rhetoric could hit the mark, Davies says, and is a risk worth taking.

If Dickson eludes Dutton’s grasp – and he’s narrowly escaped defeat before – it won’t be perceived “dog whistles” that call his constituents’ tune.

“Well, he’s right,” says Ron, a retired soldier who opposes accepting more refugees.

“If they haven’t been to school before they come here, they haven’t got education in their own language. Where they are in the detention centres, they’re not going to get a decent education anyway because they don’t want to. As far as taking jobs away from Australians, the best person for the job is the right one.”

Offshore detention for asylum seekers who arrive by boat – the guts of both Coalition and Labor policies – is “as good as it can get at this stage”, he says.

Not that people he knows think about it much, Ron says, and then “only when it’s splashed over the news”.

But it’s something else that bugs Ron about Dutton.

Cindy, social worker from Warner, who works near Peter Dutton’s office. Photograph: Joshua Robertson/The Guardian

“I rate my politicians on how available they are to the community. This guy, I’ve seen him once, and as soon as he was finished speaking he was gone,’ Ron says.

“Don’t ask me who I’m going to vote for. It’ll be a flip of a coin and hopefully it will stand on its edge.”

Cindy, a social worker from Warner, knows who Dutton is – she works two doors down from his office. She knows his portfolio, too, and says this seems to divert his concerns from his own backyard.

“I don’t see him doing too much about the issues in this area. Employment matters, and we need more public housing and housing affordability,” she says.

And Cindy, who is Indigenous, had heard the refugee comments.

“I just think it’s ignorant and scaremongering,” she says. “I’ve recently been to uni with a lot of people who are refugees and they’ve got degrees from their own country that aren’t accepted here, so they’re going back to study. I don’t see many illiterate people.”

Cindy is comfortable with lifting refugee intake “but we have to do it in a way that makes it inclusive – we don’t want to just dump them in an area and say ‘fend for yourselves’,” she says.

Flanegan, the barber, says Dutton’s refugee comments will “definitely get a mixed response” in the area, but he has no idea what people think about offshore detention.

“You can’t stop people coming in but they’ve got to come in through the appropriate channels like everyone else,” he says, adding that he liked the willingness of new Australians to do jobs that others would not.

The things on his mind include the empty shopfronts up and down the main drag leading from his shop to Dutton’s office. And things that a small business like his “doesn’t need”, such as provisional tax, and that annoying thing known as multinational corporate tax avoidance.

He’d also like to hire a second apprentice and get a decent holiday sometime, noting that while the Liberal-National Coalition is the traditional party for small business, they have indulged in “the big switcheroo” in prime ministers just like Labor before it, inspiring confidence in neither.

Blake, a Samford Valley small business owner, with daughter Leora. Photograph: Joshua Robertson/The Guardian

“I’d just like to think they could ease the tax situation,” he says.

Blake, a small business owner in Samford Valley, does not immediately recall the name of his local MP. But he observes he’s surrounded by not a very disparate group of nationalities and a lot of rusted-on LNP supporters who will not be disturbed by Dutton’s comments about refugees.

He sees the link between such comments and the cultivation of racism.

“People that say refugees are taking their jobs, I know they don’t intentionally mean it, but they actually breed a bit of racism,” he says. “If people really think they are good at their job, they shouldn’t have to worry about their job being taken away. It’s a bit of a call for us to raise our game.”

Blake says his main view on politics is that both Labor and the LNP are guaranteed to disappoint in office.

“I just think that these days people need to start taking things into their own hands and that includes the education system and their own health,” he says.

It’s the language of retreat from politics and lots of people in Dickson speak it, even as they hold the fate of one of the Turnbull government’s most contentious figures in their hands.