Prime Minister Tony Abbott says he shouldn't have winked while listening to a radio talkback call from a woman who told him she had to work on a phone sex line to make ends meet.

Mr Abbott said the wink was directed at the host of the ABC radio program and was not meant to be disparaging of the caller, "Gloria", who described herself as a 67-year-old chronically ill pensioner in tough financial circumstances.

The incident sparked an immediate backlash and Mr Abbott was pilloried on social media and by fellow politicians because it appeared he had winked and smiled as Gloria told him what she did to help pay the bills.

Mr Abbott said he was looking at the host - the ABC's Jon Faine - and was winking his assent to take the call.

"I shouldn't have done it ... I should have been more focused on the caller and less focused on the interviewer," Mr Abbott told the Nine Network said on Thursday.