The archbishop of Canterbury, Justin Welby, has said he was annoyed and embarrassed to learn the Church of England holds an indirect stake in Wonga, the payday lender he promised to bring down through fairer competition, and has called for the church to rethink its investment guidelines.

Welby told BBC Radio 4's Today programme he was disappointed to discover the church's £5.5bn investment portfolio included a stake in Accel Partners, a US venture capitalist that co-funded the launch of Wonga, who Welby singled out when he called for an expansion to credit unions as a means to "compete" payday lenders out of existence.

Asked to rate his embarrassment on a scale of one to 10, he replied, "About eight".

"It shouldn't happen, it's very embarrassing, but these things do happen," he said. "We have to find out why and make sure it doesn't happen again."

Welby said the holding in Accel was worth about £75,000. He added: "I was irritated for a few minutes, but these things happen. I understand the business – it's an incredibly complex business."

Welby said he was unsure whether the investment in Accel broke church guidelines, one of which dictates that money should not be put into companies where 25% or more of its business is connected to such lending. However, he stressed that the investment should not have happened at all: "They shouldn't be investing in Wonga. We don't think it's a good thing."

Such ratios were set by the church's ethical investment advisory group, Welby said, which was independent from him. Nonetheless, he said he would call for a change to guidelines, which also allow the church to invest in companies with very small interests in things such as gambling and pornography: "We have to review these levels and make sure we're entirely consistent between what we're saying and what we're doing."

Asked if such guidelines suggest the church thought "a little sin" was tolerable Welby, understandably, disagreed – "Just for the record, I'm not in favour of sin" – but also stressed the complexity of such investments, for example asking whether the fact hotel chains make pornography available to guests meant the church should never invest in the entire sector.

"If you exclude any contact with anything that directly or indirectly at any point gets you anywhere bad, you can't do anything at all," he said. Welby added: "We can't say that we tolerate bad things, but you've got to live in the real world, and living in the real world means life is often very complicated and you can't escape the complexity."

However, he was clear that connection to payday lending was definitely wrong: "This is an embarrassment. We think that the payday lenders charge vastly excessive amounts for the loans they make. There is a totally inadequate range of choice for consumers in deprived areas. I've seen it. I've lived in these areas and worked in them. I've had staff who've been caught up in it and have had to be helped, and have had their lives destroyed by it. This is something that really matters to me."

Welby insisted he had never specifically targeted Wonga, calling them one of the more professionally managed companies in the sector, but had been answering a direct question about them when, earlier this week, he recounted a conversation with Wonga's chief executive, Errol Damelin.

"I said to him quite bluntly: 'We're not in the business of trying to legislate you out of existence, we're trying to compete you out of existence'," he told Total Politics magazine.

Welby told Today he did not support regulation to curb payday lenders' activities, for example credit caps: "If we try and cap interest rates and drive the legal payday lenders out of business through regulation people become desperate – and there's no consumer choice in a lot of our deprived areas – will end up with the loansharks, who are just a totally different kettle of fish, very much worse."

Instead, Welby said, he wanted the church to support a massive expansion of credit unions in poorer areas. Even these, he conceded, would be obliged to lend short-term at an annualised interest rate of about 70% or 80%, something he argued was hugely less than that charged by payday lenders: "It's a huge sum of money, but it's better than 5,500%."

He conceded there were potential problems with the church becoming directly involved in such a trade, but argued: "It's better to have a go than to just stand and wring your hands and say, it's all terrible."