The community activist credited with opening Iain Duncan Smith's eyes to poverty and inspiring him to embrace compassionate conservatism has called on the minister to resign.

In a sharp attack on Duncan Smith, Bob Holman sets out how far the secretary of state for work and pensions has drifted from his original commitment to tackling poverty, and urges him to give up his ministerial role.

Duncan Smith's visit to the Easterhouse estate in 2002, when he was leader of the Conservative party, has become famous as the moment that sparked his determination to tackle societal breakdown. Holman took him around the estate's powerful community project, staffed by unemployed volunteers. The occasion became known as the Easterhouse epiphany and in 2003, Duncan Smith announced he wanted the Conservatives to become the "party for the poor".

"It is fully recorded that he was shocked by its social conditions. Less well known is that he responded to poor people in a positive manner," Holman writes, remembering the visit in an article for the Guardian. "He met families on low incomes who coped well with their children … I thought him a decent man."

In 2004 Duncan Smith set up the Centre for Social Justice thinktank to analyse the roots of poverty and come up with strategies for addressing it. "The CSJ was born through a visit to Easterhouse estate in Glasgow as the guest of Bob Holman. More than anyone else Bob Holman has opened my eyes to the problems and pockets of poverty across the UK," Duncan Smith wrote in an introduction to a CSJ report.

Since becoming minister, Holman believes Duncan Smith's analysis of the causes of poverty and the best way to tackle it have been constrained by the pressure to make cuts. "Within two years he was claiming that poverty was not directly due to a lack of money but was the result of bad parenting, drug and alcohol addiction, laziness, and the breakup of families," Holman writes.

"When I went to Westminster last year to challenge him, he acknowledged that he was under pressure and had to make trade-offs and compromises. My understanding is that to divert blame away from his policy failures he directed it at the poor themselves."

He suggests that Duncan Smith should resign and "become a campaigner for the end of poverty". A spokesman for the Department for Work and Pensions said there would be no comment.