How on earth did the government not see the Windrush row coming? That’s what everyone – including members of the government – have been asking since the rumbling story erupted a week ago, after six months of cases appearing in the press. Trying to work it out herself, Amelia Gentleman, the Guardian reporter who uncovered the scandal over six months, wrote rather modestly: “Obviously the prime minister is busy and maybe doesn’t read the Guardian much.” But Theresa May employs people to keep her abreast of brewing rows, and no one thought to mention it.

Stranger still, when you consider that this scandal has been years, not just months, in the making, as the “hostile environment” policy on immigration was formulated under the coalition. I understand that, at the time, it was the subject of a standup row in the cabinet room between May when she was home secretary and the then education secretary, Nicky Morgan. Morgan had, along with the health secretary, Jeremy Hunt, raised concerns that May’s new policy for illegal immigrants would turn teachers and health workers into immigration officials. Morgan went further and also questioned the very principle behind the “hostile environment”. No wonder she lost her job as soon as May took over as prime minister.

May had also been asking ministers from other departments to insert the phrase “hostile environment” into speeches and documents, which didn’t go down well. But while some Conservatives objected to the catchy name, few noticed that the design of the policy itself was flawed. It required people to produce documents that the Home Office itself knew no longer existed, or which it was highly unlikely that anyone, whether living here illegally or legally, would have kept for several decades.

This wasn’t just May’s mess. David Cameron was the one who had made and then stuck to the Tory promise to drive net migration down into the tens of thousands. He then took over the cabinet committee charged with creating this hostile environment after his 2015 election win.

Ministers from the time say they noticed Cameron growing rather awkward when dragged into standoffs between the Home Office and other departments over the target. “May would say to him, ‘Well, you’ve given me this target to meet so why are all these ministers getting in the way’,” says one cabinet minister. “That’s what she’s like, what she’s always been like, if you give her something to do, she’ll keep at it even when it seems impossible.”

This dogged determination once won May plaudits as a “bloody difficult woman” who focused on getting things done, rather than playing political games. But it can also lead to the sort of stubbornness that prevents pragmatism: many Tories would dearly love for their party to ditch the net migration target, as it serves largely as a quarterly reminder that the government is still failing to meet it. Unfortunately, one of the very few who would die in a ditch for the target is May.

Amber Rudd, her successor, is known to be more liberal on immigration in private. But she has stuck to May’s imprint for the Home Office, something her allies now feel is a mistake. “Successful home secretaries, like them or loathe them, set their own agenda,” says one. Some have suggested Rudd should have emulated Jeremy Hunt in pushing ahead with a personal programme of reform, though it’s difficult to see how this would have worked in practice, given May is still very interested in the Home Office and generally happy for Hunt to do as he pleases as health and social care secretary.

The Windrush issue suddenly blew up because everyone saw it as an opportunity to further their own causes

“Amber has definitely been damaged by this row,” said one supporter. “She won’t lose her job but in terms of leadership, it’s all looking a lot more difficult.” In fact, some on Rudd’s side suspect her difficult week, in which she was really unable to blame anyone for what had happened, has come about entirely because she is a leadership prospect. “This is a proxy war,” says one senior MP. “You look at the way the European Research Group [the powerful group of Brexiteer Tories, chaired by Jacob Rees-Mogg] has been trying to go after remainers. They went after Philip Hammond, trying to get him sacked. Then they had a good go at Greg Clark. Now it’s Amber’s turn.”

These leadership tensions are one of the reasons Conservatives on both sides of the party think the Windrush issue suddenly blew up in the way it did: everyone saw it as an opportunity to further their own causes. Privately, many senior Tories aren’t too worried about the implications of the row itself, beyond the usual gripes about No 10 being “flat-footed” in not realising there was a problem, and refusing to discuss the matter with those who tried to raise it. They think this is eminently solvable, and it will cause no long-term damage to the Conservatives – so long as the Home Office ensures the flaws in the hostile environment policy are ironed out quickly.

This, though, suggests few Tories have yet grasped how big the flaws are with the hostile environment approach, despite the many years in which it has now been active. It has caused problems that go far beyond the Windrush generation. If the Tories really think they can move on after merely addressing Windrush, then they’re at risk of failing to see the next big row coming, too.

• Isabel Hardman is a political journalist and assistant editor of the Spectator