Less than a week into Jeremy Corbyn's leadership, the defection rumours have already begun. The Tories are reported to have held talks with former Labour frontbenchers on crossing the floor (with two allegedly "pondering" the offer), while Lib Dem leader Tim Farron has spoken of receiving "various unsolicited texts" from "deeply distressed" MPs. It's unsurprising that some have been lamenting their party's direction to outsiders. MPs from opposite sides of the House frequently discuss their tribes' woes with gallows humour. But it does not follow that they are set to join forces.

Of the report that Labour MPs were considering defecting to the Tories, one former shadow cabinet minister told me: "It's garbage - no moderniser I can think of would defect. It is being put around by George Osborne and his crew to undermine moderates in our party. It is us who Osborne fears, after all, if we get our act together by the time he takes over the Tories."

There are Labour MPs whose stances on defence, foreign policy, and welfare are closer to those of Osborne than those of Corbyn. But party membership is based on more than a crude policy calculus (and even there differences remain). It is bound up with history, culture, identity and friendship. Many on the right speak of their party as they do of their family - an assocation which, even in troubled times, cannot be ended. As one moderniser told me: "The Labour right is even more tribal than the left". Many recite their own version of Charles Kennedy's declaration that "I will go out of this world feet first with my Lib Dem membership card in my pocket." It is for this reason that they are so hurt by the labelling of them as "Tories" by some Corbyn supporters (at this week's PLP meeting, Gisela Stuart appealed to the Labour leader to rebuke those who do). Crossing the floor would only validate this insult.

That Corbyn has already shown he is open to compromise further limits the potential for defections. The Labour leader's pledge to campaign for EU membership during the referendum avoids what could have been a trigger for departures. Though some believe that Corbyn could make it to the election, others believe that he will depart long before that point, offering the modernisers to chance to reclaim their party. His troubled first week has reinforced this assessment.

But even if Labour MPs were minded to defect (and I have seen no evidence that any are), there are pragmatic reasons for not doing so. Many of those who refused to serve under Corbyn hold safe seats which they would be more likely to lose as Conservative candidates. Nor do the Lib Dems, who may take a generation to recover, hold out the prospect of electoral success. After the precedent set by Tory defectors Douglas Carswell and Mark Reckless, they would also struggle to withstand pressure to call a by-election.

The defection rumours are likely to persist. Osborne has every interest in painting Labour MPs as Tories-in-waiting, a charge that some Corbyn supporters will echo. By dividing the opposition, he hopes to rule. But don't expect the modernisers to help him do so.