Taken at some point in the 1970s and ostensibly depicting a harmonious upper-class British family, it is a picture that will likely lodge itself in modern UK political history.

While the family nanny, Mary Kidd, stands unobtrusively behind, from left to right the photo features Boris Johnson at the age of 10, his brother Leo (seven), their sister Rachel (nine) their parents, Charlotte and Stanley and their youngest brother Jo.

While a certain wilfulness was apparent even then in the pose of the older Johnson boy – who even at an early age was pronouncing his ambition to be “world king” – few might have guessed that Jo, the smiling blonde boy prancing on the right, would emerge in time as the clan’s standard bearer for remaining in Europe.

More than that, however, the citing by Jo Johnson on Thursday of the “unresolvable tension” between family loyalty and national interest as being the reason for his decision to quit as an MP and minister reflects the extent to which Brexit has divided not just the UK, but the family itself.

The figure of Stanley looms large in any narrative of the Johnson tribe, albeit as one in which biographers record him as being largely absent for much of his children’s early upbringing.

While in the US to study creative writing, for example, he was not at his wife’s side when she gave birth to Alexander Boris de Pfeffel Johnson in New York, in June 1964, the son with whom he shares a blend of barely concealed ambition and wit.

Facebook Twitter Pinterest Boris Johnson and his younger brother Jo, who announced on Thursday that he was quitting politics. Photograph: PA

By the middle of the 1970s, after landing a job at the European commission, he was to take a more significant parenting role as bouts of depression led to the repeated hospitalisation of his first wife, Charlotte.

These days, Stanley, 79, a former MEP, remains a visible and vocal supporter of his eldest son, even if his support for remain during the 2016 referendum has led some to question the depth of his commitment to leaving the EU.

By contrast, his daughter Rachel has continued to fiercely oppose Brexit. A journalist whose CV includes stints at the Financial Times and as editor of The Lady, the one-time Liberal Democrat ran as a lead candidate for Change UK during this year’s European parliamentary elections, although she later described herself as a “rat that jumped on to a sinking ship”.

Rachel and Leo Johnson on their close relationship Read more

As the spotlight fell on the family after Jo Johnson’s resignation on Thursday, she was quick to dismiss suggestions of familial strife, tweeting: “I said last night at a charity do that the family avoids the topic of Brexit, especially at meals, as we don’t want to gang up on the PM!”

Of the other siblings, Leo Johnson has distanced himself from politics, having built a career in the City and as a co-host of a Radio 4 series that explores how innovation could transform the way society functions.

Nevertheless, his retweeting of remain accounts has been noted, not least a recent one criticising the role of the prime minister’s chief of staff, Dominic Cummings, and another lambasting “holier than thou so-called upholders of democracy” for repeating a mantra about the need to deliver Brexit.

For the second time in less than than a year, meanwhile, the departure from government of Jo Johnson has narrowed scrutiny of the family’s Brexit divisions to the relationship between him and his louder, older brother.

Facebook Twitter Pinterest Stanley Johnson, left, and Boris Johnson’s siblings Rachel and Jo Johnson. Photograph: Stefan Rousseau/AP

A former investment banker and journalist who is married to Amelia Gentleman, a reporter for the Guardian, he was appointed director of the No 10 policy unit by David Cameron in 2008 and became an MP in 2010 for what was then the safe Conservative seat of Orpington, in the London borough of Bromley.

Described by one commentator and Dallas aficionado as Bobby Ewing to Boris’s “charismatic, naughty JR”, Jo Johnson has long concealed any rivalry, even if others have suggested that the quieter brother would have taken some satisfaction from being the first Johnson in Downing Street.

Play Video 0:21 Jo Johnson quits as Tory MP and minister: 'It's time to move on' – video

By the time Jo Johnson quit Theresa May’s government in November last year, calling for the public to have a fresh say on Brexit, some onlookers had even begun to envisage the struggle between the two brothers turning into one for No 10.

While Boris enjoyed a near cult following among some parts of the party’s grassroots, to other, more moderate Tories, the other Johnson presented a potential way to reconnect with younger voters, remainers or those repelled by his elder brother’s inconsistent relationship with the truth and accusations of racism.

On Jo Johnson’s second dramatic exit from government, and with the axis of power in the Tory party now having shifted definitively to the hard Brexit right, such a scenario appears as distant as the Johnson clan’s lazy childhood summers.

• This article was amended on 19 September 2019 because an earlier version referred to Jo Johnson quitting May’s cabinet. Rather he quit her government.