A veteran of Boris Johnson’s political campaigns has resigned as a Downing Street political adviser as tensions rise between former Vote Leave members and ex-City Hall factions around No 10.

Alex Crowley, who worked on Johnson’s Tory leadership campaign, two mayoral campaigns and wrote a book about him, quit last week. His exit came as several sources with knowledge of No 10 suggested that there was an atmosphere of feuding inside Johnson’s administration.

On one side, Johnson’s chief strategist, Dominic Cummings, leads the group of former Vote Leave staffers directing the hardline policy on Brexit, signalling the prime minister could be prepared to ignore the law to take the UK out of the EU with no deal.

Quick Guide Downing Street's three tribes Show Downing Street's three tribes Vote Leave

Dominic Cummings joined Boris Johnson’s team after he became prime minister, demanding control over his operation as the price for his advice. He almost immediately took on the job of chief of staff, which was initially briefed as the role belonging to Sir Eddie Lister, a former City Hall ally of Johnson. Other veterans of the Vote Leave operation include Oliver Lewis, a Brexit policy adviser, Lee Cain, the director of communications, and Rob Oxley, his press secretary. City Hall Johnson’s leadership campaign team was initially very much a product of his time as London mayor. But Lister appears to have waned in influence after leading the Downing Street “transition team”, along with Will Walden, his former director of communications, who left over the summer. One exception appears to be Munira Mirza, a former deputy mayor, who is in charge of his policy unit. Johnson seemed to big up her role in the party’s election manifesto, saying on Sunday that the document was being “drawn up by two particularly brilliant women”. Team Symonds Those who have worked with Johnson in the last year say one of the few people he listens to seriously is his partner, Carrie Symonds. Her inner circle includes a number of special advisers who have entered Downing Street and she is close to Sajid Javid, the chancellor, after having previously worked as his special adviser, and Zac Goldsmith, the environment minister.

On the other side, there are the former City Hall advisers, including Sir Eddie Lister, Johnson’s former chief of staff, and Sir Lynton Crosby, his former election strategist, who have been sidelined under the new regime. Will Walden, his former communications director, who helped Johnson on his leadership campaign, has left No 10 to go back to his old job since Cummings consolidated his power over Downing Street.

One Tory insider told the Sun that Walden and Crosby had been cut out of Johnson’s inner circle and had reservations about the prorogation strategy that set up a battle between the prime minister and the courts. “The Cummings experiment has palpably failed, but Boris will not turn the ship,” the insider said. “He’s only listening to two voices now, Dominic and Carrie [Symonds, his partner] – and Dominic’s approach is proving a car crash. We’re getting really worried.”

A Tory special adviser told the Guardian there was a feeling that Cummings was “out of control” over his pursuit of the prorogation strategy and insistence that Brexit could be done by 31 October regardless of the law blocking a no-deal departure.

A cabinet minister also said he was personally unhappy about some of the divisive language around Cummings’s strategy but that there was little pushback against his methods because he was such an effective force.

Cummings has been far more prominent and publicly visible than is usual for a senior adviser to the prime minister. He appeared relaxed and dressed in a hoodie at the Conservative party conference on Monday, talking calmly to reporters who chased him through the hall, while activists shouted: “You have the members on your side, Mr Cummings. Get Brexit done.” Asked how he intended to circumvent the Benn Act if there was no deal, he said it was not a question for now.

However, in private briefings with journalists, he has argued there is still space for a deal and explained that proposals will be taken to Brussels soon. It is understood that Cummings strongly dismisses the idea there are tensions within No 10 over his 31 October Brexit strategy and points out that Crosby has no role in Johnson’s operation and that neither does Walden report to him.

Combative appearances by Cummings on the parliamentary estate have caused worries among some of Johnson’s other advisers, including a public altercation with the Labour MP Karl Turner and another incident in which he confronted Jeremy Corbyn, goading the Labour leader to vote for an election. He has also made remarks that it was “not surprising” that the public was angry with MPs.

Apart from Cummings, the prime minister is known to also listen closely to Symonds, his former Tory communications director partner. This has caused friction with Crosby, the electoral strategist who is a close friend of Johnson and ran his mayoral campaigns.

Downing Street denied claims that No 10 was overrun with feuding and tensions and downplayed the significance of Crowley’s resignation from the policy team, saying he was a junior aide.

Crowley was research director and then political director of Johnson’s London mayoral election campaigns in 2008 and 2012. He wrote Victory in London about Johnson’s political rise, which at one point describes how at “particularly stressful moments” the future prime minister would disappear and hide behind a giant coffin in a prop store.

Crowley did not respond to an emailed request for comment.

The jostling for position in Downing Street has been ongoing since Johnson entered No 10 in July. Cummings moved to consolidate his position in charge, sidelining Lister in the process. Since then, he has run what special advisers describe as a “reign of terror”, enforcing message discipline and making them all report to him personally rather than their ministers.

Lister was initially briefed to the media as Johnson’s newly appointed chief of staff and is described by some inside Downing Street as one of “the sensibles”. But an internal No 10 email showed last month that Cummings is now styled as “assistant to the prime minister” in charge of Brexit and domestic policy, while Lister was been downgraded to “chief strategic adviser” responsible for foreign affairs, business and security.