If the Marvel Cinematic Universe is a rich, diverse tapestry of superhero films that all connect in one way or another, the Avengers movies are the tentpoles that glue the whole thing together. Getting to Marvel’s The Avengers was a herculean task unto itself, but following that up with a second Avengers movie—crafting a reason for the heroes to come together again while also ensuring it feels different and fresh—was a completely different kind of challenge. Avengers: Age of Ultron is a weird, dark, sometimes messy sequel, but it wears its ambition on its sleeve and takes the MCU to some surprising places. As it turns out, the process of creating Age of Ultron was something of a real-life battle, with writer/director Joss Whedon on one side and Marvel Studios on the other side. Here’s how it all came together.

When The Avengers was released in 2012 to critical acclaim and enormous box office success, it was clear Marvel Studios would obviously be greenlighting The Avengers 2. The question remained, however, whether Joss Whedon would return. Whedon was unique in that he both wrote and directed Marvel’s The Avengers, making him much more than a director for hire. His voice was intrinsically linked to the success of the film, and so it came as no surprise that Marvel approached him to return.

Whedon explained that while he initially didn’t think he’d come back, he quickly changed his mind:

“Avengers 2, it wasn’t a tough decision. For a long time I thought, ‘Well, it’s just not going to happen.’ Then when I actually started to consider it, it became so clear that I desperately wanted to say more about these characters, it would’ve been an easy no and it was a spectacularly easy yes. There was no wrestling. Either scenario was delightful. It was a win win but I’m very excited about it.”

In August 2012 it became official that Whedon would return to write and direct The Avengers 2, with a twist. He signed a deal extending his Marvel contract through the end of June 2015, but it included film and television. Indeed, at the same time that Whedon signed on to helm the Avengers sequel, he also signed up to develop and help create a TV series for Marvel Television at ABC. That TV show would eventually become Marvel’s Agents of S.H.I.E.L.D., which saw Whedon co-writing and directing the pilot.

So Whedon set about writing Avengers 2 while also creating the first major MCU-related Marvel TV series, but at the same time the powers that be were busy trying to lock down Robert Downey Jr.’s return. You see, with Iron Man 3 Downey had completed his four-film deal with Marvel, and thus his contract was up for renegotiation. Obviously Marvel wasn’t about to make an Avengers sequel without Iron Man, so negotiations began not for future Iron Man sequels but to lock Downey in for at least two more Avengers installments.

The negotiations were complicated, as Downey had previously renegotiated after the first Iron Man to get a percentage of the box office from every Marvel movie he appeared in, and for The Avengers alone Downey’s pay hit a whopping $50 million. While negotiations to bring Downey back began as soon as Iron Man 3 hit theaters in May 2013, they were tense—Downey even started hinting in the press that he may be done as Iron Man. But by June 2013 he and Marvel had come to an agreement for Downey to reprise his role as Tony Stark in The Avengers 2 and The Avengers 3—a deal that would have to be reworked later on when the idea for Captain America: Civil War came about. But that’s a story for a different time.

So after some hemming and hawing, by the summer of 2013 Whedon could rest easy knowing that Downey would, in fact, be back as Tony Stark. Early on in The Avengers 2’s development, Whedon said he wanted the sequel to feel smaller in scale than the first movie:

“[It should be] smaller. More personal. More painful. By being the next thing that should happen to these characters, and not just a rehash of what seemed to work the first time. By having a theme that is completely fresh and organic to itself.”

That, well, didn’t end up happening, but not by design. At 2013 San Diego Comic-Con, in July, Whedon took the Hall H stage at the end of the Marvel Studios panel to present a short video that revealed the title of the Avengers sequel: Avengers: Age of Ultron. This in turn revealed that the villain of the follow-up would be the robot Ultron and not, as many believed, Thanos. This was always the plan:

“Thanos is not out of the mix, but Thanos was never meant to be the next villain. He’s always been the overlord of villainy and darkness.”

But despite the film’s title, Whedon didn’t intend to directly adapt the Age of Ultron comics arc either. That one saw Hank Pym aka Ant-Man creating the villainous robot Ultron, but the Pym character was being saved for the Ant-Man movie, so Whedon reworked Ultron’s origin story to essentially tell a father-son story with Tony Stark. In a pitch perfect piece of casting, Whedon turned to James Spader to fill the role via performance capture.

Whedon also aimed to make Age of Ultron visually dynamic by bringing in two new characters: Quicksilver and Scarlet Witch:

“Their powers are very visually interesting. One of the problems I had on the first one was everybody basically had punchy powers … [Quicksilver]’s got super speed. [Scarlet Witch] can weave spells and a little telekinesis, get inside your head. There’s good stuff that they can do that will help sort of keep it fresh,”

These two characters were actually shared between Marvel Studios and 20th Century Fox, so there were certain things Whedon was and was not allowed to do. For instance, Marvel Studios’ version of the characters could not refer to them in any way as “mutants,” nor could Quicksilver be revealed as the son of Magneto, since the main X-Men characters remained under Fox control.

But as development on Age of Ultron continued, so did development on Agents of S.H.I.E.L.D. which was being prepped for a September 2013 premiere on ABC, and Whedon admitted that Marvel Studios was none too happy about him splitting his duties:

“They didn’t actually want me to make it,” said Whedon. “It’s like, ‘Uh, Joss, we really wanted you to do [Age of Ultron]. Instead you created a TV show, you moron.’ ‘I thought you wanted me to!’ ‘No, we just wanted you to make a movie.’ ‘Oh. My bad.’ … It went from being absolutely 100% the driving force and totally hands-on to ‘That sounds great, [co-showrunner] Jed [Whedon]! You should do that!'”

Whedon also ran into problems when, in the midst of creating a TV show about S.H.I.E.L.D., Kevin Feige told him they were planning on destroying S.H.I.E.L.D. in the second Captain America movie:

“They had said early on, ‘Hey, we’re thinking about doing this show about the agents of S.H.I.E.L.D.,'” said Feige. “And Joss said, ‘I think I might do this.’ I said, ‘That’s cool. God bless you. But you should know that we’re destroying S.H.I.E.L.D. in Winter Soldier. You guys do whatever you want. But know that that’s what we’re going to do.'”

So Whedon helped get Agents of S.H.I.E.L.D. off the ground and co-wrote and directed the pilot, but after that he moved into Age of Ultron full-time. Filming finally began in February 2014 ahead of a May 2015 release date, but Whedon quickly found the shoot even more complicated than the first film due to the expanded cast:

“There’s stuff between Hawkeye and the Scarlet Witch that’s some of my favorite stuff in the movie, and it’s there because they’re the only actors I had. When we started shooting in Italy, I had Quicksilver, Scarlet Witch, and Jeremy Renner; everybody else was busy. So, I’m like ‘Okay, I guess these guys are gonna have a scene, and I can work with that.’”

Whedon also changed up his shooting style for the film, using multiple cameras instead of carefully planned shots:

“I’m always fighting to make the shots less perfect. To say, let’s have the bad camera; let’s find the Hulk the way we find other people. Particularly in this movie, where I shot it very, very differently than the first. Everything in the first was very deliberate, and in this one, I said I want to go multiple cameras, a lot of sort of documentary-style footage, and worry less about the 3-D — [make] it a little more grown-up and little less presentational. Sometimes I thought, ‘This is really working,’ and sometimes I thought ‘Remember when I used to make decisions?’”

And while the movie ends with a different lineup for The Avengers, Whedon originally tried to lobby Marvel to let him introduce Captain Marvel and Spider-Man, to no avail. Kevin Feige had other plans:

“[Captain Marvel] was in a draft. But to me, it would have done that character a disservice, to meet her fully formed, in a costume and part of the Avengers already when 99% of the audience would go, ‘Who is that?’ It’s just not the way we’ve done it before… The way we reveal Scarlet Witch at the end of the movie? Those were Captain Marvel plate shots. Joss said, ‘We’ll cast her later!’ And I said, ‘Yeah Joss, we’ll cast her later.’ [Whispers to an invisible associate who isn’t Joss] ‘We’re not putting her in there!’”

Whedon also shot scenes with Tom Hiddleston returning as Loki in Thor’s nightmare sequence that were cut because test audiences kept thinking it meant Loki had a larger role to play in the film. Hiddleston explained:

“In test screenings, audiences had overemphasised Loki’s role, so they thought that because I was in it, I was controlling Ultron, and it was actually imbalancing people’s expectations.”

Production wrapped in August 2014, but things then got even more complicated. Whedon was adamant about trying to make Age of Ultron even shorter than The Avengers, but was also at odds over what was important to the movie. For Whedon, the character-centric scenes at the farm were key. For Marvel, the MCU-building scenes of Thor in the cave were integral. The two weren’t necessarily on the same page, and Whedon went so far as to say there was something of a hostage negotiation going on with what did or didn’t make the cut: