Directing arguably the most anticipated sequel of the year is no easy feat. But Joss Whedon said dealing with fans’ expectations for Avengers: Age of Ultron was not the main challenge, but the expanded superhero ensemble.

"Shoot me in the face! It is a nightmare. I long for the simple movies like Serenity!" he told U.K. magazine SFX. “This is the hardest juggling act I have ever, ever tried to pull off.”

With the Avengers ranks having swelled to include Elizabeth Olsen’s Scarlet Witch and Aaron Taylor-Johnson’s lightning-fast Quicksilver, together with an expanded non-voice role for Paul Bettany with android The Vision, Whedon has had to ensure everyone fits in the same universe without looking out of place.

Related: Joss Whedon on ‘Avengers 2’: “‘Strong But Damaged By Power’ Describes Every Person in This Movie”

"They’re very disparate characters. The joy of the Avengers is they really don’t belong in the same room. It’s not like the X-Men who are all tortured by the same thing and have similar costumes," he explained. "These guys are just all over the place. And so it’s tough. Honestly, this is as though as anything I’ve ever done, and I haven’t worked this hard since I had three shows on the air."

One of the standout characters from the first outing was the Hulk, and Whedon admits that there’s going to be more of Mark Ruffalo’s massive green rage-machine in the follow up.

"There is more Hulk in this movie than there was in the first. But what I’m excited about is we shot this movie very differently. I was running a lot of cameras, I was shooting long lenses, which I don’t usually do," he said, adding that he aimed to shoot the film a little like a documentary.

Related: Robert Downey Jr. Offers Fans Chance to Live Like Tony Stark, Attend ‘Avengers: Age of Ultron’ Premiere

"What I love is we have the opportunity, because we went in with this mission statement, to shoot the Hulk like a character in a movie, and not like a ‘Look what we’ve got!’ We have ‘overs’ – blurry ‘Hulk’s over there! Bits, very quick shots. Everything isn’t ‘We built the Hulk so for god’s sake you’re going to watch him in this long take, full frame, the whole time!’ We really got to make him one of the characters in the movie. And that was a gift."

Avengers: Age of Ultron releases in the U.S. May 1, a week after it opens in the U.K. and other parts of Europe. Watch a trailer for the film below: