Last Sunday’s Game of Thrones saw the return of the Brotherhood Without Banners, Jaime and Brienne’s first meet-up in two seasons and Arya Stark’s return to plucky form (courtesy of killing The Waif), as well as a girl’s declaration that she was heading home to Winterfell. The night is dark and full of anticipation for the next episode, right?

Not according to many of the show’s fans on social media and in TVLine’s own comments section, the latter of which contained sentiments like “God awful episode. All setup, very little payoff” and “More than one episode this season has been disappointing. I am worried that without books the show will continue to go downhill.”

When we spoke with Thrones star Nikolaj Coster-Waldau earlier this week, we asked for his take on the matter.

“It’s funny. I think a lot of things happen in every episode. But, I mean, I get it,” he said. “People are spoiled — in a good way! — and you expect some crazy stuff. But it’s been like that every year.”

Still, just like Bran learning at the Three-Eyed Raven’s feet, Coster-Waldau cautions, Thrones viewers have to have a little patience.

“You also know that there’s a reason why” the show parcels out the drama over the course of a season, he says. “You can’t hit the maximum volume each episode. Then you get numb to it. It has to build up. You have to tell a story, lay the seeds. Then hopefully — and I believe — the payoff is so much better and so much greater, instead of just getting everything at once.”

But if you’re jonesing for a fight, this week’s episode, which is titled “The Battle of Bastards” and is expected to show the Jon-Ramsay face-off for which we’ve been waiting, “is the biggest they’ve made, in terms of scale,” he adds, chuckling as he declines saying more. “A lot of stunt guys were working on that episode.”