Here’s a question for Game Of Thrones fans: Why aren’t the Lannisters black?

The answer probably seems painfully obvious, at least if you’re a defender of the show: The world of Westeros is modeled after medieval England, so it would be inaccurate to have an ethnically diverse set of ruling families. This is also generally how the show’s cast and crew have responded to questions about the show’s diversity. For example, when asked why his series has a handful of black characters but no Asian ones, author George R.R. Martin wrote on his blog: “Well, Westeros is the fantasy analogue of the British Isles in its world, so it is a long, long way from the Asia analogue. There weren’t a lot of Asians in Yorkish England either.”

Meanwhile, actress Nathalie Emmanuel—who plays one of the few characters of color on the series—told The Telegraph: “We’re dealing with a fantasy world in an almost Medieval setting, set around a fantasy version of the British isles, where history tells us that a queen would have been white, generally.” Martin later wrote that “Westeros around 300 AC is nowhere near as diverse as 21st century America, of course,” seeming to forget that he created Westeros wholesale.

“We’re dealing with a fantasy world in an almost Medieval setting, where history tells us that a queen would have been white, generally.”

In other words, the show’s cast and creators believe diversity is out of their hands. Despite its fantastical elements, they argue Game Of Thrones deserves to be judged as a historical drama like The Tudors or Mad Men, both overwhelmingly white shows whose racial dynamics do presumably reflect a real-life historical reality. But is Game Of Thrones actually replicating history in a meaningful way? And, regardless, should the show be let off the hook just because it’s main characters live in castles?

It’s a complicated question. Studies have shown that diversity can positively affect both self-esteem and tolerance in their viewers, and Game Of Thrones—one of the most popular shows on TV—centers on a massive ensemble of predominantly white characters. While the show has improved its representation in latter seasons, its diversity is still mostly restricted to supporting characters in the kingdoms of Essos and Dorne, who seldom get the attention they deserve (I’m looking at you Sand Snakes!) Is there really nothing the creators can do about that?

Despite telling Time that he’s aiming to expose the darker side of medieval life, Martin actually plays fast and loose with history all the time. For one thing, Westeros is filled with magic. Dragons, White Walkers, and magical smoke babies have played an integral part in Westerosi politics. As far as I know, the Plantagenets never flew around England on the backs of giant flying lizards.

But even putting aside the fantasy elements for a moment, Game Of Thrones has made some pretty significant changes to history. For instance, Martin grafted medieval English culture onto a landmass 130 times bigger than England, as if geography plays no role in the evolution of a civilization. Time is also pretty wonky in the series: Westeros has been a functioning medieval society for roughly 5,300 years, more than 4,000 years longer than the real-life Middle Ages lasted. One has to wonder when, if ever, Westeros will get around to its own Scientific Revolution.

George R.R. Martin plays fast and loose with history all the time.

There’s also the fact that seasons last an indeterminate amount of time in Westeros (the series opens during a seven-year stretch of summer and one winter reportedly lasted for “a generation”). How do Westeroi farmers possibly store enough food for those phenomenally long winters? And why do they even have a farming system that’s anything like the one developed for a traditionally seasonal medieval Europe? Considering farming was the basis of the feudal system that defined the entirety of medieval society, Martin’s seasonal change is no small adjustment. Nor is the fact that the High Septon of the Faith Of The Seven resides in King’s Landing. For the Game of Thrones neophyte, that’s the equivalent of the Pope operating out of London.

My point here is not to nitpick plot choices. Game Of Thrones is free to reimagine history—and reality—as much as it likes. But for Martin and HBO showrunners David Benioff and D.B. Weiss to make major adjustments to the timespan, size, climate, and culture of medieval Europe while claiming that real-life racial dynamics are immutable is both disingenuous and troubling.

Ask yourself this: If you balked at my earlier suggestion that the Lannisters be played by black actors, were you similarly troubled by the fact that Martin casually changed the physics of the universe to facilitate the unusual climate in his medieval world?

How do Westeroi farmers possibly store enough food for those phenomenally long winters?

I believe the reason we’re comfortable with some historical changes but not others has more to do with our 21st century world than any medieval one. Farming and the Catholic Church were two of the biggest elements of medieval life, but they’re no longer central to mainstream Western society. That means we don’t read it as inaccurate when they’re manipulated in our historical fantasy. Race, meanwhile, remains a central facet in contemporary life—which is why the idea of a racially diverse high fantasy series can seem so jarring.

But Game Of Thrones isn’t a history lesson, it’s a contemporary show aimed at a contemporary audience. That’s why its female characters fit contemporary beauty standards (slim and tan with flowing hair) rather than medieval ones (pale and plump with thin eyebrows, high hairlines, and bulky headdresses). Despite its pseudo-historical trappings, Game Of Thrones should be no more immune to calls for diversity than any other TV show.

Game Of Thrones isn’t a history lesson, it’s a contemporary show aimed at a contemporary audience.

Of course given its phenomenal success, Game Of Thrones is unlikely to change its racial dynamics, especially considering the show is likely coming to an end in the next few years. But the series’ limitations can hopefully inspire future high fantasy creators to think differently about race.

One option would be to create a “colorblind” world in which racial prejudice isn’t a factor. Much as Martin decided Westeros had a different seasonal pattern than medieval Europe, a high fantasy creator could simply decide that her fictional country is inherently diverse. That’s the preferred way to stage Shakespeare plays nowadays, and has worked for shows like Galavant or the 1997 version Cinderella, starring Brandi and Whitney Houston.

One option would be to create a “colorblind” world in which racial prejudice isn’t a factor.

Alternately, creators who do want to keep a connect to history in their work could decide to explore the reality of being a racial minority in a mostly white medieval world, in much the same way Game Of Thrones has explored the reality of being a woman in a patriarchal society. Fans have already done some of their research for them. The Tumblr Medievalpoc has collected hundreds of examples of people of color depicted in real medieval European art. If a high fantasy series were to pick 10 random paintings from Medievalpoc’s collection (starting with, say, this one from England circa 1410, this 1240 manuscript, this 15th century German altarpiece, and these Gondoliers from 15th and 16th century Italy), it could feature an incredibly diverse ensemble without losing its relationship to history.

Admittedly, this method would require a good deal of historical cherry picking, but historical cherry picking is what makes Games Of Thrones so compelling in the first place. The real-life War of the Roses (an oft-cited inspiration for the series) stretched over 30 years, but Martin selected the best bits and crammed them into just a few years to make his show more exciting. Game Of Thrones is regularly praised for its “gritty realism” despite the fact that Westeros is probably far more violent than real-life medieval Europe (as Medievalpoc points out in this excellent essay), So why couldn’t another high fantasy series offer that same pseudo-realistic feel while being a little more diverse?

High fantasy fans are so used to almost exclusively white stories, it can be really hard to shift our mindset about race. But until we do so, we’re just depriving ourselves of compelling narratives. When Vogue asked which show she’d like to appear on the most in 2015, Lupita Nyong’o answered Game Of Thrones. Who wouldn’t want to watch an actor of her caliber sink her teeth into a juicy high fantasy role? And more importantly, why should a misguided commitment to historical accuracy prevent her from doing so?