Despite telling Time that he’s aiming to expose the darker side of medieval life, Martin actually plays fast and loose with history all the time. For one thing, Westeros is filled with magic. Dragons, White Walkers, and magical smoke babies have played an integral part in Westerosi politics. As far as I know, the Plantagenets never flew around England on the backs of giant flying lizards.

But even putting aside the fantasy elements for a moment, Game Of Thrones has made some pretty significant changes to history. For instance, Martin grafted medieval English culture onto a landmass 130 times bigger than England, as if geography plays no role in the evolution of a civilization. Time is also pretty wonky in the series: Westeros has been a functioning medieval society for roughly 5,300 years, more than 4,000 years longer than the real-life Middle Ages lasted. One has to wonder when, if ever, Westeros will get around to its own Scientific Revolution.

There’s also the fact that seasons last an indeterminate amount of time in Westeros (the series opens during a seven-year stretch of summer and one winter reportedly lasted for “a generation”). How do Westeroi farmers possibly store enough food for those phenomenally long winters? And why do they even have a farming system that’s anything like the one developed for a traditionally seasonal medieval Europe? Considering farming was the basis of the feudal system that defined the entirety of medieval society, Martin’s seasonal change is no small adjustment. Nor is the fact that the High Septon of the Faith Of The Seven resides in King’s Landing. For the Game of Thrones neophyte, that’s the equivalent of the Pope operating out of London.

My point here is not to nitpick plot choices. Game Of Thrones is free to reimagine history—and reality—as much as it likes. But for Martin and HBO showrunners David Benioff and D.B. Weiss to make major adjustments to the timespan, size, climate, and culture of medieval Europe while claiming that real-life racial dynamics are immutable is both disingenuous and troubling.

Ask yourself this: If you balked at my earlier suggestion that the Lannisters be played by black actors, were you similarly troubled by the fact that Martin casually changed the physics of the universe to facilitate the unusual climate in his medieval world?

I believe the reason we’re comfortable with some historical changes but not others has more to do with our 21st century world than any medieval one. Farming and the Catholic Church were two of the biggest elements of medieval life, but they’re no longer central to mainstream Western society. That means we don’t read it as inaccurate when they’re manipulated in our historical fantasy. Race, meanwhile, remains a central facet in contemporary life—which is why the idea of a racially diverse high fantasy series can seem so jarring.

But Game Of Thrones isn’t a history lesson, it’s a contemporary show aimed at a contemporary audience. That’s why its female characters fit contemporary beauty standards (slim and tan with flowing hair) rather than medieval ones (pale and plump with thin eyebrows, high hairlines, and bulky headdresses). Despite its pseudo-historical trappings, Game Of Thrones should be no more immune to calls for diversity than any other TV show.

Of course given its phenomenal success, Game Of Thrones is unlikely to change its racial dynamics, especially considering the show is likely coming to an end in the next few years. But the series’ limitations can hopefully inspire future high fantasy creators to think differently about race.

One option would be to create a “colorblind” world in which racial prejudice isn’t a factor. Much as Martin decided Westeros had a different seasonal pattern than medieval Europe, a high fantasy creator could simply decide that her fictional country is inherently diverse. That’s the preferred way to stage Shakespeare plays nowadays, and has worked for shows like Galavant or the 1997 version Cinderella, starring Brandi and Whitney Houston.

Alternately, creators who do want to keep a connect to history in their work could decide to explore the reality of being a racial minority in a mostly white medieval world, in much the same way Game Of Thrones has explored the reality of being a woman in a patriarchal society. Fans have already done some of their research for them. The Tumblr Medievalpoc has collected hundreds of examples of people of color depicted in real medieval European art. If a high fantasy series were to pick 10 random paintings from Medievalpoc’s collection (starting with, say, this one from England circa 1410, this 1240 manuscript, this 15th century German altarpiece, and these Gondoliers from 15th and 16th century Italy), it could feature an incredibly diverse ensemble without losing its relationship to history.

Admittedly, this method would require a good deal of historical cherry picking, but historical cherry picking is what makes Games Of Thrones so compelling in the first place. The real-life War of the Roses (an oft-cited inspiration for the series) stretched over 30 years, but Martin selected the best bits and crammed them into just a few years to make his show more exciting. Game Of Thrones is regularly praised for its “gritty realism” despite the fact that Westeros is probably far more violent than real-life medieval Europe (as Medievalpoc points out in this excellent essay), So why couldn’t another high fantasy series offer that same pseudo-realistic feel while being a little more diverse?

High fantasy fans are so used to almost exclusively white stories, it can be really hard to shift our mindset about race. But until we do so, we’re just depriving ourselves of compelling narratives. When Vogue asked which show she’d like to appear on the most in 2015, Lupita Nyong’o answered Game Of Thrones. Who wouldn’t want to watch an actor of her caliber sink her teeth into a juicy high fantasy role? And more importantly, why should a misguided commitment to historical accuracy prevent her from doing so?

📬 Sign up for the Daily Brief

Our free, fast, and fun briefing on the global economy, delivered every weekday morning.