A few weeks ago, I wrote an article recommending a number of books that might help those reeling from the events of 2016 to have a better 2017. It was a piece I was writing to myself as much as it was for anyone else. The purpose was to suggest a collection a number of helpful titles on resilience, open-mindedness, effective political resistance, love and focus.

I won’t bore you with the response (most of it was good), but given that I had created a list of things that I liked, the post was subject to what one might call the first law of social justice warrior culture: people tried to find all the ways it was sexist and offensive. Like the “FIRST!” comments that appear below every YouTube video, a number of readers wrote in very pointedly to complain about the lack of gender diversity on the list.

“Only one woman on this list, I see,” one emailed in. Another counted differently, “Every single book you’ve mentioned in your post was written by a man. How is this even possible? So disappointed. It’s fucking 2016.” In fact, both were wrong, unless Arlie Russell Hochschild and Sarah Bakewell are no longer women.

As much as I wish that were a joke, it wasn’t nor is it a phenomena limited to my articles.

Pull up an article of book recommendations. Or a piece that features quotes or a helpful collection of anecdotes. Then scroll all the way to the bottom and look at the comments. Almost inevitably you’ll find some version of those complaints. I’ve even seen it lately with Republicans who think that conservatives don’t get their due (“Where’s your story about Reagan?!” “What about all the non-offensive things Trump has said?”) It’s become its own online viral trend—everyone knows there is traffic in posts lambasting some writer for forgetting this important subgroup or assailing those editors for not recognizing the importance of that influence. It’s an argument that has become a much used leg in the stool upon which our outrage culture stands.

To a large degree, I understand the sentiment. I care about breadth and inclusion in my own writing and I have certainly drawn from diverse sources in my largely self-driven education. It was also something I learned as a research assistant for the author Robert Greene—that one must scour every culture, era and source to find universal truths. I very vividly remember him telling me that if all your examples are old dead white guys, you’re missing out on important perspectives and your work will be boring and hard to relate to. As a creator, you should never want someone to pick up a book or a list of books and think: “I’m not represented here at all.” Allowing that to happen is lazy writing and if your goal is to reach a large audience and have impact with them, it’s bad strategy too.

Yet like we have with many debates in our sanctimonious media culture, we’ve begun to take this criticism too far. Worse, I think it’s actually discouraging the diversity it seeks.

The children’s author Kwame Alexander recently wrote about well-meaning teachers and librarians who ask about the race of the animal characters in his books. They are asking because they think it will help the children relate to the characters. This misses the entire point of literature, he says, which is not to serve as a mirror, but as a window.

The social justice warrior tendency to decry the lack of diversity in “best of” lists makes a similar mistake. It confuses diversity of authorship with diversity of thought and message.

I once wrote a list called “24 Books You’ve Probably Never Heard Of But Will Change Your Life.” The list was clearly diverse in one sense because it included a book by Florida Scott-Maxwell and two black authors. But what’s exciting to me about those books is not their literal diversity, but instead in how unusual they were—Maxwell’s book was the touching end-of-life memoir of a female playwright and Jungian psychologist while Solomon Northup’s 12 Years a Slave was the story of an educated freedman who was kidnapped and sold into slavery, while Jack Johnson’s memoirs were originally published in a French newspaper.

To simply focus on racial or gender diversity also missed that the subject matter in the other books on my list ranged from narratives about a young white boy sold into slavery, the private diaries of the Emperor of Rome, a father writing a profoundly moving book about losing his son to a brain tumor, a journalist who checked himself into an asylum in the thirties for his alcoholism to the memoir of a professional gambler, fighter and criminal who rode the riverboats of the Mississippi in the 1800s. The list was not supposed to demographically represent society. It was supposed to represent books most people hadn’t heard of but would be well served to read. End of story.

To connect this to another debate on diversity, Eric Ruiz, describing himself as a “Latino in Tech” wrote here in the Observer that advocates obsessed with numbers and race in Silicon Valley are actually counterproductive: “To quantify diversity by race or skin color is lazy and disrespectful. I’m not asking nor arguing we have less diversity. No, instead I challenge us to widen our scope and view diversity from an additional vantage point, one where race is but a way a person can be diverse and not the sole factor. In fact, in an ever more multicultural society, race may be the least important factor in how we differentiate ourselves.” (Similarly, Marc Andreessen has pointed out that “white and Asian” in Silicon Valley happens to gloss over a group of “Russians, and Eastern Europeans, and French, and German, and British…the Chinese, Japanese, Koreans, Thais, Indonesians, and Vietnamese” people.”)

A similar argument can be made about literature. On the one hand, we have 5,000 years of literature and storytelling history, most of which was the product of a male-dominated society (and other forms of discrimination in the West). To read the Western Canon is an inherently lopsided exercise in one sense. But this is a superficial way to look at it. Even with centuries of systemic discrimination, the millions of books by thousands of authors are astoundingly diverse and varied. Some were rich, some were poor, some were from the far flung colonies and hardly considered “pure” in their own time, others were slaves (of every race), some were pessimists, others optimists, some were famous, others were outcasts. And this doesn’t even get into the topics that those writers wrote about—does it really matter what Sophocles’ gender was and does that really speak louder than his inspiring, strong female characters? It is not absurd to consider that a sexist White Supremacist could decide they were only going to read books written by the white men of history, never leave the titles in the Penguin Classics and Library of America catalog and still be exposed to radically opposing and different ideas.

Likewise, it would be offensive to categorize Ta-Nehisi Coates and Thomas Sowell primarily by their race. To read Between The World and Me and Knowledge and Decisions is to expand one’s horizon significantly more than to read two ethnically divergent authors in intellectual agreement. J.D. Vance’s Hillbilly Elegy is “another” book by a rich white guy, yet it contains a radically fresh and unusual perspective (To be honest, I found it more sympathetic than Arlie Hochschild’s Strangers in Their Own Land). Of course, diversity is important and it’s good to be on the lookout for instances in which talented minorities are not given their proper due. The problem is when this “Caught you!” mentality is not only counter-productive—it’s often needlessly combative.

Jerry Seinfeld joked on his episode of Comedians in Cars Getting Coffee with Steve Harvey about a firestorm of criticism over the supposed lack of diversity in the first 10 or so episodes of his webseries, “If you walked into my house and I had 10 people over for dinner and they were all white, would you come in and go, ‘Oh, whoa, what’s going on in here? What is going on in here? Is this a rally?’”

Such is the ridiculousness of using very small samples sizes to make very large assumptions. But this is the assumption we love to jump to—because it’s good for headlines and outrage. Last year, Gay Talese was pilloried at 84 years of age for supposedly saying in an interview that he wasn’t inspired by “female writers.” Of course, in digging in, one finds that he was actually asked what female journalists had inspired him in his formative years in the industry. Later, clarifying what female fiction writers inspired him early in his career and which female journalists he admired later in his career, he named several including Mary McCarthy, Carson McCullers, Susan Orlean, Larissa MacFarquhar, Lillian Ross and Nora Ephron. (I happened to have recommended 4 of those 6 writers on other lists of books I’ve written). And Seinfeld, for his part dismissed the criticism, yet has steadily featured a wide swath of comics from unusual backgrounds and lifestyles.

The problem with the presumption of guilt and the vehemence with which we brand people with horrible, incendiary labels—Jezebel insisted that Talese’s clarification still didn’t matter and he was obviously a diehard sexist—is that it misses our shared connection, whether it’s a love of books or an agreement about certain world events. I’ve been amazed since I published The Obstacle is the Way how many conservatives angrily wrote in about the fact that I included a chapter on Obama. From this, they make all sorts of wild assumptions about who I am (and happen to miss that I am a gun-owning registered Republican who lives in Texas) and also seem to miss the equally laudatory stories about Margaret Thatcher, Abraham Lincoln, Andrew Garfield, Ulysses S. Grant, Winston Churchill and a number of other people they probably do agree with. It’s the same misguided, persecutory tendency that one makes when they assume that someone is a sexist, or indifferent to racial struggle.

Having monitored a number of these comments over the years and gotten plenty of snide emails through my book club is that the “But where are the [insert group]?” question is almost never followed with a specific example of who should have been included instead. It never seems to occur to people that a better response is to suggest what you’d like to see instead. That’s how conversations happen. That’s how people who once disagreed, come eventually, to share mutual affinity for the same art or perspectives.

It’s easy to judge and act superior. It’s easy to be what Tim Ferriss has called a “bigoteer.” But is that really productive? Because it seems to me that that is just a way to dismiss the material and not have to interact with it.

Across the country, college students are doing just that. Courses on poetry or Western literature are protested because they represent patriarchy or hegemonic ideas. Rudyard Kipling is dismissed because he was an imperialist…and in the process a guy who grew up in India, experienced unimaginable adversity and abuse as a young child, and lived in a unique period in history is written off (to say nothing of the beautiful writing that readers then miss). The most important and formative periods of literature and culture are snorted at derisively because of what they don’t include—and the wisdom which they do include is missed by a generation who could use it most. Who is to say the “Bechdel Test” is actually the rubric to measure anything by? Especially when, by that rubric, plenty of feminist films would be dismissed—and plenty of uninspiring and lame movies would get the seal of approval?

Yet that is precisely what we risk when postmodern condescension meets safe-space-craving millennial entitlement meets outrage identity politics.

As readers and as writers, it’s not quotas we should be chasing. It should be truth. And very rarely does what is featured in the author photo—or in the case of ancient books, the classic bust featured on the back—have much bearing on the truth of the work.

Most importantly, the journey to find truth—windows, as Kwame Alexander beautifully put it—is going to be different for each one of us. Time is short (even the most dedicated reader’s life is only going to feature a few thousand titles at most) and individual preferences will vary.

Let each reader go on their journey without judgment or condemnation, and leave them to their search. Unless, of course, you have an amazing recommendation or suggestion. In which case, share it.

Ryan Holiday is the best-selling author of Trust Me, I’m Lying: Confessions of a Media Manipulator. Ryan is an editor-at-large for the Observer, and he lives in Austin, Texas.

He’s also put together this list of 15 books that you’ve probably never heard of that will alter your worldview, help you excel at your career and teach you how to live a better life.

Also by Ryan Holiday:

We Are Living in a Post-Shame World—And That’s Not a Good Thing

We Don’t Have a Fake News Problem—We Are the Fake News Problem

Want to Really Make America Great Again? Stop Reading the News.

Exclusive Interview: How This Right-Wing ‘Troll’ Reaches 100M People a Month

The Real Reason We Need to Stop Trying to Protect Everyone’s Feelings

Peter Thiel’s Reminder to the Gawker Generation: Actions Have Consequences

The Cause of This Nightmare Election? Media Greed and Shameless Traffic Worship