As a comedian and writer I have been called a lot of things—but up until yesterday no one had ever said I was a “baby bully.”

What did I do to deserve this new title? Have I been going around the neighborhood and shaking down babies for their lunch money? Have I been forcing babies to walk on the other side of the street when they see me? Nope.

I told a joke that many on the right—and even a few in the middle—didn’t like. This all happened Sunday when I was on MSNBC’s Melissa Harris Perry show with three other comedians for an episode called: “Look back on 2013 and Laugh.”

One of the segments featured the panelists trying to come up with funny captions for various photos, including a photo of Mitt Romney’s extended family with their recently adopted African-American baby, “Kieran.” This photo had been tweeted out to the world by Romney on Christmas Eve.

Upon seeing the photo, I commented: “It really sums up the diversity of the Republican party, the RNC. At the convention, they find the one black person.”

You may think that’s funny, unfunny, cliché, idiotic, etc. Comedy is subjective. But say what you will about my quip, I didn’t criticize the Romney family for adopting the baby nor did I mock baby Kieran in any way. Nor would I.

My joke was about the lack of racial diversity we see at the Republican National Convention—a topic lampooned for years by comedians. Little tip to GOP: If you can fit the name of all black leaders in your party in one tweet, you aren’t racially diverse.

While I received a good number of tweets about the show that day—not one person mentioned that joke or the use of the Romney photo. But on Monday apparently those on the right who monitor MSNBC were outraged. And by Monday night there were articles denouncing us as “baby bullies” in right-wing media outlets like The Blaze, Town Hall, The Daily Caller, Breitbart, et cetra. Fox News’ The Five covered the story with five its team of “racially diverse” co-hosts expressing outrage over racial insensitivity.

And, of course, my Twitter feed exploded with right-wing trolls attacking me. Leading the charge was conservative Dana Loesch who tweeted various comments to me such as: “You’re a baby-bullying bigot.” (Amazing use of alliteration, I must say.) She then apparently confused me with someone who in leadership in the Democratic Party: “Continue to show everyone you’re the party of bullying black babies and suppressing minority involvement in other parties.”

As the attacks on my joke were building, I kept waiting for Sarah Palin and Bobby Jindal to defend me like they recently stood up for Duck Dynasty’s Phil Robertson. I thought at least Jindal would call out the “politically correct crowd” as he did for good ol’ Phil. But still not a peep from them about my freedom of expression. I guess to those two, freedom of expression only applies when you are making derogatory comments about gays or saying blacks were happy during segregation.

Here’s the thing: As a comedian, I always try to be funny. It doesn’t always work and I have told jokes that offended people. And I can assure you that in the future I will offend even more people even though that was not my intention. Not only is comedy subjective, but so are sensibilities about when a comedian has “crossed the line.”

In fact, being attacked by right-wing publications for my jokes is nothing new to me. I even wrote an article about that just a few months ago for The Daily Beast titled “The Tea Party’s War on Comedy” about right-wing media outlets lashing out a joke I tweeted.

But here’s the reality: We can expect to see even more of this outrage by both the left and the right going forward. Our collective self-righteous anger keeps escalating. Perhaps it’s because of the hyper-partisan times we live in. Or maybe it’s due to social media or the media’s desperate need for content. Perhaps it’s just payback by each side for the last time one of their own was attacked. Regardless of the reason, in time, it will only get worse.

With that said, let me be clear: I want to sincerely apologize to the Romney family if anyone was offended by my joke. I did not in anyway mean to attack the Romney family for adopting a child, which is truly commendable. Nor did I intend to mock baby Kieran in any way. I would never intentionally demonize people in that manner. (Melissa Harris Perry also apologized on Twitter.)

And let me also be clear to the self-appointed right-wing pundits: I will never stop calling out the wrongs and hypocrisy of the right. Be it citing Jesus’ name to justify slashing programs that help the less fortunate, demonizing Muslims or gays for political gain, or trying to disenfranchise minority voters with voter ID laws. And for those jokes and comments, I can assure you, I will never apologize.