(CNN) David Glosser, uncle to White House senior adviser Stephen Miller, defended an editorial excoriating his nephew and the administration's immigration policy Tuesday, telling CNN, "I felt it was incumbent upon me to raise my voice to let people know that this is a country of immigrants and our family were immigrants -- in fact, we were refugees."

"If my ancestors had not immigrated to the United States when they did, if they waited a few more years until 1924, the door would have been shut," Glosser told CNN's John Berman on "New Day." "My parents would have gone up the crematory chimney, I wouldn't have born, my sister wouldn't have been born and certainly Stephen would never have existed."

Glosser's op-ed, published Monday in Politico, blasts Miller for being an "Immigration Hypocrite," and details his Jewish ancestors' decision to flee Antopol in the early 1900s, a village whose Jewish population was decimated during the Holocaust during World War II. Glosser writes, "I have watched with dismay and increasing horror as my nephew, an educated man who is well aware of his heritage, has become the architect of immigration policies that repudiate the very foundation of our family's life in this country."

"The reason I wrote it now is because the administration had an increasingly hostile posture towards desperate people trying to enter the country," Glosser, a retired neuropsychologist, told CNN Tuesday. "I had been posting my opinions on Facebook for the past year or two, but various members of the family as well as myself thought I could no longer remain, have a, shall we say, a quiet voice on the subject in light of the incarceration of all these children. This was an act of inexcusable cruelty."

Glosser doesn't expect to change his nephew's mind anytime soon, telling CNN, "No, it appears he's made his political and personal career on this single issue, for reasons I don't really know." Still, he hopes his family's story will stir sentiment in others.

Read More