Vowing to not remove any monuments from federal lands, Secretary of the Interior Ryan Zinke defended the Trump administration’s stance about not taking down Confederate statues and said it would upset “native Indians.”

“Where do you start and where do you stop? … If you’re a native Indian, I can tell you, you’re not very happy about the history of General Sherman or perhaps President Grant,” Zinke said during an interview with Breitbart Sunday, referencing the Union generals’ monuments around the U.S. despite their roles in creating federal policy that caused great harm to native Americans.

While Zinke has maintained this opinion about Confederate monuments since at least July, tensions over memorials for Confederate soldiers has risen significantly since August when a counter protester was killed at a white nationalist rally in Charlottesville, Virginia. The white supremacists gathered to protest the removal of a statue of Robert E. Lee in the city.

Zinke said removing the statues will inhibit the U.S. from being able to “learn” from history.

“I think we should never hide from our history or erase our history. I think we should embrace the history and understand the faults and learn from it. But when you try to erase history, what happens is you also erase how it happened and why it happened and the ability to learn from it,” Zinke said.