The challenging start to President Trump’s term in office appears to be taking a toll on the very voters who helped elect him: white men.

According to polls taken over the last few weeks, support for the President among white men is eroding. Since March, the percentage of white males who say they back the President has dropped from 58% to 47%, according to polling by Quinnipiac University.

Similarly, an IBD/TIPP poll conducted in March and released earlier this week showed that the President’s support among white men fell from 58% earlier in March to 49%. That poll also showed a decline in support among rural Americans from 56% earlier in March to 41%.

Those figures come as the Trump’s overall approval ratings remain in the 30s, historic lows for a new president. Declining public support could hinder the President as he attempts to usher in sweeping tax reform and a potential $1 trillion infrastructure bill.

The first few months of Trump’s administration have largely been defined by infighting at the White House, allegations of Russian meddling in the 2016 election, and a health care bill that failed to pass Congress, all of which could be weakening support among Trump’s most ardent backers.

“It could be that a candidate who built himself as a winner took a huge loss [on health care],” says Tim Malloy, assistant director of the Quinnipiac poll. “It could be that a candidate who built himself as a leader is presiding over a White House that to the minds of a lot of people is in chaos.”

For many of Trump’s supporters, the President’s election victory came in part because they saw him as a “winner and in their minds tough and a closer,” Malloy says. “And Americans have not seen that side of Donald Trump and not any kind of victory lap yet.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.