Paul Ryan has had a trying election season thanks to his party's nominee, Donald Trump.

"Will you support him?" Jake Tapper asked Ryan in an interview on CNN in May.

"To be perfectly candid with you, Jake, I'm just not ready to do that at this point," Ryan replied.

But Election Day might not put an end to his electoral headaches.

See images of Paul Ryan throughout his career:



30 PHOTOS Paul Ryan through his career See Gallery Paul Ryan through his career Speaker of the House Denis Hastert (L) administers the oath of office to Rep. Paul Ryan (R) of Wisconsin as his family looks on January 6, 1999 at the start of the 106th Congress. The oath is a recreation as the formal oath is administered to the entire congress as a body on the floor of the House. (photo by Rex Banner) UNITED STATES - DECEMBER 12: Rep. Paul Ryan, R-Wis., speaks at a news conference in which House Republican leaders called for Permanent Tax Relief. (Photo By Tom Williams/Roll Call/Getty Images) KRT US NEWS STORY SLUGGED: SOCIALSECURITY-DISCUSSION KRT PHOTOGRAPH BY GEORGE BRIDGES/KRT (April 14) Conversation on Social Security between Rep. Paul Ryan (R-WI), shown, and William Novelli, head of AARP, April 5, 2005 (lde) 2005 (Photo by George Bridges/MCT/MCT via Getty Images) UNITED STATES - JULY 22: MEDICARE BRIEFING--Rep. Paul Ryan, R-Wisc., speaks at a Cato Institute briefing on Medicare reform in the Rayburn House Office Building. Tom Miller, director of Health Policy Studies at Cato, looks on. (Photo by Scott J. Ferrell/Congressional Quarterly/Getty Images) WASHINGTON - SEPTEMBER 24: Rep. Paul Ryan (R-WI) questions Peter Orszag, director of the Congressional Budget Office, during a hearing on Capitol Hill about the impact of recent market turmoil on the federal budget on September 24, 2008 in Washington, DC. Orszag reported that while the impact is currently unknown, it is likely to be substantially less than $700 billion. (Photo by Brendan Hoffman/Getty Images) WASHINGTON - APRIL 27: House Budget Committee ranking member Rep. Paul Ryan (R-WI) (R) delivers an opening statement during a conference committee meeting with Senate Budget Committee Chairman Kent Conrad (D-ND) (L) and House Budget Committee Chairman John Spratt (D-SC) in the U.S. Capitol April 27, 2009 in Washington, DC. House and Senate lawmakers have already struck a tentative deal on the FY2010 budget resolution and they hope to file a conference report after today's meeting. (Photo by Chip Somodevilla/Getty Images) WASHINGTON - MARCH 19: (L-R) U.S. House Minority Whip Rep. Eric Cantor (R-VA), Rep. Dave Camp (R-MI) and Rep. Paul Ryan (R-WI) listen during a news conference on the health care legislation March 19, 2010 on Capitol Hill in Washington, DC. The House will vote on the Health Care Reform Legislation on Sunday, March 21. (Photo by Alex Wong/Getty Images) Representative Paul Ryan, a Republican from Wisconsin and chairman of the House Budget Committee, speaks during a news conference at the U.S. Capitol in Washington, D.C., U.S., on Tuesday, April 5, 2011. U.S. House Republicans today unveiled a plan to overhaul the federal budget and slash the deficit in coming years by about three-quarters, with a $6-trillion cut in spending and 25 percent cap on tax rates. Photographer: Joshua Roberts/Bloomberg via Getty Images MEET THE PRESS -- Pictured: Rep. Paul Ryan (R-WI) left, and moderator David Gregory, right, appear on 'Meet the Press' in Washington, D.C., Sunday, April 10, 2011. (Photo by William B. Plowman/NBC/NBCU Photo Bank via Getty Images) MILWAUKEE, WI - APRIL 01: Republican Presidential candidate, former Massachusetts Gov. Mitt Romney (R) jokes with U.S. Rep Paul Ryan (C) (R-WI) during a pancake brunch at Bluemound Gardens on April 1, 2012 in Milwaukee, Wisconsin. With less than a week before the Wisconsin primary, Mitt Romney continues to campaign through the state. (Photo by Justin Sullivan/Getty Images) WASHINGTON, DC - APRIL 26: House Budget Chairman Rep. Paul Ryan (R-WI) is introduced before speaking about 'America's Enduring Promise,' and the federal budget, in a speech at Georgetown University April 26, 2012 in Washington, DC. During his speech, Ryan said that his proposed budget confronts the nation's growing $15 trillion debt before it impacts future generations of Americans. (Photo by Win McNamee/Getty Images) NORFOLK, VA - AUGUST 11: Republican presidential candidate, former Massachusetts Gov. Mitt Romney (L) jokes with Rep. Paul Ryan (R-WI) (R) after announcing him as the 'next PRESIDENT of the United States' during an event announcing him as his running mate in front of the USS Wisconsin August 11, 2012 in Norfolk, Virginia. Ryan, a seven term congressman, is Chairman of the House Budget Committee and provides a strong contrast to the Obama administration on fiscal policy. (Photo by Win McNamee/Getty Images) UNITED STATES - SEPTEMBER 14: Rep. Paul Ryan, R-Wis., Vice Presidential candidate, waves to the crowd after addressing the Values Voter Summit at the Omni Shoreham Hotel in Woodley Park. Photo By Tom Williams/CQ Roll Call) NEWPORT NEWS, VA - SEPTEMBER 18: Republican vice presidential candidate, U.S. Rep. Paul Ryan (R-WI), pauses as he speaks during a campaign rally at Christopher Newport University September 18, 2012 in Newport News, Virginia. Ryan continued to campaign for the upcoming presidential election. (Photo by Alex Wong/Getty Images) MIAMI, FL - SEPTEMBER 22: Republican vice presidential candidate, U.S. Rep. Paul Ryan (R-WI) hugs waitress, Lourdes Alcerro, during a campaign stop at Versailles restaurant in the Little Havana neighborhood on September 22, 2012 in Miami, Florida. Ryan continues to campaign for votes across the country. (Photo by Joe Raedle/Getty Images) Republican presidential candidate Mitt Romney (R) and running-mate Paul Ryan share a laugh as they are introduced at a campaign rally September 25, 2012 at Dayton International Airport in Vandalia, Ohio. AFP PHOTO/Mandel NGAN (Photo credit should read MANDEL NGAN/AFP/GettyImages) Representative Paul Ryan, a Republican from Wisconsin, arrives at a news conference on Capitol Hill in Washington, D.C., U.S., on Tuesday, Oct. 20, 2015. Ryan said he'd be willing to run for speaker of the U.S. House if Republicans unify behind him now, end leadership crises and let him continue spending time with his family. Photographer: Andrew Harrer/Bloomberg via Getty Images Representative Paul Ryan, a Republican from Wisconsin and chairman of the House Ways and Means Committee, center right, walks down the steps of the U.S. Capitol building following a vote in Washington, D.C., U.S., on Friday, Oct. 9, 2015. Ryan is under heavy pressure from fellow Republicans to run for U.S. House speaker after a hard-line faction forced Speaker John Boehner to resign and his top lieutenant to drop out of the race. Photographer: Andrew Harrer/Bloomberg via Getty Images Representative Paul Ryan, a Republican from Wisconsin, walks to a meeting in the basement of the U.S. Capitol in Washington, D.C., U.S., on Tuesday, Oct. 20, 2015. Ryan is set to meet with a group of House conservatives Tuesday as he weighs a potential run to replace Speaker John Boehner under pressure from fellow Republicans. Photographer: Andrew Harrer/Bloomberg via Getty Images Representative Paul Ryan, a Republican from Wisconsin and chairman of the House Ways and Means Committee, center, talks to the media after walking out of the U.S. Capitol building following a vote in Washington, D.C., U.S., on Friday, Oct. 9, 2015. Ryan is under heavy pressure from fellow Republicans to run for U.S. House speaker after a hard-line faction forced Speaker John Boehner to resign and his top lieutenant to drop out of the race. Photographer: Andrew Harrer/Bloomberg via Getty Images UNITED STATES - OCTOBER 20 - Rep. Paul Ryan, R-Wis., speaks at a news conference following a House Republican meeting, on Capitol Hill in Washington, Tuesday, Oct. 20, 2015. Ryan is stating that he will run for speaker only if he receives enough GOP support by the end of the week. (Photo By Al Drago/CQ Roll Call) US Speaker of the House Paul Ryan, Republican of Wisconsin, awaits the arrival of Israeli President Reuven Rivlin for a meeting at the US Capitol in Washington, DC, December 10, 2015. AFP PHOTO / SAUL LOEB / AFP / SAUL LOEB (Photo credit should read SAUL LOEB/AFP/Getty Images) WASHINGTON, DC - DECEMBER 10: House Speaker Paul Ryan (R-WI) holds his weekly press briefing on Capitol Hill on December 10, 2015 in Washington, D.C. Paul Ryan spoke on topics including Donald Trump and the spending bill. (Photo by Allison Shelley/Getty Images) WASHINGTON, DC - DECEMBER 15: Speaker of the House Paul Ryan, (R-WI) speaks during a Politico interview at the Grand Hyatt on December 15, 2015 in Washington DC. Ryan was interviewed by Politico's Chief White House Correspondent Mike Allen during a Politico Playbook Breakfast. (Photo by Mark Wilson/Getty Images) U.S. House Speaker Paul Ryan (R-WI) (L) holds ceremonial swearing-in for Representative-elect Ralph Norman (R-SC) on Capitol Hill in Washington, U.S., June 26, 2017. REUTERS/Yuri Gripas Speaker of the House Paul Ryan (R-WI) speaks after Senate Republicans unveiled their version of legislation that would replace Obamacare on Capitol Hill in Washington, U.S., June 22, 2017. REUTERS/Joshua Roberts Speaker of the House Paul Ryan introduces his new tax policy at the National Association of Manufacturers Summit in Washington, U.S., June 20, 2017. REUTERS/Aaron P. Bernstein Speaker of the House Paul Ryan visits members of the Republican team prior to the Congressional Baseball Game at Nationals Park in Washington, U.S., June 15, 2017. REUTERS/Aaron P. Bernstein U.S. Speaker of the House Paul Ryan speaks to the press about President Donald Trump, former FBI Director James Comey and Russia investigations as Republican Conference Chairman Rep. Cathy McMorris Rodgers (R) looks on after a closed meeting of the Republican leadership of the House of Representatives on Capitol Hill in Washington, U.S. May 17, 2017. REUTERS/Aaron P. Bernstein Speaker of the House Paul Ryan (R-WI) listens as U.S. President Donald Trump speaks during a meeting with members of the Republican Congressional leadership at the White House in Washington, U.S., June 6, 2017. REUTERS/Joshua Roberts Up Next See Gallery Discover More Like This HIDE CAPTION SHOW CAPTION of SEE ALL BACK TO SLIDE

SEE MORE: Paul Ryan Has Lost A Lot Of Popularity Among Republicans

A new report from The Hill said Ryan's position as speaker of the house could be in jeopardy — even if Republicans are able to hold on to their majority in the lower chamber.

To become speaker of the house, a candidate from the party in power needs the support of a simple majority, or 218 of the 435 representatives.

Ryan's unwillingness to embrace Trump drew criticism from the candidate himself on Twitter, and some House Freedom Caucus members are reportedly upset with Ryan behind the scenes, as well.

But even before Trump, lots of Republicans weren't sold on Ryan taking over as speaker after John Boehner retired.

Scott Wong wrote in The Hill that if Clinton beats Trump in a close race, "he and his die-hard loyalists will almost certainly try to pin some of the blame on Ryan."

It could only take a few angry representatives pulling their support from Ryan to ensure he doesn't have the support needed to remain as house speaker. Now rumors are swirling that, rather than lose an election, Ryan might simply step down.

But Ryan's camp insists he's not thinking about that. A spokesperson said, "The speaker's only focus until Election Day is defeating Democrats and protecting our majority, and nothing else."

RELATED: Politicians that refuse to support Trump

