Washington (CNN) As football season heats up, the New England Patriots are increasingly unlikely to celebrate their Super Bowl victory with one of their biggest fans -- President Donald Trump.

The two sides told the New York Times that they were unable to come to an agreement over a date that works for both the team and Trump. Both sides say it has nothing to do with politics.

"We would welcome them to the White House in the future should a future date work out," White House spokesman Judd Deere told CNN on Sunday.

Three Patriots players, though, have already said in interviews with the New York Times that they are not interested in attending a ceremony at the White House, one even suggesting he would rather meet with former President Barack Obama.

The friendly relationship between the Patriots and the President has been well-documented, but at times has been bogged down by controversy. Robert Kraft, the team's owner who has been close friends with Trump for decades, was charged in February with soliciting sex acts at a massage parlor. He has pleaded not guilty to those charges.

The President has, over the years, sung the praises of Brady, and said he was "totally innocent" after the "deflategate" scandal in 2016. The day before the election, Trump announced to a rally that Brady had told him he had voted for him. Brady later said he had not voted yet . Brady did not attend the 2017 Super Bowl ceremony at the White House, and he said he was spending time with his sick mother.

Bill Belichick, who has ushered the team through six Super Bowl wins as head coach, wrote a letter to Trump days before the 2016 election, which Trump read on stage at a rally.

When Trump invited the team after their 2017 Super Bowl win, at least two dozen players skipped the event, according to the New York Times. At the time, a Patriots spokesperson told the New York Times that the team's frequent Super Bowl wins could have affected turnout negatively, because White House visits could begin to feel less special.

The Patriots have visited the White House five times after Super Bowl victories. All of their visits have been in April or May, according to the Times.

It wouldn't be the first time a sports team didn't attend a ceremony despite an official invitation from Trump.

Last year, Trump ultimately disinvited the Philadelphia Eagles from celebrating their win at the White House after players and coaches boycotted the visit over Trump's displeasure of players kneeling during the National Anthem. Earlier this year, the University of Virginia basketball team declined to celebrate their victory at the White House, also citing logistical concerns.

The 2018 NBA champion Golden State Warriors spent time with former President Barack Obama during their trip to Washington instead of a White House visit this year. In September 2017, several months after the team won the 2017 NBA championship, Trump rescinded his invitation to the team to visit the White House in honor of their win. The decision came after Stephen Curry, the team's superstar player, publicly criticized Trump's attacks on black athletes protesting police brutality during the National Anthem.

In January, Trump celebrated the victory of the Clemson University football team, providing a feast of various fast food to the players during the government shutdown. And in April, Trump for the first time welcomed a women's sports team to the White House for their own event: the 2019 NCAA women's tournament champion Baylor Bears.