Washington (CNN) Last year, Justice Anthony Kennedy traveled to the White House, robes and all, and found himself in a familiar spot: the center of attention.

The assembled audience was there for the swearing in of Justice Neil Gorsuch, but many eyes were trained on Kennedy, who like no other justice in recent history controls the outcome of the highest profile cases before the court.

Kennedy's role that day was to deliver the oath to his former clerk, but some of the judicial conservatives assembled in the Rose Garden were wondering if Kennedy might soon provide them with the opportunity to return to their vetting binders. Would he retire and give President Donald Trump a second vacancy and the chance to cement a conservative majority on the court for decades to come?

A few weeks later, even the President got pulled in to the guessing game: "I've heard the same rumors that a lot of people have heard," he said from the oval office, according to the Washington Times

In the end, Kennedy made no announcement. After almost 30 years on the highest court in the land, he determined he wasn't quite finished.

Now, as retirement speculation ramps up again, court watchers wonder what went into his calculation for staying on the bench last term -- and if anything's changed since then.

Photos: Supreme Court Justice Anthony Kennedy Photos: Supreme Court Justice Anthony Kennedy Anthony Kennedy, the longest-serving member of the current Supreme Court, has announced that he will be retiring at the end of July. Kennedy, 81, was appointed by President Ronald Reagan in 1988. He is a conservative justice but has provided crucial swing votes in many cases. Hide Caption 1 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy was born in Sacramento, California, on July 23, 1936. In this photo, circa 1939, he sits between his mother, Gladys, and his sister, Nancy. Hide Caption 2 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy wears his Cub Scout uniform as he poses with his brother, Tim, circa 1946. Hide Caption 3 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, third from right in the front row, stands with other Cub Scouts in the 1940s. Hide Caption 4 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, spent time with the California Army National Guard after finishing law school in 1961. The man on the left, John J. Hamlyn Jr., also became a lawyer like Kennedy. Hide Caption 5 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, and Hamlyn pose for a photo after basic training. Hide Caption 6 of 38 Photos: Supreme Court Justice Anthony Kennedy After more than a decade as a lawyer, Kennedy became a judge on the US Court of Appeals in 1975. He was nominated by President Gerald Ford on the recommendation of California Gov. Ronald Reagan. Hide Caption 7 of 38 Photos: Supreme Court Justice Anthony Kennedy This courtroom photo of Kennedy was taken in 1976. Hide Caption 8 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy has breakfast with his wife, Mary, and his son Gregory in 1984. Hide Caption 9 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and his wife walk together in Sacramento, California, in 1987. Hide Caption 10 of 38 Photos: Supreme Court Justice Anthony Kennedy From 1965 to 1988, Kennedy was also a professor of constitutional law at the University of the Pacific's McGeorge School of Law. Hide Caption 11 of 38 Photos: Supreme Court Justice Anthony Kennedy In 1987, Kennedy was nominated by President Reagan to fill the Supreme Court seat vacated by Lewis Powell's retirement. The nomination came after the confirmation failures of nominees Robert Bork and Douglas Ginsburg. Hide Caption 12 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, center, talks with US Sens. Ted Kennedy, left, and Joe Biden before a confirmation hearing in Washington. The two Kennedys are not related. Hide Caption 13 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy meets with President Reagan in the Oval Office. Hide Caption 14 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is joined by his wife as he is sworn in by Chief Justice William Rehnquist on February 18, 1988. Reagan is on the right. Hide Caption 15 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, top right, appears in a formal Supreme Court portrait in April 1988. In the front row, from left, are Thurgood Marshall, William Brennan Jr., Chief Justice William Rehnquist, Byron White and and Harry Blackmun. In the back row, from left, are Antonin Scalia, John Paul Stevens, Sandra Day O'Connor and Kennedy. Hide Caption 16 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks at the McGeorge School of Law in 1991. He delivered the inaugural address in a lecture series named for the late Archie Hefner, whose portrait is behind Kennedy. Hefner was a prominent Sacramento attorney active in numerous civic and charitable groups. He died in 1988. Hide Caption 17 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is on the far right in this Supreme Court portrait from 1998. In the front row, from left, are Antonin Scalia, John Paul Stevens, Chief Justice William Rehnquist, Sandra Day O'Connor and Kennedy. In the back row, from left, are Ruth Bader Ginsburg, David Souter, Clarence Thomas and Stephen Breyer. Hide Caption 18 of 38 Photos: Supreme Court Justice Anthony Kennedy In 2004, Kennedy speaks to high school students at the Ronald Reagan Presidential Library in Simi Valley, California. Hide Caption 19 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks during a Senate subcommittee hearing in 2002. Hide Caption 20 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy discusses the court's budget requests with a House committee in April 2005. Hide Caption 21 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy participates in a panel discussion in Washington in November 2005. Hide Caption 22 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy receives an honorary degree at New York University in May 2006. Hide Caption 23 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy delivers the commencement address at New York University. Hide Caption 24 of 38 Photos: Supreme Court Justice Anthony Kennedy In February 2007, Kennedy testifies at a Senate committee hearing on judicial security and independence. Hide Caption 25 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies before a House subcommittee in March 2007. He and fellow Justice Clarence Thomas spoke about concerns with the ongoing remodeling of the court building, the reduction of paperwork due to electronic media, and the disparity of pay between federal judges and lawyers working in the private sector. Hide Caption 26 of 38 Photos: Supreme Court Justice Anthony Kennedy The Supreme Court meets with President Barack Obama and Vice President Joe Biden in September 2009. From left are Samuel Alito, Ruth Bader Ginsburg, Kennedy, John Paul Stevens, Chief Justice John Roberts, Obama, Sonia Sotomayor, Biden, Antonin Scalia, Clarence Thomas, Stephen Breyer and retired Justice David Souter. Hide Caption 27 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy leaves after a Catholic Mass in Washington in October 2009. Hide Caption 28 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy joins the President and other officials at a memorial for the victims of a shooting in Tucson, Arizona, in 2011. Hide Caption 29 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy smiles as he is introduced to faculty members at the University of Pennsylvania Law School in October 2013. Kennedy was teaching there for a week. Hide Caption 30 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is saluted by sailors as he tours the USS John C. Stennis in 2015. Hide Caption 31 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies about a Supreme Court budget request during a House subcommittee meeting in 2015. Hide Caption 32 of 38 Photos: Supreme Court Justice Anthony Kennedy President Obama greets Kennedy and other Supreme Court justices before his final State of the Union address in January 2016. Hide Caption 33 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, second from left, joins other Supreme Court justices in February 2017 during President Donald Trump's first address to a joint session of Congress. Hide Caption 34 of 38 Photos: Supreme Court Justice Anthony Kennedy As President Trump looks on, Kennedy administers the judicial oath to new Justice Neil Gorsuch in April 2017. Hide Caption 35 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and Trump walk together after Gorsuch's swearing-in ceremony. Hide Caption 36 of 38 Photos: Supreme Court Justice Anthony Kennedy Trump stands with the Supreme Court at Gorsuch's formal investiture ceremony in June 2017. From left are Elena Kagan, Samuel Alito, Ruth Bader Ginsburg, Kennedy, Chief Justice John Roberts, Trump, Gorsuch, Clarence Thomas, Stephen Breyer and Sonia Sotomayor. Hide Caption 37 of 38 Photos: Supreme Court Justice Anthony Kennedy Several members of the Supreme Court pose for a portrait before taking part in a procession to mark Harvard Law School's bicentennial in October 2017. On the top row, from left, are Kennedy, Roberts, Breyer and Gorsuch. In front of them are Kagan and retired Supreme Court Justice David Souter. Hide Caption 38 of 38

Significant cases facing the court could have served as the siren call, or maybe Kennedy felt the weight -- and pull -- that one of his colleagues called the "awesome responsibility" of holding the critical vote for the biggest cases. It's conceivable he was spooked by the volatile first few months of the Trump administration, or perhaps retirement is easier to contemplate than to execute.

Whatever motivated Kennedy to stay, conservatives waiting for a chance to push the court to the right were disappointed. And liberals, even though they have been on the losing end of Kennedy votes on issues such as campaign finance and the Second Amendment, now cling to him. He has, after all, given them victories in LGBT rights, abortion access, affirmative action and his replacement under the Trump administration would surely be much younger and more conservative.

There is no underestimating his impact. Kennedy was in the majority for 97% of all cases during the 2016-17 Supreme Court term, according to statistics compiled by Scotusblog . In cases where judges were divided, Kennedy was in the majority 93% of the time, by far the highest percentage of the nine justices.

Partisan gerrymandering

The speculation about Kennedy's future comes as the justices are considering an unusual number of extraordinarily consequential cases where his vote and input will be critical.

For example, partisan gerrymandering is front and center. The Supreme Court has never been able to articulate a test concerning the overreliance on politics in map drawing.

Back in 2004, the court rejected a challenge to a political gerrymander in Pennsylvania. While four justices believed that courts should never hear such cases, Kennedy held the door open in a separate opinion.

Kennedy acknowledged that ordering a correction of all election district lines for partisan reasons might amount to an "unprecedented intervention in the American political process," but he would not foreclose the possibility of judicial relief down road. He said that in the future a "limited and precise rationale" might emerge.

That day may or may not have come this term as the court is hearing not one, but two cases concerning the issue.

At oral arguments during one of the cases concerning Wisconsin last October, Kennedy did seem open at one point to the possibility that a standard grounded in the First Amendment could be found.

In late March, the court heard a second challenge regarding a map in Maryland. It was clear during oral arguments that the justices were still struggling with the issue.

At one point Justice Stephen Breyer even suggested that the court take the two cases and another one pending out of North Carolina and hold rearguments next term.

"Some of us saw this as a plea to get Justice Kennedy to stay on the court to see this to the end," Rick Hasen a professor at University of California-Irvine, opined

A workable framework could be a landmark opinion that could revolutionize how states draw their lines.

"If Justice Kennedy agrees with plaintiffs in either of these cases, then he'll be known for decades as the justice who succeeded in rooting out the worst abuses in partisan gerrymandering," said Josh Douglas, a professor University of Kentucky College of Law. "Making his mark on this issue could be one of the reasons he hasn't retired yet."

JUST WATCHED How gerrymandering got its name Replay More Videos ... MUST WATCH How gerrymandering got its name 01:57

Gay marriage and religious liberty

Kennedy will forever be remembered for penning Obergefell v. Hodges, the 2015 opinion that cleared the way for same-sex marriage nationwide.

This term, the justices are considering Masterpiece Cakeshop v. the Colorado Civil Rights Commission , a case that some believe is a follow up to Obergefell.

In Obergefell, Kennedy wrote that opponents to same-sex marriage reached their conclusions based on "decent and honorable religious" beliefs.

In Masterpiece, however, Colorado cake baker Jack Phillips, who believes marriage should be between a man and a woman, refused to make a cake for customers seeking to celebrate their same-sex marriage.

Now Kennedy might have to reconcile two strands of his jurisprudence: the dignity of same-sex couples and the respect for religious liberty.

At oral arguments, Chief Justice John Roberts referred back to Kennedy's words in Obergefell and noted that the majority had "gone out of its way to talk about the decent and honorable people who may have opposing views."

Kennedy did seem, at one point, to be receptive to the religious liberty concerns of the baker. Kennedy pointed out that a commissioner on the Colorado Civil Rights Commissioned had expressed hostility toward Phlipps's beliefs.

"It seems to me that the state in its position here has been neither tolerant nor respectful of Mr. Phillips's religious beliefs," Kennedy said.

JUST WATCHED Hear Supreme Court arguments about cakeshop Replay More Videos ... MUST WATCH Hear Supreme Court arguments about cakeshop 03:11

Ivanka Trump visits the court

Journalist Charlie Rose asked Justice Ruth Bader Ginsburg in 2017 about Kennedy's role as the deciding vote in so many hot button cases.

It's an "awesome responsibility" she said.

At another event held in California in 2017, one questioner took a different tack when asking the question.

"Who do you want to eat more kale in Washington?" the questioner asked.

Ginsburg and other liberals are well aware of the President's promise to reshape the courts.

One question is whether Kennedy would feel comfortable retiring during the Trump administration.

Heads spun just after the inauguration in February, when Ivanka Trump appeared at the court and sat in the section reserved for VIPs to hear oral arguments. According to the court's spokeswoman, the President's daughter was there as the invited guest of Kennedy, who she met at the inaugural lunch at the Capitol.

It wasn't lost on court watchers that Trump chose a former Kennedy clerk as his first pick. Other former Kennedy clerks such as Judge Brett M. Kavanaugh, of the US Court of Appeals for the District of Columbia Circuit, and Judge Raymond Kethledge of the 6th Circuit are high on a list of potential Supreme Court nominees.

Midterm impacts

Like every other Justice, Kennedy cares about the institution of the court and he will take that into consideration whenever he chooses to step down.

As a rule, justices don't like to retire in an election year, always trying to keep the court at an arm's length from the political process.

When Justice Antonin Scalia unexpectedly died in 2016 the court was forced to work with only eight members for more than a year as the political branches went to war over Scalia's replacement. The eight-member court is not ideal because if the justices split 4-4, they are simply left to affirm the lower court opinion creating no new precedent.

And all eyes will be on November, with the potential for Democrats to take over the Senate.

In January, conservative court watcher Ed Whelan wrote for the National Review that "if the rumors are true that Justice Kennedy has been looking to retire, then it seems a reasonable bet that he would do so this spring," he said.

"Indeed, the very real prospect that Democrats will win control of the Senate in the November 2018 elections might clinch his decision to do so. If he waits until next year, and if Democrats take control of the Senate, his seat would probably remain empty until 2021," Whelan speculated.

Between now and June, Kennedy has almost a full docket of cases to consider. It's a real possibility that he hasn't even made up his mind.

He's hired clerks for next term, but warned at least one, two sources say, that retirement is always a possibility. There is precedent for the clerks who are hired by one justice to be absorbed by another justice if a retirement occurs. Indeed, Gorsuch was hired by Jusstice Byron White but served out his clerkship with Kennedy after White retired in 1993.

Court watchers, even some in the White House at the time, thought that Justice John Paul Stevens would retire long before he finally did so in 2010.

And some may say that Kennedy, is still a youngster at 81.