Washington (CNN) Justice Anthony Kennedy, the man who so often determines the outcome of the most controversial Supreme Court cases, is himself the center of brewing speculation.

Will he stay or will he go?

The rumors have swirled for months and the 80-year-old justice has done nothing either personally or though intermediaries to set the record straight on whether he will step down.

Helping drive the speculation, dozens of Kennedy's former law clerks traveled to Washington this weekend to participate in a private clerk reunion that occurs regularly -- and many of them wondered if it will be their last chance to meet with him while he is still on the bench.

At the end of a dinner with the former clerks Saturday night, Kennedy addressed the crowd, saying he had heard some speculation about an announcement tonight, "and here it is," he said: The "bar will be open after dinner."

But sources close to Kennedy say that he is seriously considering retirement, although they are unclear if it could occur as early as this term.

His departure would cause a seismic shift and offer President Donald Trump a chance to continue reshaping the court. Trump's first nominee -- Justice Neil Gorsuch, himself a former Kennedy clerk -- joined the court earlier this year.

President Reagan meeting with Judge Anthony Kennedy in the Oval Office

Why is Kennedy so important?

Like no other justice in recent history, Kennedy has cast the vital swing vote in cases that grab the countries' attention.

To liberals he is a hero for Obergefell v. Hodges -- a landmark opinion that cleared the way for same-sex marriage in 2015 and will likely be his most lasting legacy.

"They ask for equal dignity in the eyes of the law and the Constitution grants them that right," Kennedy wrote.

To the delight of abortion rights supporters, Kennedy voted to reaffirm the core holding of Roe v. Wade in 1992.

"As the court's most important Justice -- at the center of the institution's ideological balance -- Justice Kennedy's ability to bridge the divide between left and right on critical issues such as the right to access abortion cannot be overstated," said Elizabeth Wydra, president of the Constitutional Accountability Center. "Replacing Justice Kennedy with a Trump nominee would almost certainly sound the death knell for Roe, just as candidate Trump promised during the 2016 campaign."

But nine years later, he sided once again with the liberals on the court to strike down a Texas law that abortion rights supporters thought was the most strict nationwide. Without Kennedy's vote, the law would have been allowed to go into effect, inspiring other states to pass similar legislation.

In the same term, Kennedy pivoted on the issue of affirmative action when he voted for the first time in favor of a race-conscious admissions plan at a public university.

After that term, former acting Solicitor General Neal Katyal said, "It is very much Justice Kennedy's Court."

"You can't understand how important his affirmative action opinion is without understanding his earlier jurisprudence," said Katyal. "For decades, he has been the court's most eloquent voice on the need to be color blind -- why he changed his mind is something historians will debate for decades."

Photos: Supreme Court Justice Anthony Kennedy Anthony Kennedy, the longest-serving member of the current Supreme Court, has announced that he will be retiring at the end of July. Kennedy, 81, was appointed by President Ronald Reagan in 1988. He is a conservative justice but has provided crucial swing votes in many cases. Hide Caption 1 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy was born in Sacramento, California, on July 23, 1936. In this photo, circa 1939, he sits between his mother, Gladys, and his sister, Nancy. Hide Caption 2 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy wears his Cub Scout uniform as he poses with his brother, Tim, circa 1946. Hide Caption 3 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, third from right in the front row, stands with other Cub Scouts in the 1940s. Hide Caption 4 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, spent time with the California Army National Guard after finishing law school in 1961. The man on the left, John J. Hamlyn Jr., also became a lawyer like Kennedy. Hide Caption 5 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, and Hamlyn pose for a photo after basic training. Hide Caption 6 of 38 Photos: Supreme Court Justice Anthony Kennedy After more than a decade as a lawyer, Kennedy became a judge on the US Court of Appeals in 1975. He was nominated by President Gerald Ford on the recommendation of California Gov. Ronald Reagan. Hide Caption 7 of 38 Photos: Supreme Court Justice Anthony Kennedy This courtroom photo of Kennedy was taken in 1976. Hide Caption 8 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy has breakfast with his wife, Mary, and his son Gregory in 1984. Hide Caption 9 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and his wife walk together in Sacramento, California, in 1987. Hide Caption 10 of 38 Photos: Supreme Court Justice Anthony Kennedy From 1965 to 1988, Kennedy was also a professor of constitutional law at the University of the Pacific's McGeorge School of Law. Hide Caption 11 of 38 Photos: Supreme Court Justice Anthony Kennedy In 1987, Kennedy was nominated by President Reagan to fill the Supreme Court seat vacated by Lewis Powell's retirement. The nomination came after the confirmation failures of nominees Robert Bork and Douglas Ginsburg. Hide Caption 12 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, center, talks with US Sens. Ted Kennedy, left, and Joe Biden before a confirmation hearing in Washington. The two Kennedys are not related. Hide Caption 13 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy meets with President Reagan in the Oval Office. Hide Caption 14 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is joined by his wife as he is sworn in by Chief Justice William Rehnquist on February 18, 1988. Reagan is on the right. Hide Caption 15 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, top right, appears in a formal Supreme Court portrait in April 1988. In the front row, from left, are Thurgood Marshall, William Brennan Jr., Chief Justice William Rehnquist, Byron White and and Harry Blackmun. In the back row, from left, are Antonin Scalia, John Paul Stevens, Sandra Day O'Connor and Kennedy. Hide Caption 16 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks at the McGeorge School of Law in 1991. He delivered the inaugural address in a lecture series named for the late Archie Hefner, whose portrait is behind Kennedy. Hefner was a prominent Sacramento attorney active in numerous civic and charitable groups. He died in 1988. Hide Caption 17 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is on the far right in this Supreme Court portrait from 1998. In the front row, from left, are Antonin Scalia, John Paul Stevens, Chief Justice William Rehnquist, Sandra Day O'Connor and Kennedy. In the back row, from left, are Ruth Bader Ginsburg, David Souter, Clarence Thomas and Stephen Breyer. Hide Caption 18 of 38 Photos: Supreme Court Justice Anthony Kennedy In 2004, Kennedy speaks to high school students at the Ronald Reagan Presidential Library in Simi Valley, California. Hide Caption 19 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks during a Senate subcommittee hearing in 2002. Hide Caption 20 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy discusses the court's budget requests with a House committee in April 2005. Hide Caption 21 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy participates in a panel discussion in Washington in November 2005. Hide Caption 22 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy receives an honorary degree at New York University in May 2006. Hide Caption 23 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy delivers the commencement address at New York University. Hide Caption 24 of 38 Photos: Supreme Court Justice Anthony Kennedy In February 2007, Kennedy testifies at a Senate committee hearing on judicial security and independence. Hide Caption 25 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies before a House subcommittee in March 2007. He and fellow Justice Clarence Thomas spoke about concerns with the ongoing remodeling of the court building, the reduction of paperwork due to electronic media, and the disparity of pay between federal judges and lawyers working in the private sector. Hide Caption 26 of 38 Photos: Supreme Court Justice Anthony Kennedy The Supreme Court meets with President Barack Obama and Vice President Joe Biden in September 2009. From left are Samuel Alito, Ruth Bader Ginsburg, Kennedy, John Paul Stevens, Chief Justice John Roberts, Obama, Sonia Sotomayor, Biden, Antonin Scalia, Clarence Thomas, Stephen Breyer and retired Justice David Souter. Hide Caption 27 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy leaves after a Catholic Mass in Washington in October 2009. Hide Caption 28 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy joins the President and other officials at a memorial for the victims of a shooting in Tucson, Arizona, in 2011. Hide Caption 29 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy smiles as he is introduced to faculty members at the University of Pennsylvania Law School in October 2013. Kennedy was teaching there for a week. Hide Caption 30 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is saluted by sailors as he tours the USS John C. Stennis in 2015. Hide Caption 31 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies about a Supreme Court budget request during a House subcommittee meeting in 2015. Hide Caption 32 of 38 Photos: Supreme Court Justice Anthony Kennedy President Obama greets Kennedy and other Supreme Court justices before his final State of the Union address in January 2016. Hide Caption 33 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, second from left, joins other Supreme Court justices in February 2017 during President Donald Trump's first address to a joint session of Congress. Hide Caption 34 of 38 Photos: Supreme Court Justice Anthony Kennedy As President Trump looks on, Kennedy administers the judicial oath to new Justice Neil Gorsuch in April 2017. Hide Caption 35 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and Trump walk together after Gorsuch's swearing-in ceremony. Hide Caption 36 of 38 Photos: Supreme Court Justice Anthony Kennedy Trump stands with the Supreme Court at Gorsuch's formal investiture ceremony in June 2017. From left are Elena Kagan, Samuel Alito, Ruth Bader Ginsburg, Kennedy, Chief Justice John Roberts, Trump, Gorsuch, Clarence Thomas, Stephen Breyer and Sonia Sotomayor. Hide Caption 37 of 38 Photos: Supreme Court Justice Anthony Kennedy Several members of the Supreme Court pose for a portrait before taking part in a procession to mark Harvard Law School's bicentennial in October 2017. On the top row, from left, are Kennedy, Roberts, Breyer and Gorsuch. In front of them are Kagan and retired Supreme Court Justice David Souter. Hide Caption 38 of 38

Still a conservative

However, sometimes Kennedy voted with the four conservatives on the bench. It was Kennedy who penned the majority opinion in Citizens United v. FEC -- striking down election spending limits for corporations and unions in support of individual candidates.

He's also sided with the right side of the bench on issues such as gun control and voting rights. Kennedy joined Chief Justice John Roberts' 2012 opinion, Shelby County V. Holder striking down a key provision of the Voting Rights Act.

And Kennedy sided with George W. Bush in the case that essentially decided the 2000 presidential election for the GOP candidate.

Why now?

On one side is his age -- a desire to spend more time with his grandchildren is driving any decision, and in many ways he has already established an enduring legacy on the court.

In terms of a replacement, Kennedy might take comfort in the list of 20 judges Trump has vowed to draw from when considering the next vacancy on the court.

Another consideration is that if Kennedy were to delay his retirement for a year, his replacement would face confirmation during the mid-term election year, something that could further inject politics into an already controversial process.

On the other hand, Kennedy is well aware of his role on the court and could be alarmed by how politicized the confirmation process has become. Indeed, Republicans were forced to change Senate rules to make it easier to confirm Gorsuch after Democrats objected.

Kennedy might think it would make sense to remain on the bench until the political climate simmers down -- although there's no guarantee that would ever happen.

Retiring at 81 would not be all that much different than retiring at 80 and he would get to serve longer with Gorsuch as well as take up a case on next term's docket concerning partisan gerrymandering -- an issue that might once again keep Kennedy in the spotlight.