There are still 42 days and four debates left before the presidential election and many signs point to a close outcome, but recent polling both nationally and in key battleground states like Ohio has conservatives concerned about the impact President Obama could have on the judiciary in a second term.

The New Yorker's Jeffrey Toobin joins Morning Joe to discuss President Obama's relationship with the Supreme Court, Chief Justice John Roberts and his ruling on the Affordable Care Act, and the relationships the justices have with one another.

So far Obama has appointed 159 judges to the federal bench, including his two Supreme Court selections, Sonia Sotomayor and Elena Kagan.

With Sotomayor and Kagan, Obama simply replaced two members of the liberal bloc on the court (John Paul Stevens and David Souter) with two younger liberals. But most of the Obama-appointed judges – 127 of them – are trial judges who exert less influence on the broad direction of the law than do appeals court judges or Supreme Court justices.

Recommended: Obama denounces violence in Middle East, calls for tolerance and democracy

Both on the Supreme Court – where “swing vote” Justice Anthony Kennedy and conservative Justice Antonin Scalia are age 76 – and on the courts of appeal, where there are now 14 vacancies, Obama would be able to nudge the courts in a progressive direction if he wins a second term.

Slideshow: On the campaign trail Reuters, Getty Images In the final push in the 2012 presidential election, candidates Mitt Romney and Barack Obama make their last appeals to voters. Launch slideshow

The high court begins its 2012-13 term on Monday and has scheduled arguments on the use of race in undergraduate admissions decisions at the University of Texas and on whether multinational firms can be sued for their alleged role in human rights abuses that occur outside the United States. It’s also likely the court will take up challenges to the 1996 Defense of Marriage Act, which defines marriage as solely the union of one man and one woman.

“It would be a bad mistake for conservatives to get gloomy and defeatist. This election is very winnable,” said Ed Whelan, president of the Ethics and Public Policy Center and a former Scalia law clerk who served in the Justice Department under President George W. Bush.

But Whelan added, “There are lots of reasons why anyone concerned about America’s future should rue the prospect of President Obama’s re-election. What President Obama would do to the Supreme Court is high on the list.”

Alex Wong / Getty Images

“The potential impact of the next president on the Supreme Court is immense,” said Carrie Severino, the chief counsel and policy director at the Judicial Crisis Network, a right-of-center advocacy group. “There could easily be three nominations during the next term. … Most people expect there to be at least one vacancy.”

She said if Obama were re-elected and got three more high court nominations, “that would put him in the position of having nominated the majority of the justices on the Supreme Court. That’s an incredible influence over the way the court shapes American society.”

She said there’s now a 5-4 split on issues that have been before or are coming before the court, such as whether schools can use applicants’ and students’ race or ethnicity in admissions or in assigning students to specific public schools.

From racial preferences to gun owners’ rights to immigration to same-sex marriage, “you name it – there are so many issues where the outcome turns on one vote at the Supreme Court and the president could easily be shaping that next vote,” Severino said. And those concerns extend even to a possible Romney presidency.

Curt Levey, president of the Committee for Justice, a conservative group that tracks judicial nominations, said Chief Justice John Roberts’ decision to join the four liberal-leaning justices in upholding Obama’s Affordable Care Act “has made conservatives think somewhat differently” about judicial nominees. “There’s a lot of sober thinking among conservatives that it is not just enough to appoint somebody who we know to have the right philosophy – you have to appoint somebody who has shown, either as a judge or perhaps in some other setting, that they will stick with that philosophy even when there’s political pressure to do otherwise.”

Of course, it’s not just those on the right who are warily watching the election outcome and its impact on the judiciary. Doug Kendall, president of the Constitutional Accountability Center, a progressive think tank and advocacy group, said, “I think a Romney presidency has a far greater potential to shift the court to the right than a second Obama term would have the potential to shift it to the left.”

Kendall said the most likely high court retiree is Justice Ruth Bader Ginsburg, 79, appointed to the court by President Bill Clinton in 1993. “If President Romney nominates her successor, it will have a huge impact on the ideological balance of the court. If President Obama appoints Justice Ginsburg’s successor it will simply continue the current ideological balance and will not move the court to the left at all.”

As for appeals court and trial court judges, Severino contends that in his first term Obama has not put great emphasis on appointing them. “Many of us were surprised – and relieved – that he didn’t do that in his first term. It will be interesting to see whether his focus shifts” to judicial nominations if he wins a second term, Severino said.

After eight years of Democrats delaying and blocking Bush judicial nominees such as Miguel Estrada and William Pryor, Levey said Senate Republicans decided when Obama became president “they weren’t just going to roll over” on his judicial nominees. But “the bigger reason Obama has had a record low number of confirmations, at least taken as percentage of the judiciary, is that Obama hasn’t made it a priority.”

For example, there are now 14 appeals court vacancies but only 7 Obama nominees for those vacancies.

Senate Republicans have succeeded in blocking a couple of Obama’s appeals court picks: Goodwin Liu in California and Caitlin Halligan in the District of Columbia. And they have slowed the pace of confirmation for those whom Obama has nominated.

A report issued last week by the Congressional Research Service (CRS) looked at waiting times for uncontroversial appeals court nominees, defined as nominees whom the Judiciary Committee approved by voice vote or by a unanimous roll call vote, and who were ultimately confirmed by the full Senate by voice vote, or with five or fewer ‘nay’ votes.

The CRS report found that the average number of days from nomination by the president to Senate confirmation went from 64 days during Ronald Reagan’s presidency to 201 days in George W. Bush’s presidency to 227 days during Obama’s presidency.

“My expectation is that President Obama will have to take that battle on more directly in a second term,” Kendall said.

Noting that Republicans had prevented confirmation of Obama’s two nominees to the powerful U.S. Court of Appeals for the District of Columbia Circuit, which hears most regulatory agency appeals, Kendall said, “He needs to fill those seats” and needs to “exercise his constitutional authority as president to fill vacancies on that incredibly important court.”