Oct 10 (Reuters) - We heard Republican presidential nominee Donald Trump call Democratic presidential nominee Hillary Clinton, to her face, "a liar." We heard Clinton say, flatly, "Donald Trump is not fit to be president or commander in chief."

The two candidates who squared off in the debate Sunday night were not just challenging each other's views. They were attacking each other's fitness to serve. Trump even promised to have Clinton prosecuted if he is elected, saying, "She belongs in jail."

How did Americans end up in this dark place? What happened to the U.S. system for vetting presidential candidates?

The answer is, we don't have a system for vetting presidential candidates.

It used to be done by party bosses. They were supposed to take the candidates into a "smoke-filled room'' and ask, "Is there anything in your past that would keep you from getting elected?" But the American public took that power away from the party bosses decades ago, when nominations were turned over to primary voters.

So who vets the candidates now? Answer: the press. The problem is that most voters don't trust the press.

There was plenty of evidence of Trump's unsuitability for the White House, starting the day he announced. He attacked Mexican immigrants as criminals and rapists. His supporters didn't really care. As a political outsider, Trump carried their angry message when no one else did.

The most damaging revelations came too late. Like the tax return documents leaked to the New York Times that showed how Trump could get away with not paying income taxes for almost two decades. And his degrading remarks about women leaked to the Washington Post on Friday. Imagine taped banter with a Hollywood entertainment show host bringing down a presidential candidate.

Trump's response in the debate was to go on the offensive and attack Clinton, and her husband, relentlessly. Trump set the tone: harsh, divisive and pessimistic. He accused Clinton of having "hate in her heart" because of her criticism of his supporters ("deplorables").

Her supporters might have hoped for a response in kind, but Clinton refused to go on the defensive. For the most part she just dismissed his accusations as factually inaccurate.

So far, the revelations about Clinton from WikiLeaks have been damaging but not disqualifying. They confirm what people already know and dislike about her - that she is calculating. She behaves like a typical politician. That is not news.

Moreover, WikiLeaks is hardly a neutral source. Its founder has said he intends to leak information that will bring Clinton down.

Trump's response to the outcry over his degrading remarks about women has been to try to shift public attention to Bill Clinton's behavior when he was president. That may backfire. It will remind voters of how Clinton embarrassed the country. Do people really want to go through that again?

At one point in the debate, Trump blamed President Barack Obama and Hillary Clinton for creating "a divided nation." His own performance did not raise the prospect that electing him would do anything to heal the division. He was the very embodiment of masculine aggression. He constantly invaded Clinton's allotted time and space.

Trump complained bitterly that in Clinton's 30 years of public service, she failed to reform the tax system, end terrorism, bring back jobs and do many things she says she wants to do. "Why haven't you done them?" he demanded to know.

Clinton's response was that the U.S. system of government doesn't give anyone, even the president, the power to solve complex problems at a stroke. She's right, but Trump refuses to accept that argument.

He promises to be the all-powerful leader who will, in fact, do all those things. The point is, his followers believe him.

What we are likely to see now are growing numbers of undecided voters who will never vote for Clinton but can't bring themselves to support Trump. We may see an increase in support for Libertarian Party candidate Gary Johnson. Some former Trump supporters may stay home. Many will cast write-in ballots.

Republican Party leaders have no power to reverse the Republican convention's nomination of Trump. Some of them are urging Trump to resign from the ticket. But defiance is Trump's signature attitude. He made his political career by defying GOP leaders. He's not going to give in to them now. He'll just bring them down with him.

Even if Trump were to withdraw, it's too late to change the ballots in many states. Early voting has already started in many of them. People who have already voted are stuck. Once you cast your ballot, you can't get it back.

The person who is really on the spot is Trump's running mate, Indiana Governor Mike Pence. If Trump loses, Pence will probably run for president in 2020. If Pence feels he cannot in good conscience do his job, which is to promote Trump, he may want to think about resigning from the ticket

That would enrage Trump supporters. By 45 percent to 39 percent, Republicans polled by Politico say they want Trump to continue his campaign. Yet if Trump loses next month and the Republican Party suffers a devastating defeat, it could poison Pence's prospects for years to come. Pence will be Trump's man, and Republicans may not want to be reminded of the 2016 debacle.

What a terrible irony it would be if Trump stays in the race and Pence resigns.

---

By Bill Schneider

Bill Schneider is a professor at the Schar School of Policy and Government at George Mason University. He is a visiting professor in the Communication Studies Department at the University of California - Los Angeles. The opinions expressed here are his own.