It’s nothing new to accuse your political opponents of allowing foreign money to be used in a U.S. election, as President Obama did this week with the U.S. Chamber of Commerce. In 2008, Republicans demanded an audit of the Obama campaign, because federal law did not require the campaign to disclose contributions of less than $200, and Republicans said some of that money could be coming from foreign individuals or governments. Before that, Senate Republicans investigated Bill Clinton over possible Chinese government money influencing the 1996 election.

We owe the recurring controversy to the inadequacy of our campaign-finance disclosure laws, which don’t allow for random audits of political committees to make sure the foreign money is kept out or segregated for nonpolitical purposes. And the Supreme Court reignited this issue in its divisive Citizens United opinion last term by specifically leaving open the question whether the ban on foreign spending in U.S. elections violates the First Amendment. As Democrats broadcast their breathless allegations that Republicans and the Chamber of Commerce are taking “secret foreign money to influence” the midterm elections, the real problem is why the logic of Citizens United can’t justify a ban.

In Citizens United, the Supreme Court held that U.S. corporations have the right to spend unlimited sums in U.S. elections. That ruling, as well as a combination of lower court cases, Federal Election Commission action, and byzantine tax rules has led to record spending on this election and the least amount of disclosure of its sources since Watergate. The court opened the money spigot with a kind of First Amendment absolutism: The five-justice conservative majority repeatedly stated that when it comes to political speech, the identity of the speaker does not matter, that more speech is always better, that the public cares only about the message, not the messenger, that even millions of dollars in “independent” spending cannot corrupt a candidate, and that the public won’t lose confidence in the electoral process because of these independent gobs of money.

In response to these arguments, Justice Stevens in his dissent raised the specter of World War II’s “Tokyo Rose,” the Japanese radio propagandist who aimed her message at U.S. troops. Stevens said that the logic of the Court’s ruling would allow for foreign spending on American elections, also undermining American interests. Rather than countering with an argument as to why foreign spending might be different, the majority just punted: “We need not reach the question whether the Government has a compelling interest in preventing foreign individuals or associations from influencing our Nation’s political process.” The court said it would decide another time whether the government could demonstrate a “compelling interest in limiting foreign influence over our political process.”

But Justice Stevens was absolutely right. If it is true that the identity of the speaker does not matter, that more speech is always better, and that independent spending from whatever source cannot cause corruption or undermine voter confidence, then why stop foreign spending on elections?

There are of course good reasons to limit foreign money in the electoral process—it’s just that none of them are compatible with the Supreme Court’s First Amendment absolutism. Unlike American citizens, foreign individuals, governments, and associations are unlikely to have allegiance to the United States. A foreign entity may even have military or economic interests adverse to the United States. Foreign individuals or groups could support candidates to curry favor, or at the least, to secure preferential access to elected officials. Even putting aside the possibility of corruption and the sale of access, would we really want the close and intense battle for a majority in the House of Representatives to be influenced by money from a foreign government, corporation, or millionaire? The answer is obviously no, whether you sit on the Democratic or Republican side of the aisle. And foreign spending on U.S. elections could undermine public confidence in the electoral process.

In fact, the case against foreign spending on elections is so clear that I have a hard time believing that the Supreme Court would actually strike down the ban on foreign spending. The justices would find some way to distinguish the foreign spending from U.S. corporate spending, even if the solution only makes campaign finance law even more incoherent. At bottom, such a ruling would have to recognize, at least implicitly, that when it comes to our elections, more speech is not always better. The identity of the speaker does matter. And that the public cares not only about what is said in an election but about who is saying it. These are all defensible principles. They also are nowhere to be found in Citizens United.

Like Slate on Facebook. Follow us on Twitter.