Some of the neo-Nazis and white supremacists, as well as a number of protesters within the Antifas group, openly carried assault rifles during the events in Charlottesville. The governor has said the police were outgunned at the event. Given the hate-filled speech and the ensuing physical violence, it is remarkable that no shots were fired. We were a trigger-pull away from a bloody shootout with semi-automatic weapons.

Some have suggested that the First Amendment right to congregate and to speak one’s mind — no matter how repugnant the content of the speech — conflicted with the Second Amendment right to bear arms at a political rally. The problem with this is that, while the courts have unambiguously affirmed the former, there is no Constitutional right to bear arms at a political rally.

In 1977, the Supreme Court ruled that the First Amendment guaranteed neo-Nazis the right to display their swastikas and march through a predominantly Jewish neighborhood in Skokie, Illinois, despite the offensiveness of that conduct: even outrageous speech is guaranteed to be free. The decision was clear and all-encompassing.

Yet in 2008’s District of Columbia v. Heller, a bare conservative majority led by Justice Antonin Scalia overturned the historic understanding of the meaning of the Second Amendment and held that the right to bear arms was not limited to the states’ militias. The result was that a D.C. ordinance regulating possession of handguns was struck down as being at odds with the rights granted by the Amendment; this is likely where the thinking about carrying semi-automatic rifles at political rallies comes from.

But the Heller court made clear that its ruling was limited to guns maintained in the home for purposes of self-protection. The Court explicitly said its decision did not go beyond that and that local governments could still otherwise make reasonable regulations regarding firearms. Scalia, writing for the Court, explicitly circumscribed the reach of its ruling with these words:

Nothing in our opinion should be taken to cast doubt on longstanding prohibitions on the possession of firearms by felons and the mentally ill, or laws forbidding the carrying of firearms in sensitive places such as schools and government buildings, or laws imposing conditions on the commercial sale of arms.

The result has been a number of cases in which federal circuit courts have approved legislation curtailing concealed carrying of weapons, types of ammunition and, specifically, possession of semi-automatic rifles, such as were carried by people participating in the Charlottesville rally. The Fourth Circuit Court of Appeals, which sits in Richmond, Virginia, and whose jurisdiction also includes all of Maryland and the Eastern District of North Carolina, recently upheld a Maryland statute banning assault rifles. The en banc vote was 10-4, and while it was a Maryland statute, the ruling is applicable to Virginia as well. The NRA’ s application to the Supreme Court for a writ of certiorari was rejected with but two dissents. Indeed, subsequent to Heller, the Supremes have consistently refused to review the decisions of circuit courts that have approved legislative limits on guns and ammunition, including the ban of assault weapons.

There is no constitutional conflict. Citizens have a constitutional right to attend a political rally, but the right to bring guns to that rally exists only if the state wants it that way. There was no conflict between the First and Second Amendments in the Charlottesville riot: the conflict was between common sense and the legislative will of the Commonwealth of Virginia.

It’s worth mentioning that after granting the neo-Nazis the right to rally at the base of the statue, the Charlottesville city council withdrew the permit and moved the rally to a larger venue one mile away. The ACLU, representing the neo-Nazis, went to court and urged that the anti-Semitic racists had the right to protest at the base of the Lee statue. They succeeded, and the Charlottesville riot ensued. The ACLU has asserted that it was not responsible for the violence — and it is not responsible for the murder of Heather Heyer — and that it just protected the free-speech rights of the hate groups.

This raises questions about why the organization did not consider that the presence of assault rifles by one group might inhibit the free-speech rights of the other group. The ACLU has since adopted a no-guns policy. It’s up to the Commonwealth and states like it to do the same.

Virginia has the power to regulate possession of assault rifles at political rallies. Indeed, the state has the right to outlaw them entirely. Until it and other states do so, the responsibility for shots fired at political rallies rests with state legislators and governors.

Sign up for Inside TIME. Be the first to see the new cover of TIME and get our most compelling stories delivered straight to your inbox. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.