On Labor Day, alongside stories about parades and final trips to the beach, we can expect to read the usual depressing statistics about the decline of labor unions in the United States. The problem with this coverage isn’t the facts, which are undeniable — it’s the tone of inevitability.

Today, less than 11 percent of workers, including just 7 percent in the private sector, are members of a union — a dramatic drop from the 1950s, when more than one-third of the workforce was unionized. The recent loss by the United Auto Workers at a Mississippi Nissan factory, where workers voted by a three-to-one ratio against union representation, is just the latest in a long string of defeats for the labor movement.

And this decline has a real effect on families’ financial security: Researchers have shown that nearly half of the decline in middle-class incomes is due to the shrinking rates of unionization.

Few articles probe deeply into the cause of the decline, but here’s a hint: It isn’t because unions no longer make sense in the modern economy, nor is it about American workers’ supposed skepticism toward unions (especially in the South). And it’s not more evidence that the white working class stubbornly insists on voting against its own interests.

Rather, the decline of unions is a direct result of our nation’s badly broken labor laws — specifically, the ways in which those laws and court decisions fail to acknowledge, much less protect, the constitutional rights of workers.

American courts have never been kind to labor. From the beginning of our nation’s history well into the 20th century, union organizing efforts were treated by conservative jurists as criminal conspiracies and interferences with employers’ sacrosanct contract rights. Unions spent the 19th and early 20th centuries decrying “judge-made law” and seeking, essentially, to get the government and courts out of labor disputes.

This approach did notch a few victories. The Norris-LaGuardia Act of 1932 prevented the federal courts from issuing injunctions against union picket lines (which were frequently enforced at bayonet point). Many states passed similar laws to keep their courts and local police out of the fray.

But the modern labor era began in many ways in 1935, with the National Labor Relations Act, which made it the official policy of the United States to encourage collective bargaining. The act established a federal agency, the National Labor Relations Board (NLRB), which would certify unions and punish employers that refused to deal with them. While well intended, passage of the NLRA in the form it took has had many unintended, and some extremely detrimental, consequences for organized labor.

A failure to ground labor laws in the Bill of Rights

The framers of the legislation, who at the time were leery of a conservative Supreme Court, made the pragmatic decision to root the law’s authority in the Commerce Clause, which grants Congress the right “to regulate Commerce with foreign Nations, and among the several States.” This was the subject of no small debate at the time.

Many advocates believed that labor’s rights should be based on First Amendment rights to free speech and free assembly; others thought that the Fifth Amendment right to “due process” — interpreted as a broad protection of citizens’ rights — made sense. Leaders of the American Federation of Labor had been arguing for a half-century that the 13th Amendment, which prohibited both slavery and, crucially, “involuntary servitude,” was the appropriate constitutional basis for labor rights. But the act’s framers were convinced that the Lochner-era Court that made laissez faire economics a virtual state religion would never go for more lofty human rights justifications.

Practically speaking, the court was motivated to accept the NLRA more by the wave of sit-down strikes roiling the country in 1937 than by legal theories. But conceiving of unionization as a matter of business regulation led to a very narrow interpretation of the act. Before the end of World War II, the court took away legal protections for sit-down strikers, denied striking workers the right to return to his job, and granted employers a First Amendment right to run vicious union-busting campaigns.

Much of this anti-union thrust was endorsed and enhanced by the Taft-Hartley Act of 1947, passed by a Republican Congress over President Truman’s veto. The law declared it illegal for union members to boycott or picket a “secondary” employer — that is, a company that they do not work directly for, but who has significant or even essential business dealings with their employer.

Most people think of the Taft-Hartley Act for its “right-to-work” provisions. The Act permitted states to allow workers to decline to join a union, or pay fees, even if they benefitted from union-negotiated contracts.

But it’s the prohibition against solidarity activism, the heart and soul of the labor movement, that’s been most devastating. A cable provider that gets into a fight with a television channel over revenue sharing can black that channel out. But if the workers at your local unionized grocery store want to keep Oreo cookies created at a scab factory off the shelves, they face steep fines, effectively banning the practice.

Restrictions on labor accelerated in 1949, when 10 unionized technicians at the Jefferson Standard Broadcasting Company, in Charlotte, North Carolina, were fired for distributing handbills criticizing their employer. Workers quickly filed a complaint with the NLRB arguing that they were participating in a legally protected activity, but the NLRB ruled that the technicians’ actions were not protected because they were not explicitly connected to the union contract campaign — they involved more general complaints.

Courts recognized “disloyalty” as lawful grounds for firing a worker

Upon appeal, the US Supreme Court issued Labor Board v. Electrical Workers, one of its most aggressively anti-labor decisions. The majority wrote, among other things: “There is no more elemental cause for discharge of an employee than disloyalty to his employer.”

Interpretation of the Jefferson Standard decision has led to decades of harsh and contradictory court decisions that chill the rights of workers to speak out. This past July, the Eighth Circuit ruled that six workers at a Jimmy Johns sandwich shop could be fired because they circulated memes suggesting that customers might be eating food made by sick workers. They were protesting the franchise owner’s refusal to provide paid sick days (ill workers also had find their own replacements).

The “poster attack” was “so disloyal” that it was not protected under the law, the court found. Unsurprisingly, the judges had little to say about the employer’s reciprocal responsibility to be loyal enough to its workers to let them take a day off when they get sick.

And consider the recent vote at the Mississippi Nissan plant, in which efforts to form a union were turned back. In order to determine whether workers had a legal right to form a union, the NLRB held an election with rules set forth by Congress and the courts. Among them was the requirement that only one side — the party that opposes unionization — can force employees to attend mandatory presentations of their arguments as a condition of employment. The union, according to the federal government, has no equivalent right of access or response.

Employers’ speech to captive audiences of employees is protected

As expected, managers at Nissan forced employees to attend multiple one-on-one and small group “captive audience” meetings, in which they threatened that the plant could be relocated, or shuttered, if workers voted for a union. That is nothing new. Across the country, employers make fake “economic predictions” to threaten workers’ jobs in two-thirds of all union elections, according to Cornell professor Kate Bronfenbrenner.

But restrictions on workers’ speech don’t end at rigged elections; the government also censors what unions can do at the bargaining table. Federal law requires employers to negotiate “in good faith,” but only on a tightly circumscribed set of issues — basically just wages, hours and some working conditions. A decision to relocate a factory, as Nissan managers threatened, is not an issue unions can negotiate.

This is what makes threats of plant closures such effective anti-union tactics. Workers want unions that have the ability to veto or amend management’s decisions to downsize, subcontract, automate or shift work overseas. But in the American system, unlike in much of Europe, such matters are the prerogative of management alone. When the law restricts unions from being what workers want them to be, of course there are going to be low levels of union representation.

Simply put, employees are hampered by rules that would never be applied to corporations, or to any other form of political activism. That the government can dictate to workers what they can or cannot write on a flyer, where and how they can march, and what they can and cannot boycott, is not just unfair — it ought to be illegal.

In March 2016, the Supreme Court considered the issue of speech in bargaining in Friedrichs v. CTA, a case that deadlocked 4-4 (without Justice Scalia’s vote); in that case, conservatives argued that public-sector unions violate workers’ speech rights when they collect mandatory union fees. The goal was to bankrupt the last of the big unions. The parties are suing for a do-over in the pending Janus v. AFSCME.

Union advocates should wrest the free-speech issue from conservatives, because we have the far better case: Every interaction between the government and a union is a matter of political speech. Unions and their allies should challenge unequal restrictions on free speech and assembly as the violations of workers’ constitutional rights that they are.

As economists scratch their heads this Labor Day about why the longest economic expansion on record is not translating to increases in most Americans’ standard of living, let’s take a good, hard look at precisely how the power of unions has been curbed. Just as laws permitting collective bargaining helped build the American middle class, a vigorous First Amendment defense of workers’ constitutional rights can help rebuild it today.

Shaun Richman is a leading expert on employment issues and author of the new Century Foundation report, “Labor’s Bill of Rights.”

The Big Idea is Vox’s home for smart discussion of the most important issues and ideas in politics, science, and culture — typically by outside contributors. If you have an idea for a piece, pitch us at thebigidea@vox.com.