A traffic jam, by definition, is caused by all of us. The root cause may be an accident, or construction, or the crush of mid-sized SUVs leaving a Billy Joel concert, but if you’re part of the traffic flow, you’re part of the problem.

But for some kinds of traffic jams — those that appear for no obvious reason — there’s a not-obvious solution. A single driver, armed with a rudimentary knowledge of fluid dynamics, can dissipate or prevent a miles-long jam. With the same methods, drivers working cooperatively (and aided by some here-and-now technology) could significantly and continuously reduce traffic backups on highways. And we don’t have to wait for self-driving cars to do it.

Part 1. The shocking truth behind traffic jams

In the browser-based video game Error-Prone, you and as many as 25 of your friends each control the acceleration of a cartoon car by pressing and releasing a letter on a keyboard. Crowding aside, things will end as badly as you would predict, as slight variations in speed cause a ripple effect around the circle, which is then magnified according to some very predictable laws, until these virtual commuters lie in smoking heaps. Try it yourself, controlling just one letter, and you’ll probably get the same result. Or watch a video of some scientists trying it in real life.

Error-Prone is meant to show how much safer and more efficient self-driving cars can be on the road; indeed, when all the cars are centrally controlled to travel at the same speed in the same path, they do not crash. Which, well… obviously.

But the game is also a lovely demonstration of something called a phantom traffic jam, one of those everyday occurrences in which traffic backs up for no apparent reason. As in most things, the cause is the innate imperfection of humanity. Somewhere, miles ahead of where you will eventually be stuck fuming, a driver slows slightly, causing the car behind it to slow a bit more. This wave of slowed traffic moves backwards along the roadway, simultaneously growing larger and slowing down, until the parkway becomes a parking lot. The Japanese scientists who made the video above were the first to prove the existence of these waves in traffic flow. (Though it should be noted that these waves can also be explained using digital simulations, and that there are plenty of engineers and scientists who are doing just that.)

Mathematically, these waves resemble the pressure waves created by explosions, and were modeled (and named “jamitons”) by a team supported by the National Science Foundation in 2009. Even back then, a technology was seen as the solution, providing a future in which “electronic driving assistence [sic] hardware that helps drivers (in a subtle fashion) to accelerate and decelerate more smoothly, and thus to make the occurrence of jamitons less likely.” This isn’t purely academic: Jamitons cause stop-and-start traffic, which leads to accidents, which lead to serious delays and occasionally serious injury.

Now, we are (eternally) on the cusp of autonomous vehicles, we expect that technology to save us from traffic jams. Our smart streets will carry sensor-laden cars, which will communicate with each other as well as a central network. Some advanced algorithm will optimize traffic flow, and cars will give each other the proper space to merge, anticipating slowdowns so that brake lights never illuminate. In that glorious future, cars move around like items on an assembly line, or train cars linked up together (sometimes quite literally).

But maybe we don’t need to wait. Maybe we already have the technology, and just need to put it together.

Part 2. The Traffic Ninja

“In 1996, I decided to singlehandedly change attitudes toward driving,” says William Beaty, referring to the year he started the website trafficwaves.org. It’s just “a tiny little backwater” of the thousands of pages and links that the Seattle research engineer has assembled over the years, but it has earned him a reputation as a big thinker. (He was interviewed by the BBC back in 2000.)

“I started it when I was arguing with people in Usenet newsgroups in the early days of the internet,” he recalls. “Rather than have to type out three paragraphs every time someone started telling me how people should drive, I just created an HTML page.” On that page, Beaty set forth a theory that likened phantom traffic jams, as well as those that linger after an accident is cleared, to shockwaves. But it also laid out a solution to the problem, because Beaty wasn’t going to wait around for technology to save us.

Beaty is hopeful that drivers-ed teachers will add his theories to their curriculum, but he knows his biggest obstacle is drivers themselves, and their obsessive need to be somewhere sooner.

If these phenomena were like shockwaves, he reasoned, then he could cancel out those waves by removing the medium through which they travel. Adding extra space between cars would cause jams to dissipate, and allowing people to merge early would ease bottlenecks. In general, driving more slowly will get drivers to their destinations faster. At its essence, the execution of his theory just encouraged drivers to keep a steady average speed, rather than racing ahead only to brake to a stop. He made models and videos, and practiced what he preached on his 45-minute commutes.

Aside from his own experiments, he has other real-world examples. Highway patrol cars often sweep into traffic to slow the flow for seemingly no reason at all; often, they’re providing a traffic break to stop a wave travelling back from an accident miles ahead. “Belgium and Netherlands have for about 15 years used pace-cars — a method they call ‘Blokrijden’ — whenever the whole country goes on vacation at the same time,” notes Beaty.

Beaty’s methods (which, really, boil down to “be nice, and don’t be in a hurry”) are catching on with some drivers, though it’s impossible to say just how much of an effect they’re having on your daily commute, since things like GPS navigation, increased load, and adaptive cruise control must be factored in as well. Beaty is hopeful that drivers-ed teachers will add his theories to their curriculum, but he knows his biggest obstacle is drivers themselves, and their obsessive need to be somewhere sooner.

Part 3. Today’s technology, tomorrow!

Humans are irrational beasts. Even though they may know, empirically, that allowing distance between cars and not racing forward just to slam on your brakes at a stoppage will get us and the rest of the humans behind us to destinations more quickly, the truth is we probably won’t, especially if it means that some other jerk is going to arrive at his destination 2.35 seconds before we do. But laziness may prevail, in the form of Adaptive Cruise Control. “If a few drivers use ACC in heavy congestion, their cars won't exhibit aggressive and dangerous tailgating behavior,” says Beaty. “Placing about 10% of such cars on the highway tends to make traffic jams evaporate.”

But, he adds, “I suspect that if a driver finds that their ACC prevents their own aggressive behaviour, they'll simply turn it off.” It probably doesn’t help the cause that vehicles equipped with ACC tend to be the sorts of vehicles that have luxe leather interiors and throbbing, feline engines, and are not known to have drivers versed in the art of selflessness.

But there is another class of driver that may be able to overcome a natural disposition toward jerkishness: commercial drivers. From big rigs to delivery trucks, Uber drivers to bus drivers, all have a financial incentive to keep traffic moving. Couple that with municipal and government vehicles, and it’s easy to imagine that we’ll be over the 10% threshold once those vehicles are properly equipped and the use of ACC required.

Part 4. A future without traffic jams

Artificial Intelligence is going to solve this all for us. But sci-fi movies aside, AI doesn’t assume autonomy — it’s just a (scientifically imprecise) way to denote systems that learn in ways humans can’t, typically through speedy analysis of data and sophisticated sensor arrays. It’s all algorithms, and in that world, traffic jams are all math and really fast computers.

Dr. Larry Head, of the newly formed Transportation Research Institute at the University of Arizona, sees two R&D approaches to solving the traffic jam. “The first and nearest term is connected vehicles, called V2X in Europe,” he says. “Vehicles will talk to each other (V2V) to share information 10 times per second. Each vehicle will know what the vehicles around it are doing much faster than a human could sense and react. This will allow vehicle-based applications, such as cooperative adaptive cruise control, to assist human drivers to achieve significant improvements in safety.” V2V technology is on the horizon; in the US, the National Highway Transportation Safety Board has made implementation a priority, and the Cadillac CTS will roll out with the technology in 2017.

“Connected vehicles are an important step towards reducing the 34,000 deaths due to traffic crashes in the US,” says Head. “We believe the benefits will be similar to seat belts.”

“Another application, called Speed Harmonization, will set the speed limit dynamically to smooth the flow,” says Head. “An application called Queue Warning will alert vehicles and drivers of shockwaves downstream so that drivers can be alert and react more quickly if needed.” This technology — which dampens out those pressure waves by slowing all drivers to the roadway’s average speed — has recently replaced the Low Countries’ Blokrijden, and it’s even been applied to the very same section of highway that William Beaty drove when videoing his first driving experiments, both to great success.

“The second research path is the autonomous, or self-driving vehicles,” says Head. “Currently, these two research paths are independent. I believe that this is due to the need of autonomous vehicles to operate in all environments — with or without other connected vehicles. Until the market penetration of connected vehicles is sufficient to benefit autonomous vehicles, there will likely be little integration.”

And so, the self-driving car — with complete market penetration — remains the holy grail of smooth and safe traffic flow. But Beaty is skeptical even about that. “Self-driving cars will be well-behaved and safe, and therefore get cut off by aggressive drivers.” He channels an enraged commuter:"'This will not stand!' and click, they’ve switched into manual driver mode.”

In the quest for perfect traffic flow, we humans can’t seem to say out of our own way. Maybe that’s the real reason Google built its driverless car prototype without a steering wheel.

If you would like to comment on this or anything else you have seen on BBC Autos, head over to our Facebook page or message us on Twitter. And while you're at it, join the BBC Autos community on Instagram.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Autos, Future, Earth, Culture, Capital and Travel, delivered to your inbox every Friday.