Doctors can look inside you with magnetic fields and pill-mounted cameras. They use robots to perform surgeries and lasers to fix your vision. And yet, in so many other ways, the field of medicine seems stuck in the past. Doctors carry pagers. They make you call them to schedule an appointment. They force new patients—and even old patients with new needs or new insurance—to fill out a bunch of paperwork … on actual paper. Doctors are Luddites in white coats.

This is what makes medical hackathons so much fun to read about. Originally the brainchild of Silicon Valley, a hackathon brings people from different fields—clinicians, engineers, entrepreneurs, software developers—together in one place. At the beginning of a medical hackathon, speakers are invited to make 60-second pitches about a problem facing the health system. “We want people to focus on a problem, rather than jumping to the technology,” says Ally Yost, a co-leader of the MIT group Hacking Medicine and a Ph.D. candidate in mechanical engineering. “Many people talk about a problem they’ve observed in their families or something a loved one experienced.”

The participants then self-divide into mixed groups based on problems they’re interested in solving. There is usually an element of competition: Teams with the best ideas take home a cash prize and an open door to people who can help them grow the idea into a real business.

A hackathon in no way resembles a traditional medical conference. In a recent story on MIT’s Grand Hackathon, held last month, the Wall Street Journal described youngsters wearing hoodies and backpacks, drinking from juice boxes. The nattily dressed men and women at medical conferences don’t drink from juice boxes.

The events have become media darlings because they hint at a shifting medical culture. Plodding, multiyear, multisite trials one day yielding to the Steve Jobs-ian ideal—let your mind run free, and you can solve any problem. It’s important to recognize, however, what these get-togethers are good for, and what they’re not good for.

Hackathons are excellent at launching business and solving knotty little problems. A good example is Smart Scheduling, a hackathon-born company that uses an algorithm to predict which patients are likely to show up late for doctor’s appointments and which ones won’t show up at all. No-shows cost the health care system $150 billion annually and make going to the doctor a pain for the rest of us because they force offices to overbook. Giving doctors software to mitigate this problem is smart and, if broadly adopted, could save the system a lot of money.

The really big challenges in health care are likely too complicated for a hackathon, though. Take the problem of nonadherence to medication regimens. Many people have to take an ever-changing combination of pills at different times of the day and on different days of the week. Patients who fall off the medication wagon inflict between $100 billion and $300 billion in avoidable expenses on the U.S. health care system every year. PillPack, which was also conceived in a hackathon, is a clever attempt to solve this problem. For $20 per month, PillPack takes over management of a patient’s medication regimen, sending them little baggies of pills organized by and labeled with the time the drugs should be taken.

PillPack is a good service. But is it going to solve the global problem of nonadherence to medication? No. People fail to take their medicines for a wide variety of reasons, and PillPack only addresses one of them. One-fifth of patients who don’t adhere to their medication schedules are afraid of side effects; 17 percent can’t afford to buy their medications; 14 percent don’t think the medications will work.

Forgetfulness is, indeed, the largest reason for nonadherence, accounting for 24 percent of failures, but it’s not clear what proportion of those failures PillPack would solve. The added cost of the service will deter many people who can barely afford their meds already. (Even relatively well-off people forgo their medications for cost reasons.) Some of the forgetters haven’t confused their schedules—they’ve truly forgotten about the medications altogether. PillPack can’t prevent that. Finally, PillPack is probably only going to reach people who are already reasonably good at taking their meds. People with psychological problems, the homeless, those with no social support structure—the people most likely to neglect their meds—aren’t going to visit PillPack’s website.

There’s an element of hubris to medical hackathons that can’t be ignored. Medical experts around the world have been trying to solve most of these kinds of problems for years. There are countless ideas to get people to take their medicines, ranging from no-tech to cutting edge. Directly observed therapy requires patients to appear at a hospital so a nurse can watch them take the medications. It has proven its efficacy in clinical trials all over the world. There are electronic pillboxes that beep, flash, and send text messages to both the patient and her doctor when a dose is missed. The boxes are currently undergoing trials as well, and it will be years before we know how well they work.

I’m not picking on PillPack. It’s an excellent idea and seems much better than going to a pharmacy. The packaging is sleek and simple. The introductory videos are great and have fun banjo music. If you have a complex schedule of pills, you should seriously consider the service. I definitely would.

Medical breakthroughs, though, require more than an excellent idea. In most cases, the problem is not ideas—ideas are everywhere. The problem is proof. Proving a medical intervention works is far more complicated than proving that a piece of software code works or that a business model can turn a profit. It usually takes years, millions of dollars, and many, many people willing to offer themselves as guinea pigs. Getting together for a weekend of concentrated, cross-disciplinary creativity is unlikely to shorten that process.

If hackathons can nudge medical culture toward greater openness to change, that’s probably a good thing. It’s impossible to justify the full extent of neo-Luddism that infects medicine. The fact that only one-third of doctors communicate with patients electronically, and fewer than one-quarter provide online scheduling and access to test results, is inexcusable. Paper forms should be gone. It’s absurd that the Washington Post ran a story entitled “Doctors Slow to Adopt E-Records for Patients” all the way back in 2006, and seven years later there are brand-new stories with nearly the same headlines. In other cases, though, doctors move slowly for a reason. The best physicians don’t make a move until there is ample reason to believe an innovation is an improvement. Hackathons are a great way to generate a spark. But in medicine, it’s a long way from spark to fire.