Pablo Garcia’s hospital ward doubled as UCSF’s pediatric research center, where patients on clinical trials frequently receive unusual medications. Brooke Levitt, still a bit baffled by the number of Septra pills, now wondered whether that explained the peculiar dose — perhaps Pablo was on some sort of research protocol. She thought about asking her only colleague on the floor, the charge nurse, but she knew that the charge nurse was busy seeing her own patients and delivering their medications.

Of course, Levitt now beats herself up for not tapping her colleague on the shoulder. But it’s not that surprising that she failed to do so. Studies have found that one important cause of errors is interruptions, so clinicians at UCSF and elsewhere have been counseled to avoid them, particularly when their colleagues are performing critical and exacting tasks like giving children potentially dangerous medications.

In some hospitals, nurses now mix or collect their medications wearing vests

that say “Don’t Interrupt Me,” or stand inside a “Do Not Interrupt” zone

marked off with red tape.

But there was probably something else, something more subtle and cultural, at play. Today, many healthcare organizations study the Toyota Production System, which is widely admired as a model for safe and defect-free manufacturing. One element of the TPS is known as “Stop the Line.” On Toyota’s busy assembly line, it is every frontline worker’s right — responsibility, really — to stop the line if he thinks something may be amiss. The assembly line worker does this by pulling a red rope that runs alongside the entire line.

When a Toyota worker pulls the cord for a missing bolt or a misaligned part, a senior manager scrambles to determine what might be wrong and how to fix it. Whether on the floor of an automobile manufacturing plant or a pediatrics ward, the central question in safety is whether a worker will “stop the line” — not just when she’s sure something is wrong but, more important, when she’s not sure it’s right.

Safe organizations actively nurture a culture in which the answer to that second question is always yes — even for junior employees who are working in unfamiliar surroundings and unsure of their own skills. Seen in this light, Levitt’s decision to talk herself out of her Spidey sense about the Septra dose represents one nurse’s failure in only the narrowest of ways. More disturbing, it points to a failure of organizational culture.

Levitt’s description of her mindset offers evidence of problems in this culture, problems that are far from unique to UCSF. “When I was counting all the pills and seeing them fill half a cup, my first thought was, that’s a lot of pills. Obviously it didn’t alarm me enough to call someone. But it was more than just a nagging sensation.”

Why didn’t she heed it? Another factor was her rush to complete her tasks on an unfamiliar floor. The computer helps create the time pressure: a little pop-up flag on the Epic screen lets nurses know when a medication is more than 30 minutes overdue, an annoying electronic finger poke that might make sense for medications that are ultra-time-sensitive, but not for Septra pills. She also didn’t want to bother the busy charge nurse, and she “didn’t want to sound dumb.”

As is so often the case with medical mistakes, the human inclination to say, “It must be right” can be powerful, especially for someone so low in the organizational hierarchy, for whom a decision to stop the line feels risky.

Finally, the decision to stop the line sometimes hinges on how much effort it takes to resolve one’s uncertainty. Remember that Levitt was usually assigned to the pediatric ICU, where nurses, doctors and pharmacists still generally work side by side, hovering over desperately ill babies. “I’m so used to just asking a resident on the spot, ‘Is this the dose you really want?’” she said. But on the wards, where the pace is slower and the children are not as critically ill, the doctors have all but disappeared. They are now off in their electronic silos, working away on their computers, no longer around to answer a “Hey, is this right?” question, the kind of question that is often all that stands between a patient and a terrible mistake.

But there’s another major reason Levitt didn’t call anyone for help. She trusted something she believed was even more infallible than any of her colleagues: the hospital’s computerized bar-coding system. The system — not unlike the one used in supermarkets and stores everywhere — allows a nurse to scan a medication before she gives it to be sure it’s the right medicine, at the right dose, for the right patient.

In a seminal 1983 article, Lisanne Bainbridge, a psychologist at University College London, described what she called the “irony of automation.” “The more advanced a control system is,” she wrote, “so the more crucial may be the contribution of the human operator.” In a famous 1995 case, the cruise ship Royal Majesty ran aground off the coast of Nantucket Island after a GPS-based navigation system failed due to a frayed electrical connection. The crew members trusted their automated system so much that they ignored a half-dozen visual clues during the more than 30 hours that preceded the ship’s grounding, when the Royal Majesty was 17 miles off course.

In a dramatic study illustrating the hazards of overreliance on automation, Kathleen Mosier, an industrial and organizational psychologist at San Francisco State University, observed experienced commercial pilots in a flight simulator. The pilots were confronted with a warning light that pointed to an engine fire, although several other indicators signified that this warning was exceedingly likely to be a false alarm. All 21 of the pilots who saw the warning decided to shut down the intact engine, a dangerous move. In subsequent interviews, two-thirds of these pilots who saw the engine fire warning described seeing at least one other indicator on their display that confirmed the fire. In fact, there had been no such additional warning. Mosier called this phenomenon “phantom memory.”

Computer engineers and psychologists have worked hard to understand and manage the thorny problem of automation complacency. Even aviation, which has paid so much attention to thoughtful cockpit automation, is rethinking its approach after several high-profile accidents, most notably the crash of Air France 447 off the coast of Brazil in 2009, that reflect problems at the machine–pilot interface. In that tragedy, a failure of the plane’s speed sensors threw off many of the Airbus A330’s automated cockpit systems, and a junior pilot found himself flying a plane that he was, in essence, unfamiliar with. His incorrect response to the plane’s stall — pulling the nose up when he should have pointed it down to regain airspeed — ultimately doomed the 228 people on board. Two major thrusts of aviation’s new approach are to train pilots to fly the plane even when the automation fails, and to prompt them to switch off the autopilot at regular intervals to ensure that they remain engaged and alert.

But the enemies are more than just human skill loss and complacency. It really is a matter of trust: humans have a bias toward trusting the computers, often more than they trust other humans, including themselves.

This bias grows over time as the computers demonstrate their value and their accuracy (in other words, their trustworthiness), as they usually do. Today’s computers, with all their humanlike characteristics such as speech and the ability to answer questions or to anticipate our needs (think about how Google finishes your thoughts while you’re typing in a search query), engender even more trust, sometimes beyond what they deserve.

An increasing focus of human factors engineers and psychologists has been on building machines that are transparent about how trustworthy their results are. In its 2011 defeat of the reigning Jeopardy champions, the I.B.M. computer Watson signaled its degree of certainty with its answers. Before he passed away last month, George Mason University psychologist Raja Parasuraman was working on a type of computer Trust-o-Meter, in which the machine might have a green, yellow or red light, depending on how trustworthy it thinks its result is.

But that might not have bailed out Levitt, since the bar-coding machine probably felt pretty darn sure that it was prompting her to deliver the correct dose: 38½ pills. So we are left struggling with how to train people to trust when they should, but to heed Reagan’s admonition to “trust but verify” when circumstances dictate. The FAA is now pushing airlines to build scenarios into their simulator training that promote the development of “appropriately calibrated trust.” Medicine clearly needs to tackle its version of the same problem.