T. C. Worley/NYT/Redux/eyevine

Before making the first incision, confirm the patient's identity. Mark the surgical site. Ask about allergies. Discuss any anticipated blood loss. Introduce yourself by name. These are some of the 19 tasks on the World Health Organization (WHO) Surgical Safety Checklist, a simple list of actions to be completed before an operation in order to cut errors and save lives.

In 2007 and 2008, surgical staff at eight hospitals around the world tested the checklist in a pilot study1. The results were remarkable. Complications such as infections after surgery fell by more than one-third, and death rates dropped by almost half. The WHO recommended that all hospitals adopt its checklist or something similar, and many did. The UK National Health Service (NHS) immediately required all of its treatment centres to put the checklist into daily practice; by 2012, nearly 2,000 institutions worldwide had tried it. The idea of checklists as a simple and cheap way to save lives has taken hold throughout the clinical community. It has some dynamic champions, including Atul Gawande, a surgeon at Brigham and Women's Hospital in Boston, Massachusetts, who led the pilot study and has spread the word through talks, magazine articles and a best-selling book, The Checklist Manifesto (Metropolitan, 2009).

But this success story is beginning to look more complicated: some hospitals have been unable to replicate the impressive results of initial trials. An analysis of more than 200,000 procedures at 101 hospitals in Ontario, Canada, for example, found no significant reductions in complications or deaths after surgical-safety checklists were introduced2. “We see this all the time,” says David Urbach, a surgeon at the University of Toronto who led the Ontario analysis. “A lot of studies that should be a slam dunk don't seem to work in practice.” The stakes are high, because poor use of checklists means that people may be dying unnecessarily.

A cadre of researchers is working to make sense of the discrepancies. They are finding a variety of factors that can influence a checklist's success or failure, ranging from the attitudes of staff to the ways that administrators introduce the tool. The research is part of the growing field of implementation science, which examines why some innovations that work wonderfully in experimental trials tend to fall flat in the real world. The results could help to improve the introduction of other evidence-based programmes, in medicine and beyond.

“We need to learn the lessons from programmes and interventions like the checklist so we don't make the same mistakes again,” says Nick Sevdalis, an implementation scientist at King's College London.

Replication frustration

One of the first to demonstrate the potential of checklists in health care was Peter Pronovost, an anaesthesiologist and critical-care physician at Johns Hopkins University School of Medicine in Baltimore, Maryland. In 2001, Pronovost introduced a short checklist for health-care workers who insert central venous catheters, or central lines, which are often used in an intensive care unit (ICU) to test blood or administer drugs. The trial showed that asking practitioners to confirm that they had performed certain simple actions, such as washing their hands and sterilizing the insertion site, contributed to a dramatic reduction in the risk of life-threatening infections3. The list got a larger test in a now-famous trial4 known as the Keystone ICU project, launched in Michigan in October 2003. Within 18 months, the rate of catheter-related bloodstream infections fell by 66%.

Checklists were not completely new to medicine, but Pronovost's work attracted attention because it suggested that they could save lives. Gawande penned an inspiring feature in The New Yorker5, asking: “If something so simple can transform intensive care, what else can it do?” Checklists began to proliferate. Now there are checklists for procedures involving anaesthesia, mechanical ventilation, childbirth and swine flu. Many studies have generated promising results, showing that the lists improve patient outcomes in hospitals from Norway to Iran.

But there have also been some failures. This January, less than a year after the report from Ontario, a different team of scientists reported6 that a surgical checklist modelled on Pronovost's list did not improve outcomes at Michigan hospitals. And although the central-line checklist for ICUs has provided lasting benefits in Michigan, a British initiative called Matching Michigan, which aimed to replicate the Keystone programme, seemed to make no difference to infection rates7.

Some experts suspect that the failure to replicate could be a matter of how the initial trials or the follow-up studies were designed. Gawande's pilot study of the WHO surgical checklist, for example, was not randomized and had no control group. Instead, it compared complication and death rates before and after the checklist was introduced. Critics say that this makes it difficult to determine what other factors might have influenced outcomes.

Gawande acknowledges the limitation, which was due to cost restrictions, but he points out that many subsequent trials, including ones that were randomized, have also demonstrated large reductions in complications and mortality following the introduction of the checklist. The list works, he says — as long as it is implemented well. “It turns out to be much more complex that just having the checklist in hand.”

Ticking boxes

Implementation scientists are trying to make sense of that complexity. After the NHS mandated the WHO checklist, researchers at Imperial College London launched a project to monitor the tool's use, and found that staff were often not using it as they should. In a review of nearly 7,000 surgical procedures performed at 5 NHS hospitals, they found that the checklist was used in 97% of cases, but was completed only 62% of the time8. When the researchers watched a smaller number of procedures in person, they found that practitioners often failed to give the checks their full attention, and read only two-thirds of the items out loud9. In slightly more than 40% of cases, at least one team member was absent during the checks; 10% of the time, the lead surgeon was missing.

Source: Ref. 10

Going through all the steps in the list really mattered, the research showed. The more of the checklist that teams completed, the lower the complication rates. Several other studies have also revealed that higher compliance with the checklist is associated with better outcomes.

“If it's used well, if it's used in the original spirit and intention with which it was designed, I think it has real potential,” says Sevdalis, who was part of the Imperial College research team. “If it's used for people to tick the box and say, 'Oh yes, we've done it,' but without really thinking about the patient, without really informing their team members about aspects of the procedure that are relevant to them, I don't think the checklist will make any difference.”

To find out why checklists were not being used properly, Sevdalis and his colleagues interviewed more than 100 members of operating-theatre staff at 10 NHS hospitals10. Half of the respondents reported that senior surgeons and anaesthesiologists sometimes actively resisted the checklists, making it difficult for the rest of the team to complete the tasks. Staff also complained about the checklist itself: that it was poorly worded, time-consuming, inappropriate for certain procedures or redundant with other safety checks. Some also questioned whether there were enough data to support the checklist's use (see 'Why checklists fail').

About one-quarter of the respondents objected to how the checklist had been introduced. Although some hospitals provided training and solicited feedback from staff, at other institutions there was little involvement from those actually working in the operating theatre. That strategy might make it difficult for staff to feel invested in the checklist, and ultimately undermine its correct use. “When it was introduced without any programme or support, it was just impossible, I think, for teams to buy into it,” says psychologist Stephanie Russ, who was part of the research team and is now at the University of Aberdeen, UK.

Mary Dixon-Woods, a medical sociologist at the University of Leicester, UK, interviewed staff members at 17 of the ICUs participating in Matching Michigan11. She found that by the time the programme began, British hospitals had already been involved in numerous government-led efforts to reduce infections. The checklist, she says, was viewed as “yet another example of these top-down, intrusive, imposed initiatives”. It became “something that had to be endured rather than enjoyed”. In Michigan, by contrast, the tool was considered new and exciting. And it was not imposed by the government — it was organized by the well-regarded state hospital association, and participation was voluntary.

Dixon-Woods did identify one exemplary ICU, in which a high infection rate fell to zero after Matching Michigan began. The unit was led by a charismatic physician who championed the checklist and rallied others around it. “He formed coalitions with his colleagues so everyone was singing the same tune, and they just committed as a whole unit to getting this problem under control,” says Dixon-Woods.

Other work has also found that it might be helpful to enlist local champions who can promote an intervention within a hospital, and some have hinted at how to get colleagues on board. In a 2011 study12 of five hospitals in Washington state, Gawande and his colleagues found that it is crucial that leaders take the time to explain how to use the checklist and why it should be used. “That might have included pulling on somebody's heart strings, it might have included sharing as much evidence as possible, it might have included talking through the theoretical story or giving some important example,” says Sara Singer, a health-policy researcher at the Harvard T. H. Chan School of Public Health in Boston, Massachusetts, who co-authored the study.

A local list

Experts also recommend that hospitals modify standard checklists to help the tool fit into the local workflow and to produce a feeling of investment and ownership. Pronovost encouraged the ICUs that participated in the Keystone project to make his checklist their own. “They were 95% the same, but that 5% made it work for them,” he says. “Every one of these hospitals thought that theirs was the best.”

Pronovost and Dixon-Woods also think that several other factors contributed to the success in Michigan ICUs. Providing the hospitals with regular feedback on their infection rates created social pressure for improvement, they say, and regular in-person workshops allowed staff from different hospitals to share their experiences and created the sense of a shared mission.

“When it was introduced without any programme or support, it was just impossible, I think, for teams to buy into it.”

Beyond that, logistics are crucial. When Pronovost was first developing his checklist at Johns Hopkins, he noticed that ICU doctors had to go to eight different places to collect all the supplies they needed to perform a sterile central-line insertion. As part of the Keystone programme, hospitals assembled carts that contained all the necessary supplies.

In a 2013 study13, Dixon-Woods found that an African hospital using the WHO surgical checklist had regular shortages of the basic tools — such as surgical markers, antibiotics and pulse oximeters — that are required to complete the list. But the staff often ticked those boxes anyway; as one anaesthetist pointed out, it was often better for a patient to undergo surgery without these supplies than not to have surgery at all. If the checklist is going to succeed in low-income settings, these problems have to be addressed. “There's no point in having an item that says, 'Have the antibiotics been given?' if there are no antibiotics in the hospital,” says Dixon-Woods.

The clear lesson for hospital leaders is that they cannot just dump a stack of checklists in an operating room — they must observe them being used. Are team members all present? Are they rushing, or skipping steps? If so, then the lapses should be discussed and addressed.

Implementation researchers say that the checklist story may hold lessons for the introduction of other programmes in fields including medicine, education and social work. “We have this massive influx of money to develop innovations,” says Dean Fixsen, who co-founded the US National Implementation Research Network at the University of North Carolina at Chapel Hill. “But the track record of getting that science into practice where it actually produces the kinds of outcomes that we want to see — that track record is abysmal.” Over the past few decades, researchers have published countless papers on evidence-based literacy programmes and teaching strategies. And yet literacy rates for US nine-year-olds, for instance, have barely budged.

Fortunately, Fixsen says, the lessons of implementation science are “completely generalizable”, and all programmes could benefit by noting the importance of engaged leadership, local adaptation and user buy-in. “It doesn't matter how good the innovation is, it doesn't matter how much has been invested,” says Fixsen. “If we don't have the implementation savvy, we're going to get the crummy outcomes that we have seen decade after decade.”