Implementing Change

Illustrations by Katrina Stanley

In 1979, when he returned to his native country, one of the first things Haitian physician Jean Pape noticed was a worker walking the wards of Port-au-Prince's State University Hospital, picking up bodies of dead children and collecting them in a large bag. At the time, almost half of the children admitted to Haitian hospitals for severe diarrhea died.

Sitting in his Port-au-Prince office on a January morning, Pape, now 60, hardens his mouth and strokes his salt-and-pepper goatee while his dark eyes squint behind spectacles when he recalls those days. "I wasn't sure I would stay," says Pape, who had spent several years training at New York City hospitals before his return. "It was so bad."

Pape did stay, and he reversed a trend that was decimating Haiti's youngest generation. Treating infants with antibiotics to eliminate the microbial causes of diarrhea was the standard of care in Haiti at the time. Instead, Pape replaced that with an oral rehydration regimen, and he saw children recovering from their conditions rather than succumbing to dehydration.

Pape then extended the reach of the innovation by encouraging families to adopt oral rehydration, training Haitian doctors to implement it, and closely monitoring the effects of the treatment in his patients. Pape eventually saw the number of infantile diarrhea cases seen in the State University Hospital drop from 8,000 per year when he started to less than 200 in the year 2000. The death rate in Haitian children admitted to the hospital for diarrhea dropped from 44% in the early 1980s to about 1% now. Pape realized that "very simple steps could save lives."

"It started as a calamity and was brought under control," he says. Pape had begun building a robust system for evaluating the effects of his interventions as he closely monitored his infant patients. This metered approach to improving health care in Haiti would inform the next 25 years of Pape's work in the troubled nation, and others' work around the world. "Our entire concept was born," he says.

Pape's concept is seemingly so simple that heralding it should be unnecessary. Yet, public health interventions designed to improve conditions in the developing world rarely incorporate specific follow-up studies into their plans. An agency might distribute vaccines in a rural community, for example, and look for a decline in the number of cases of that particular infection years later. When the precise physiologic effects of a program are not examined with appropriate follow-up, important lessons are lost and mistakes are repeated needlessly elsewhere.

In a recent study assessing the impacts of the distribution of bed nets and antimalarial agents on childhood malaria in Africa, for example, researchers showed that malaria death rates fell dramatically in some countries, but not as much in others. That conclusion was limited because the data collected involved only whether children admitted to hospitals for malaria had died. If scientists had collected more data - such as information on the population dynamics of malaria vectors and changes in transmission rates - they could have compiled a more complete picture, and future efforts would have benefited.

Statistics on the lack of implementation in global heath efforts are scant, but a couple of international organizations have analyzed how well they evaluate outcomes. The results are not promising. A recent internal study of program evaluations carried out at offices of the United Nations Children's Fund (UNICEF) in several countries in 2000 - 2001 concluded: "The 'outputs' of the programme or project were often not adequately described or measured and, with this missing link, the causal chain from activities to outcomes was broken." The report found that one-third of the UNICEF programs studied failed to even mention the measurement of intervention outcomes. "The paucity of output-based contracts to provide specific and measurable targets for projects and programmes, and the related lack of output analysis in evaluations, are serious deficiencies."

Another report published by the World Bank, which funds several public health interventions in developing nations, found similar results. The report revealed that of all the health, nutrition, and population projects that ended in FY 2004 and 2005, only 25% had satisfactory monitoring and evaluation of outcomes. The report also found that only "50% of the projects included any outcome-output indicator in their results framework and, of those, less than 50% measured these indicators at least twice."

Of the more than $5.2 billion the Global Fund has awarded for AIDS, tuberculosis, and malaria programs since 2002, only 3% - or about $158 million - has been dedicated to evaluating outcomes. Other organizations funding global public health projects (such as the Bill and Melinda Gates Foundation and the World Bank) said they could not provide such data.

The lack of rigorous evaluation of the impact of public health interventions goes by various names. Some call it the "evaluation gap," while others refer to it as the "know-do gap." To close that gap, public health researchers have begun referring to a field known as "implementation science." Such efforts are directed not only at the developing world; an entire journal, Implementation Science, publishes studies from highly developed nations on similar subjects.

Roger Glass, the director of the National Institutes of Health's John E. Fogarty International Center in Bethesda, Md., says that the institutionalized use of implementation science to ferret out the causes for program failure has largely been ignored in efforts to ameliorate the public health woes of the developing world. "There are many different things that can interfere with our ability to implement an intervention," says Glass. "That's an area where we've paid precious little attention." Ellis McKenzie, a Fogarty researcher who models malaria control efforts in Africa, borrows a business world metaphor: "You can't manage what you don't measure."

In Bangladesh, the modern epicenter of waterborne cholera infection, international organizations and the Bangladeshi government have been trying to tamp down epidemics for decades. One program, supported in part by UNICEF, involved drilling tube wells across Bangladesh in the 1970s to provide rural populations with safe sources of drinking water. It was a simple intervention on its surface: Increase access to cholera-free water, and the rate of cholera infection should decrease. The wells were drilled, well water was tested and deemed cholera-free, and villagers instructed to use the wells instead of rivers, streams, or ponds.

A year later, however, researchers not involved with the well-drilling program sampled one population of rural Bangladeshis and found that the population's rate of cholera infection held steady. ¹ It turns out that villagers living near the wells were drinking from them, but they were also using non-well water in sufficient quantities to keep the cholera rate high. Because monitoring and evaluation were not incorporated into the design of the program, says Glass, the intervention seemed to be missing its mark. Though he was not involved with the project, Glass says that systematically monitoring people's preferences for, or aversions to, well water throughout the implementation of the program could have increased the success of the program, and possibly saved lives. "Scientifically, we did the right thing," Glass says, "but we didn't understand people's behavior."

Since that early failure, cholera rates in Bangladesh have indeed remained lower thanks in part to access to clean drinking water. (Unfortunately, no one was testing for arsenic when the wells were drilled, and the country now faces the unintended consequence of arsenic poisoning.) But on a global scale, public health programs have lacked the systematic evaluation of outcomes that seem necessary for success. Researchers observing public health interventions from the outside have, however, uncovered similarly disappointing results in the implementation of other programs in the developing world.

For example, a program centered on the distribution of insecticide-laced bed nets to slow the spread of malaria in sub-Saharan Africa seems to have faltered initially because young children were not sleeping under the nets in great numbers. ² Researchers didn't know that because they were conducting follow-up surveys only once every three to five years. Had they conducted annual studies, they might have been able to change the outcome more rapidly.

Why have even well-intentioned public health programs given the evaluation of outcomes short shrift? The reasons are complex and touch on every aspect of health program planning, from the economics of fund allocation to ingrained modes of thinking about large-scale public health interventions. Public health program administrators "really don't like to talk about technical issues - whether the programs they're designing are working or not," says David Sack, a professor at Johns Hopkins Bloomberg School of Public Health. Sack says that administrators tend to place emphasis on the immediate amelioration of public health concerns and view tracking the impact of interventions as a secondary consideration. "If you call something 'research,' then it may seem irrelevant or an ivory tower type of thing," says Sack, who is also the director of the Center for Health and Population Research in Bangladesh.

Then there's the aspect of devoting already scarce funding to evaluation in global health programs. Johanna Daily, an infectious disease doctor and professor at Harvard Medical School, says that thorough evaluations may cost less than purchasing drugs, hiring health workers, and other implementation costs, they still cost something. "When people are criticized for not measuring, you have to understand that they are making that decision," she says. "There's a cost to measurement that has to be balanced with rational decisions."

Daily says that public health funding is so hard to come by in the developing world, that money is spent on treatments and interventions that are known or suspected to work, while funding for evaluation of the impact is often ignored. "There's so much work to be done and there's so little money," she says. Given the paucity of resources, says Daily, who also works on malaria projects with a Boston-based aid organization, Partners in Health, this disconnection is understandable. "If I had $100,000 I guess I would buy $100,000 worth of vaccines."

Ruth Levine, a senior fellow at the Center for Global Development (CGD), says that program managers must change how they think about large public health programs. "Their fundamental reward system is not built around learning," Levine says, "and it's certainly not built around discovering that you've made a mistake."

Levine, a health economist who has worked at the World Bank, says she has witnessed several programs whittle back money set aside for evaluation and redirect it towards implementation as interventions progressed. "On any given day, the incentive choices all drive toward getting the money out the door or getting the next incentive: getting the next drug or training the next teacher," she says. "There are lots of disincentives to doing rigorous evaluation."

Levine, who cochaired CGD's Evaluation Gap Working Group, says that the public health community is awaking to the need for structured evaluation. To bring about wide changes, however, policy makers should devote and protect program resources with firewalls that prevent their "cannibalization."

Making seemingly obvious changes to public health programs will enhance implementation science's effect, she says. Increasing the reach and utility of implementation science in public health programs can be achieved by increasing access to existing epidemiologic and demographic information, building the capacity of local researchers in developing nations to conduct new impact evaluations, and devoting more resources to impact evaluations. This investment in implementation science, says Levine, would lead to more rigorous random control evaluations, which would aid public health administrators and policy makers immensely.

Emmanuela Gakidou, a Harvard researcher, was part of an international team that in 2006 exhaustively quantified the effects of sweeping Mexican healthcare reforms. ³ She agrees that health programs should be evaluated with every bit of the rigor that bench scientists use to approach basic biological questions. "We should hold health policy experiments to the same standards that we hold clinical trials," she says. This is especially crucial in the developing world, where unique cultural, infrastructural, and demographic challenges can doom well-intentioned public health interventions to failure, whereas a firm grasp of the outcomes of interventions can spell success. "Policy makers need to send a signal that says, 'We care about impact,'" Levine says.

"How will we know if we're really seeing the impact we expect?" Glass asks of the public health community. "We spend a lot of effort to develop new tools, vaccines, and drugs, and then we wash our hands of them."

Some programs have relied on rigorous evaluations for success. Onchocerciasis - known as river blindness because of its most extreme symptom - is the second leading infectious cause of blindness in the world. According to World Health Organization (WHO) statistics, the disease exists in more than 30 countries, with 99% of cases confined to tropical Africa. Onchocerciasis commonly strikes populations living near fast flowing African rivers where blackflies breed and transmit the parasitic filarial worm, Onchocerca volvulus, through their bites. The disease manifests itself gradually with itching, skin lesions, and eventually blindness as exposure to blackfly bites increases.

A boy with river blindness in the Nuba mountains, Sudan.

In 1974, WHO with the support of the World Bank, the United Nations Development Program, and several other donors, launched the Onchocerciasis Control Program (OCP) in West African countries hardest hit by the disease. At the OCP's outset, transmission of onchocerciasis could be controlled in only one way - aerial insecticide spraying of blackfly breeding areas where swarms of aquatic larvae metamorphosed into disease-harboring adults. Widespread larviciding seemed to be working well, but in the late 1970s, a new disease control tool emerged.

Ivermectin, an antiparasitic drug, was tested in clinical trials throughout Africa and found to be effective in treating onchocerciasis. By the late 1980's, the OCP program had adopted ivermectin as another prong in its attack on onchocerciasis, and it began distributing the drug in the most onchocerciasis-ridden areas of West Africa, thanks to donations by pharmaceutical giant Merck. The OCP didn't just distribute the drug and hope that the desired effect would materialize. Instead, it enlisted the assistance of Dik Habbema, a public health researcher at Rotterdam's Erasmus Medical Center, to carefully track the effect of the drug on populations in the path of onchocerciasis.

Habbema says that he originally collaborated with the OCP earlier, studying the longevity of the adult Onchocerca volvulus worm in the human body⁴ and the likelihood of the disease's recrudescence in core OCP areas where larviciding was reducing blackfly populations. ⁵

Left: A pharmacist sorts antiretroviral drugs at GHESKIO's Institute of Infectious Diseases and Reproductive Health. Inset: Jean Pape, cofounder and director of GHESKIO.

Now Habbema and his colleagues were called in to model the transmission and control of the disease, given a variety of potential strategies. They conducted community-based drug trials in which infected and uninfected individuals were treated with ivermectin. They showed that administering the drug resulted in a 96%-99% drop in parasite loads in individuals with onchocerciasis, and that these loads rebounded more slowly in the population than had been shown in clinical trials. ⁶ With constantly refreshed data, support from the OCP's financial backers, and open communication between his research team and OCP administrators, Habbema was able to influence OCP policy decisions through evaluation of the program's impact concurrently with its implementation.

"The continuation [of my collaboration with the OCP] was not planned at all," Habbema recalls, "but has been a consequence of mutual trust and satisfaction with how things were going." It's not hard to imagine why the OCP has been satisfied with Habbema's contribution to the program. In continuing the systematic evaluation of the program's administration, Habbema has affected substantive changes in ivermectin distribution strategies, which have helped to prevent 600,000 new cases of onchocerciasis, according to WHO estimates.

The OCP ended in 2002, but Habbema's research continues to inform an African-based onchocerciasis control program, the African Program for Onchocerciasis Control (APOC), which started in 1995 to battle the disease in less severely afflicted countries outside the OCP's control area. "We use both data from the new program and scientific results from the old program," Habbema says. "We have combined them as much as possible for the APOC program."

Pape's office sits at the end of a hallway. As I enter GHESKIO - the clinic, research, and training facility that he cofounded in 1982 - I step over the concrete threshold and walk through water. Tides regularly invade this hallway, leaving staffers to bail water from the floor as if trying to save a sinking boat.

By the mid-1980s, though he had effected real change in the way infantile diarrhea plagued his country, neither Pape nor his countrymen realized the magnitude of the health crisis that was poised to befall Haiti next. They began seeing patients with a baffling constellation of symptoms, including rare opportunistic infections and tumors that were thought to affect only elderly men.

With funding from US universities and international aid organizations, Pape cofounded The Haitian Study Group on Kaposi's Sarcoma and Opportunistic Infections (GHESKIO, which Haitians pronounce jess-kee-o). Early studies at GHESKIO sought to describe a new and frightening disease on the already precarious Haitian healthcare landscape. Pape and his colleagues at GHESKIO were the first to examine the clinical manifestations of HIV/AIDS in Haiti. ⁷ In 1985, he and his colleagues compared different doses and administration routes of a common drug for pneumonia in their efforts to curb skin reactions in a handful of GHESKIO's patients with AIDS.⁸

More implementation studies followed. One compared the efficacy of two drugs used to treat esophageal yeast infections commonly seen in GHESKIO's patients with AIDS, ⁹ while another evaluated different medication regimes for AIDS patients with severe diarrhea caused by a microbial infection. ¹⁰

Pape and his colleagues have been fine-tuning the ways that they administer care, treatment, and counseling to GHESKIO's patients. A constant flow of research from the center, utilizing the almost 150,000 patients GHESKIO sees per year, helps Pape and his colleagues to adapt GHESKIO policies to better serve the beleaguered Haitian populace. For example, recent GHESKIO research has explored the use of a simplified p24 antigen-capture ELISA as an alternative to using PCR (scarce in Haiti) to provide the facility's 30 or so rural extension clinics with a low-cost, low-tech method to test infants for HIV.¹¹

The HIV epidemic in Haiti continues, though it appears to have abated. National prevalence peaked at 5% in 1994.¹² According to current WHO estimates, the prevalence of HIV in Haiti is now 3.8%. That progress has come in the midst of chaos. GHESKIO has withstood a procession of transitional governments, two revolutions, and widespread disorder in the poorest nation in the Western Hemisphere in a guarded compounded girded by high walls topped with razor wire.

GHESKIO sits in the middle of one of Port-au-Prince's worst neighborhoods. Inhabitants of the city call the slum-fringed area "Kosovo." Just four years ago, the echoes of gunfire crackled off crippled buildings in Kosovo as acrid smoke wafted through the cacophony of another revolution. Kosovo is quieter now, but the chaos of revolution has been replaced with the white noise of a country teetering between collapse and rejuvenation. Smoke still swirls through the air, but now the fires cook Haitian street fare instead of razing buildings. A river of people flows through the wide expanse of Harry Truman Boulevard and fills the winding side alleys, but these days instead of rioting, people jam the streets to hawk goods such as phone cards and rubber steering-wheel covers. Wiry dogs stalk the crumbled sidewalks, and United Nations peacekeepers still patrol the area.

The center's leader, dressed in his white lab coat, roams the crumbling hallways conversing with his staff and asking for input and advice. "This is what encourages me," Pape says of his staff, "the fact that we have a really excellent team. It's not me who created this model, it's all of us."

Pape and his collaborators have kept this project alive through the tumult of Haiti's recent history by utilizing outside help from global health organizations. They've built an internal network to systematically monitor the success of health interventions and establish a robust link between evaluation and adaptive management to fit the unique challenges of delivering public health in Haiti.

Hatians wait to be tested for HIV at GHESKIO's Institute of Infectious Diseases and Reproductive Health.

left: Malcolm Linton/Getty Images

"Jean Pape does that better than anyone else," says NIH's Glass. Pape has no plans of resting on his laurels, and there are a few, including a Legion d'Honneur award from the French government. Instead, he is thinking broadly about exporting the model for combining research, care, and training at GHESKIO beyond the chaotic m￩lange that surrounds his clinic. "The fact that it has worked here means it will work anywhere," Pape says. "I think it will be best exported to sub-Saharan Africa. One of the goals is to share experience."

Though his path to success among the hardscrabble realities of living, doctoring, and studying in Haiti was a winding one, Pape, leaning back into his chair behind a desk filled with manuscripts to review and a computer screen choked with e-mails to answer, renders it down to a simple equation. "It's not easy," he says, "but you can obtain huge results if you concentrate, stay out of politics, and focus on your work."