When a funder or policymaker wants to achieve a certain goal—for instance, increasing citizen security—it is not always obvious which strategy will have the largest impact on the intended beneficiaries. Will increasing the police presence in “hot spots” for crime reduce crime overall or simply push it around the corner? For good policies and programs to be implemented, funders and policymakers need to know the impacts that their different options have on the outcomes they are targeting. Throughout this blog, we will refer to policymakers, taking this term to include funders too. Rigorous impact evaluations, such as those carried out by the Abdul Latif Jameel Poverty Action Lab (J-PAL) and others provide a valuable way to generate insights into the effectiveness of various potential strategies. J-PAL is a research center at MIT whose mission is to reduce poverty by ensuring that policy is informed by scientific evidence. The 145 professors in our research network specialize in rigorously evaluating the effectiveness of social programs. In the last fifteen years, J-PAL’s network has contributed to a growing body of impact evaluations across a variety of fields including agriculture, education, governance, and health.
In our work we are increasingly seeing that the problem facing policymakers is usually not that there are no good approaches to solve a problem, or that there are only bad solutions, but finding ways to choose among multiple effective interventions. It is not only about the effectiveness, but also the costs of the many approaches policymakers might pursue to achieve a certain goal. One way to make it easier for policymakers to compare a wide variety of programs is to present evidence in the form of comparative cost-effectiveness analysis. In the simplest terms, cost-effectiveness analysis calculates the ratio of the amount of “effect” a program achieves for a given amount of cost. J-PAL conducts comparative cost-effectiveness analysis to provide tools for policymakers to compare the relative efficiency of programs targeting the same outcome. To date, our cost-effectiveness analysis has focused on the education sector, but outcomes from many other sectors, including peacebuilding work, could also benefit from this type of analysis. J-PAL’s cost-effectiveness analyses on education-focused programs that aim to improve student participation and test scores have been widely disseminated to help policymakers consider effective strategies to improve education while accounting for the implicit cost tradeoffs required for any decision about how to spend scarce resources. And, when communicated effectively, cost effectiveness analysis can have wide-reaching influence on the strategies used to address social challenges. For instance, after a randomized evaluation found that providing students in Kenya with deworming pills was an incredibly cost-effective way to increase student attendance by ensuring they were healthy enough to attend, an organization called Deworm the World was launched to coordinate technical assistance and advocacy efforts for sustainable, large-scale school-based deworming programs. In the 2015-16 school year, over 190 million children around the world received deworming treatment. While cost-effectiveness analysis is not the only factor on which policymakers make decisions, it does provide a useful starting point for researchers and policymakers to collaborate in assessing the efficacy of the different programs and their relevance to the particular situation. Detailed cost-effectiveness results, combined with an understanding of the problem being addressed and of other contextual factors such as current input prices and local institutions, can provide important insights into which programs are likely to provide the greatest value for money, and to identify the key factors to which these outcomes are most sensitive. J-PAL will continue to produce cost-effectiveness analyses to help policymakers in a variety of fields, including energy and the environment and health. However, a first step in conducting comparative cost-effectiveness analysis is to identify a single outcome that has been rigorously evaluated across the programs being included. While J-PAL affiliates have conducted more than 800 impact evaluations across a wide range of topics, experimental evidence on what works in the areas of crime, peacebuilding, and violence is limited. Together with our partners at Innovations for Poverty Action, J-PAL is working to reduce this evidence gap. J-PAL’s Crime and Violence Initiative and IPA’s Peace and Recovery Program will support experimental research in several broad areas: reducing crime and violence, promoting peace, reducing “fragility,” and preventing and coping with crises. This research has the potential to generate new evidence on both the costs and the effectiveness of many possible strategies to promote peace. Cost-effectiveness analyses can build on the results of these evaluations to help policymakers identify the best strategies to increase peace and overall well-being. Read J-PAL’s methodology paper outlining their approach to comparative cost-effectiveness analysis. Sam Carter is a Policy Associate at J-PAL Global, where she supports J-PAL’s cost-effectiveness analysis, Finance sector, and Government Partnership Initiative (GPI). Sam works with governments, international and non-governmental organizations, and the private sector to develop new research projects and to help encourage decisions based on rigorous evidence.
1 Comment
I recently attended a roundtable discussion on the Cost-Effectiveness of Peacebuilding hosted by the Alliance for Peacebuilding and the Institute for Economics and Peace (IEP) where IEP presented its report on Measuring Peacebuilding Cost-Effectiveness. As an Industrial Engineer and Certified Cost Analyst, I have spent my career analyzing costs in the software, consumer products, aerospace, technology, and defense industries. For those industries, cost is a critical factor in deciding how to allocate resources, establish budgets, measure performance and make trade-offs about investment decisions. I am excited that the peacebuilding community is beginning to examine the concepts of cost analysis and cost-effectiveness in order to make more informed decisions in allocating scarce resources. As I participated in the discussion and met with peacebuilders afterwards to share my experiences, I realized that the concepts of cost analysis and cost-effectiveness are new to many in this community. Understandably, there are many questions about their value, application to peacebuilding, and how findings would be used. Peacebuilders’ valid concerns about the application of cost analysis to their work fall into two primary categories related to findings and processes. Note: Cost analysis is a term used to describe the broader field of study that includes cost estimating / forecasting, should-cost analysis, and business case development. Cost-effectiveness studies are a subset of cost analysis which will be defined later in this blog entry. Findings: As the community works hard to empirically demonstrate the impact of peacebuilding programs and secure scarce resources, it can feel tangential at best, and threatening at worst, to focus on measuring the cost-effectiveness of peacebuilding programs. In our current political climate, some worry about diverting time and money away from what is perceived to be an existential battle for the future of peacebuilding to a more nascent evaluation agenda. Many are concerned that looking at cost might add fuel to the political fire by revealing opportunities to reduce spending on costly or ineffective programs. Throughout my career as a cost analyst, I have found the opposite to be true. Those organizations and programs that use the tools of cost analysis to strengthen the case for more resources typically end up with more funding, not less. For mission-driven and impact-hungry practitioners, cost analysis enables better decision making about where to spend each dollar across a program or portfolio to achieve greater impact. Processes: The second category of concern is related to processes. Many scholars and practitioners in the peacebuilding field ground their practice in systems-based approaches. Transforming interconnected social, political, and economic interests that drive violent conflict demands a complexity-aware approach. Systems thinking offers an invaluable mindset and toolkit. It is therefore promising that cost analysis has been tailored to complex, adaptive problems and corresponding programs. For example, it has been leveraged in similarly complex fields such as health care, space exploration, and advanced and emerging technologies (laser weapons, robotics, artificial intelligence). Complexity is not a reason to avoid cost analysis but it will require us to do so in systems-sensitive ways. For example, as we examine a portfolio of peacebuilding programs to determine their cost-effectiveness, we should include a spectrum of activities and programs that range from complicated (challenging technical programming such as community-oriented media programs to counter hate speech) to more complex (adaptive problems and programs such as support to transitional justice programs). A complex transitional justice program, such as support to prosecutors to convict a war criminal, might take millions of dollars over a couple of years, but could serve as a powerful deterrent to future warlords and conflict entrepreneurs, creating important ripple effects across a conflict eco-system. As is the case in other complex situations, cost is rarely the only factor in deciding between courses of action. Oftentimes the costlier option is the most prudent. The tools of cost analysis can demonstrate that while making the case for additional funding based on data and objective evidence. Regardless of the complexity of the environment, industry, program, or activity, there are three key questions that cost estimators seek to answer when performing cost analysis.
To determine a reasonable cost for an activity, cost estimators collect and analyze data to establish cost benchmarks and “should-cost” estimates. They also evaluate cost drivers that either increase or decrease the target cost of an activity and provide a method for establishing reasonable costs across different contexts and environmental conditions. This allows decision makers to evaluate the cost of alternatives and allocate resources to maximize effectiveness based on various funding levels. The ultimate goal of cost-effectiveness is to determine the optimal allocation of resources that provides the maximum effectiveness. Costly and ineffective activities should be eliminated in favor of low cost, highly effective activities. The figure to the right is a simple illustration of this principle. Those that are both effective and cost efficient (quadrant I) may warrant more funding and expanded implementation. Those that fall into quadrant IV (high cost and ineffective) should be eliminated so that resources are not wasted and can be re-allocated. Activities that fall into quadrant II and III warrant further investigation. For highly effective but costly activities, steps might be taken to reduce costs on those programs to make them more cost efficient. For low cost activities that are not achieving the desired outcomes, additional funding may increase their effectiveness making them better options than more costly alternatives. Increasingly, the peacebuilding community (practitioners, researchers, and donors) has committed resources to monitoring and evaluation (M&E) to measure both program outputs (did we do what we said we would do) and outcomes / effectiveness (did the program have the intended impact). M&E seeks to determine if a program was successful by determining the extent to which it achieved the desired outputs and outcomes. However, this view of success almost always ignores the question of whether the program objectives were achieved at a reasonable cost to the donor/funder. I believe most would agree that a successful peacebuilding / conflict prevention program should not only be effective but also cost-effective. The figure below highlights the key questions that cost analysts like myself traditionally seek to answer when evaluating cost-effectiveness. The UK’s Department for International Development has developed a similar concept to cost-effectiveness called Value for Money (VFM) that uses four principles to evaluate “the optimal use of resources to achieve intended outcomes”.
The VFM model presents a simple framework to apply the principles of cost-effectiveness to the peacebuilding community. Although the framework and concepts are simple, the task of collecting the necessary data can seem daunting. There are also many questions about whether the data even exists to evaluate cost-effectiveness for peacebuilding programs. After discussing this challenge with several experts in the peacebuilding community, I believe the data to answer the questions of cost efficiency, program effectiveness, and cost effectiveness are all available in the form of expenditure data (i.e. actual costs from accounting data) and M&E data as shown in the figure below. In my experience, one of the primary benefits of cost analysis is that it frequently identifies program’s that are insufficiently funded to achieve program objectives. Most of the programs that I have worked on as a cost estimator had insufficient funding because program budgets were based on overly optimistic and subjective assumptions instead of established cost benchmarks. I have also seen many programs awarded to the lowest bidder without any independent evaluation of whether the bid price was reasonable. These programs invariably fail to achieve the intended outcomes due to unrealistic and unreasonable budgets.
Establishing cost benchmarks for peacebuilding activities would provide insights into whether programs are sufficiently funded and increase the likelihood of program success for programs with insufficient funding. Cost analysis would allow funders to make informed decisions that balance the cost of activities with their expected benefits. In my time as a cost estimator, I have seen the use of cost analysis:
I believe that the peacebuilding community would experience many of these benefits by adopting the principles of cost analysis and developing a culture of cost awareness. At a time when U.S. government funding for peace and development programs is likely to decrease while violence is increasing, the need for cost analysis is more urgent than ever. Steve Sheamer is the Chief Operating Officer of Frontier Design Group, a consulting firm focused on solving the world’s most complex national and human security challenges. He has spent his entire professional career focused on reducing the cost of doing business through rigorous cost analysis, performance measurement, improved business processes, and technology implementation. He is a Certified Cost Estimator, Industrial Engineer, Lean Six-Sigma Black Belt, and Project Management Professional with over 15 years of experience supporting large, complex programs. His industry experience includes consumer product manufacturing, computer chip fabrication, aerospace and defense, and management consulting. He is passionate about identifying cost savings, improving program management processes, and implementing tools to improve the effectiveness of large, complex organizations and programs. Moving Towards More Cost-Effective Peacebuilding
By Evan Hoffman Recently I was invited to design and deliver the first-ever graduate-level course on ethics and best practices in peacebuilding and conflict resolution for the Department of Conflict Resolution Studies at Nova Southeastern University in Fort Lauderdale, Florida. One of the topics that I may include is effectiveness and, more specifically, cost-effectiveness. An ethical peacebuilder has a responsibility to select the best peacebuilding tool for the situation and follow widely accepted best practices when implementing it. EFFECTIVENESS AND COST-EFFECTIVENESS Some say the peacebuilding field formally emerged about fifty years ago. Since then, we have seen many peacebuilding initiatives launched in numerous locations around the world in an effort to prevent or end violent conflict and war. The range of activities that fall under the peacebuilding umbrella are diverse. They include everything from art-based initiatives to sports for peace, and a range of other activities like grassroots dialogue, diplomacy and mediation. Sanctions and the use of force are also used to promote peace in some circumstances. Steven Pinker argues that we are currently living in one of the more peaceful periods in human history. Perhaps this is attributable to peacebuilding efforts. Many make the case that peacebuilders intentionally choose to launch only those specific peacebuilding activities that they believe will effectively and efficiently reduce or eliminate violent conflict. Donors that fund such activities believe these efforts are worthy of receiving funding. Others will observe that, to date, there have been few objective and definitive studies addressing the most fundamental question: do we really know what works to build peace in the most effective or cost-effective manner? The people from this camp would argue that peacebuilding activities are ad hoc and occur because the right mix of people and resources come together at the right moment. No study has directly addressed these fundamental questions about which peacebuilding tools are the most (cost) effective though some aspects of this question have been looked at. What does it mean that a peacebuilder should select the tool that will be the most effective? By most effective we mean it will have the greatest possible impact on creating a lasting peace (this, of course, implies that we can measure impact!). The best tool will be effective and cost-effective. While little empirical research exists about which specific tools are the most cost-effective, some initial thinking I did on this topic suggests lobbying, unarmed civilian peacekeeping and conflict resolution training are the 3 best tools because they have low costs and the potential to create major impacts.[1] More research is needed on this topic. EFFECTIVE PEACEBUILDING, BEST PRACTICE Until that research is done, however, we need to follow existing best practices to ensure that our peacebuilding efforts are as effective and ethical as possible. What is effective peacebuilding? Is there any good guidance for this? One of the key principals of the field is that we should “do no harm”. All of us would agree that we need to do more than just avoiding harm. We also need to do some good! When we aim to simply do no harm perhaps we are not even doing “good”! We must move beyond this minimum threshold and ensure our programing is designed to create a positive and lasting impact in the country or region where we are working. We could thus revise the “do no harm” mantra to “let’s do no harm while doing good!”[2]. Another aspect of effective peacebuilding involves using an underlying theory of change and a structured decision-making model to guide our efforts. This became evident to me in my 5 years of work helping prevent violent conflict from emerging in Guinea-Bissau. We made preventing and reducing violence the main objective of our work. We deliberately took numerous targeted actions to address all of the actors and factors that were driving violence based on a new tool we developed for preventive action decision-making. Seeking a cost-effective approach to peacebuilding is easier said than done because there is no clear indication of which tools are the most cost-effective. For now we should aim to be as effective as possible while the relative costs associated with different options are researched.[3] However, one thing is crystal clear: the costs of dialogue will always be less than the costs of war. For this reason alone we should be putting more resources into preventative activities such as dialogues, mediation and Track 2 diplomacy. Evan Hoffman, PhD, is a Senior Associate at the Canadian International Institute of Applied Negotiation (CIIAN). Dr. Hoffman has published numerous articles on the themes of conflict prevention and resolution, peacebuilding, and mediation and he has provided consulting services to Global Affairs Canada (GAC), the Carter Center, the UN, the EU, the Ottawa Police Service, St. Lawrence College (Cornwall), the Vietnamese Ministry of Justice and others on these topics. Over the last ten years, he’s conducted workshops and trainings with hundreds of community leaders, university students, police officers, and government officials from around the world. [1] Another interesting ethical question arises from this: would you trade off effectiveness for reduced costs? [2] This recent article on UN peacekeeping argues that peacekeeping must also do no harm. See http://reliefweb.int/report/world/peacekeeping-must-be-more-flexible-adapting-evolving-threats-top-officials-tell-special [3] I could see donors also needing to develop criteria and checklists to help assess project proposals for their potential cost-effectiveness.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
October 2017
Categories |