In our recent article 'How Are Impact-Driven Funders Thinking About Giving Better?' we interviewed over 20 leading impact-driven funders to identify common needs the philanthropic sector is facing in response to COVID-19. One of the top issues highlighted was the need for new models of determining program effectiveness given the constraints of the pandemic.
This article offers 3 recommendations for how the social sector can better understand its impact during and after Covid-19. The recommendations are in response to the challenges highlighted by the philanthropic sector during the course of our recent research and offer global best practices and real-life examples of organizations tackling each solution.
#1 Cross-check results and use independent validation techniques when implementing digital and phone-based data collection processes
Challenge: Most impact-driven organizations collect some form of data from their beneficiaries to track progress, improve programs, and report results to funders and board members. However, in-person data collection methods are being upended due to restrictions on movement and close physical contact during COVID-19. Both formal data collection, such as in-person forms, surveys, and interviews, as well as informal data collection, such as casual conversations with beneficiaries during the course of service delivery, are being disrupted. As a result, many organizations are implementing digital and phone-based data collection processes during restrictions on in-person interaction.
The rise of remote data collection processes begs the question of whether these practices are being implemented correctly and yielding accurate results. Organizations who normally collect data face-to-face are looking for new ways to collect data remotely and ensure that these new strategies are as accurate as their previous methods.
Sector Inspiration: Organizations should consider investing upfront resources to understand the advantages, disadvantages, and implementation considerations when switching to remote data collection techniques. It is important to assess the pros and cons of various remote data collection techniques, understand how to pick the best tool for the situation, and avoid common pitfalls when implementing new techniques.
Once new methods have been implemented, use systems to cross-check and independently validate results. Cross-checking results and using independent validations can reveal how new systems compare to in-person data collection methods.
Example: Organizations that specialize in remote data collection methods have conducted extensive research on the pros and cons of various remote data collection techniques. For instance, 60 Decibels leverages expertise from the international development field, where remote collection techniques are common. They provide free toolkits and best-practice resources for implementing remote data collection strategies during COVID-19. Additionally, they offer resources detailing the advantages and disadvantages of different technologies, such as SMS, web-based surveys, and phone surveys, along with recommendations for selecting the most suitable method for a given context.
When implemented correctly, remote data collection methods can offer a more cost-effective and pandemic-friendly way to gather essential data from recipients. As restrictions ease, organizations can compare the accuracy and results of in-person versus remote collection and make informed decisions about which new practices to adopt by conducting independent validation studies. In the long run, remote data collection may prove to be a more efficient means of data collection if implemented correctly.
#2 Explore innovative evaluation techniques to get more out of existing data
Challenge: In response to the vast needs arising out of Covid-19, social sector organizations are adapting and designing programs to meet the needs on the ground. However, many organizations report a lack of data and reduced ability to collect new data during the current crisis. Organizations are looking for practical ways to determine whether their efforts are working and how they can improve.
Sector Inspiration: Now is the perfect time for organizations to explore creative solutions that offer low-cost, efficient impact assessment options. Many evaluation techniques can utilize existing data and resources within an organization to help leaders understand what works and how they can do better.
Example: Data integration projects allow organizations to merge existing data sets to draw valuable new insights. For example, in the Equality of Opportunity Project, Professor Raj Chetty linked anonymized government tax data with other datasets to analyze outcomes related to social mobility in the United States. Integrating data from different sources and analyzing that data using statistical and econometric tools allowed the researchers to draw national longitudinal insights. Most foundations and service providers, especially those who fund or manage multiple programs, already collect a wealth of data that can be utilized in often surprising ways. In some cases, data integration and analysis can even offer a cheaper, more accurate way to analyze outcomes than traditional data collection.
#3 Create clearinghouses to make evidence-based decisions about where to prioritize resources
Challenge: Impact-driven funders make tough decisions every day about which organizations to support, whether they should concentrate resources or spread funds across multiple organizations, and which programs within an organization to fund for maximum impact. The economic fallout of Covid-19 exacerbates the need to prioritize scarce resources wisely. However, limited social sector capacity and lack of information required to compare the benefits and costs of potential interventions often leads to suboptimal allocation of resources.
Sector Inspiration: To address this problem, countries around the world have developed clearinghouses that help funders, governments, and the social sector allocate resources effectively. Clearinghouses are used as centralized intermediaries between 'buyers' and 'sellers' of social programs that provide objective evaluations of program impact and enable easy cross-comparison through cost-benefit analyses. They are usually a shared public resource that compile assessments of social interventions across a number of issue areas and use a standardized measurement system to compare their costs and effectiveness.
For example, a clearinghouse may compile all the assessments of interventions in K-12 education, youth mental health, and substance use disorders and identify which interventions are most successful at achieving specified outcomes. They also standardize the outcomes so that they can easily compare the cost-effectiveness of existing interventions in each issue area. The results can then be used by governments, nonprofit organizations, or impact-driven donors to ensure their decisions about which social programs to develop, improve, or fund are based on the latest evidence.
Example: The Washington State Institute for Public Policy (WSIPP) has been directed by the Washington State Legislature in the United States to identify evidence-based policies that lead to better outcomes on a range of social issues. WSIPP compiles existing interventions in each issue area and conducts cost-benefit analyses of each policy to determine the amount of taxpayer dollars it would cost to achieve the improved outcome. This allows WISPP to compare the costs and benefits of various options in monetary terms and determine the likelihood that the social benefits will exceed the costs of the program. This information is then used by the government to determine how to achieve the most positive outcomes with limited resources.
The clearinghouse approach could be applied to the nonprofit sector in Asia by creating a common aggregation platform to compile evidence-based interventions in specified issue areas and create a standardized analysis method to compare their effectiveness and cost. This would offer funders an easy, evidence-based way to direct their funding for maximum impact and support service providers in developing interventions based on the latest evidence.
If you're interested in learning more about how and why to measure program effectiveness and allocate resources, join our upcoming webinar 'New Challenges and Opportunities in Impact Measurement' on June 24th from 3:00 - 4:30 PM. You can register your interest here.
To stay updated with our work and the article series, please subscribe below:
Authors
Kevin Tan
kevin_tan@trisector.org
