Autores: Thomas Aston, Florencia Guerzovich e Alix Wadeson.
It’s strategy refresh time
The Biden administration is figuring out whether and how to walk the talk on anti-corruption, the Open Society Foundations and the Hewlett Foundation’s Transparency, Participation and Accountability (TPA) Programme are doing a strategy refresh. The World Bank’s Global Partnership for Social Accountability (GPSA) is considering a Strategic Review. These are some of the biggest players in the sector. Each player has its own “niche” and approach to build a portfolio. Yet, in considering some possible new directions, the Hewlett Foundation has put out a consultation. Al Kags offered some thoughts on what to fund and offered a wish list earlier this week. Hewlett asked an important set of questions for all of us which we have slightly amended:
- How should we measure progress toward outcomes and in priority countries in a given portfolio?
- How can we best contribute useful knowledge to the field through grantmaking, commissioning evaluations, and facilitating peer learning?
- And what can a portfolio’s monitoring and evaluation system do to link the answer to both questions together?
To answer these questions, we must first acknowledge a wider issue — null results terrify us. Every time a new Randomised Control Trial (RCT) offers anything less than unambiguously positive results, we have groundhog day about whether the whole sector is a worthy investment or not.
Nathanial Heller captured this trepidation well for yet another study in Uganda about to publish its results:
A handful of initiatives have given the impression to donors that transparency and accountability efforts don’t work. One of these was the Making All Voices Count (MAVC) programme, which some donors (unfairly) called a “failure,” point blank in 2017. Further, as one of us explains, two studies in 2019, the Transparency for Development project (T4D) and another from Pia Raffler, Dan Posner and Doug Parkerson found null results, and this caused collective consternation in the sector.
The conversation seems stuck in a vicious feedback loop. So, to demonstrate success, many rely on idiosyncratic cases and lean very heavily on a handful of country contexts which conducted a lot of RCTs or narrowed the focus of study to common tools (e.g. scorecards) and/or outcome measures (for an effort to standardize indicators). Many others have sought refuge through “pivots” and “innovation” rather than having a candid conversation about mixed evidence and what we might do (or not) to escape the feedback loop. As ex-International Development Secretary, Rory Stuart, recently argued [talking about Afghanistan], ‘“we have to stop [saying] “either it was a disaster or it was a triumph.”’
The myth of homogeneous and generalisable success
Despite this sage advice, one expert recently told the Hewlett Foundation that a “lack of evidence about the impact of TPA initiatives is now an existential threat to the field.” And one thought leader was said to have remarked that “the window of opportunity for social accountability will remain open only if we can surface evidence that social accountability is worthy of continued support.”
There are literally a dozen evidence reviews of the TPA sector which refute this claim (we have read them, alongside hundreds of evaluations and studies). Evidence is certainly mixed, but it’s hardly absent. Part of the fear expressed recently is about heterogeneity. This is a nightmare for anyone that seeks to use evaluations to find generalisability from interventions about complex TPA processes. Many impact evaluators have opted to reduce interventions to a single tool, omitting too many components of the work, seeking findings about the “average beneficiary” that are universally valid and hold in all contexts. In the TPA sector, variation in outcomes in different contexts and sectors is something to be expected, not feared. We regularly assert that “context matters,” and yet we forget this when it actually matters. As Howard White and Edoardo Masset from the Centre for Excellence for Development Impact and Learning (CEDIL) highlight, we should focus on transferability — findings that tell us what contextual factors condition (or not) the transfer of a finding from one setting to another.
On balance, if you read the evidence reviews in the sector, the message is generally positive. A Qualitative Comparative Analysis (QCA) of the UK’s former Department for International Development’s (DFID) Empowerment and Accountability portfolio found positive results across nearly all 50 projects reviewed in 2016 (prior to the sector’s apparent fall from grace). But, this was largely ignored — perhaps because it wasn’t an RCT. Other groundbreaking reviews in the sector using realist methods which present an array of outcomes and take context into account in particular sectors were also ignored. Either there is collective amnesia, a selective reading of the evidence, or experts’ expectations of “worthiness” may be rather too elevated.
As Peter Evans of the UK Foreign, Commonwealth and Development Office (FCDO) explains, evidence reviews have their flaws. We would argue that many of them have unwarranted methodological biases and some make grand arguments without much empirical evidence. Evans is also right that “no-one ever opens an evidence review and finds the perfect answer to their question.” But, when evidence reviews don’t quite cover it, that doesn’t mean that we should resign ourselves to the wisdom of a few researchers’ hot takes, loud voices in our echo chambers, or give undue credence to a handful of expensive impact evaluations.
The supposed “existential threat” is not primarily empirical, but semantic and discursive.
The question for us remains — how can portfolio-level M&E in the TPA sector build a more inspiring narrative to help make the case to continue investing in the collection of evidence of TPA’s impacts over the medium to long term?
In our second blog post, we start answering this question. We share insights from our work as M&E consultants working with different portfolios and connecting the dots across projects and portfolios.