Autores: Alix Wadeson, Florencia Guerzovich e Thomas Aston

As we support different organizations, think about their MEL systems, and reflect on our own work, we’re finding that we can measure progress and aggregate results to conduct a symphony that is larger than the sum of its ‘instruments’ (continuing with our ‘school of music’ metaphor). In our last post of this blog series, we discussed the importance of engaging with programs and portfolios in place, while working towards a new framework by incorporating “layers” of learning from experimentation, both at project and portfolio levels. We also learned about similarities with the thinking of other colleagues, such as the UNDP’s Accelerator Labs and Innovation teams. Our understanding is slowly evolving, but we are already learning various lessons; we hope these can inform others also working on MEL or managing ‘schools of music’, and continue this dialogue with peers and funders.

We have organized this learning and current thinking into five key areas: (i) construct validity and comparisons; (ii) collective and meaningful theory building; (iii) realistic timelines for realizing and measuring impact; (iv) purposeful and appropriate data collection and aggregation; and (v) MEL utility for all.

1) Are we measuring what we think we are measuring? And how can we compare and learn across projects and contexts?

We recommend starting by defining standard concepts, and indicators with specific guidance on what is (and is not) important to document at the project and portfolio level. This decision is often associated with a theory of change (ToC). There is some recent debate about the language, but it’s essentially a hypothesis of how and why change is expected to happen (or happened). We should also consider the main questions that stakeholders have about this theory (or theories), rather than generate concepts or indicators for their own sake.

Two of us are developing and testing a Monitoring, Evaluation, Reporting and Learning (MERL) guide for GPSA grant partners and evaluation consultants. This process has helped us reflect on common challenges in establishing other TPA portfolio MEL systems. For us, this process includes unpacking the key indicators that could prove useful to help us monitor, evaluate, iterate, and narrate a general trajectory of change across different contexts.

Also baked into this process is a set of core assumptions that the ‘school of music’ identified with its partners over the years. This included compromises for prioritizing certain assumptions over others, across the portfolio (e.g., zeroing in on the non-financial contribution of the funder as part of the effort rather than only on the work of local groups; or focusing on the development of relationships for meaningful action as opposed to focusing on the design of tools).

A set of core concepts is also embedded here, for example, ‘sustainability’. The GPSA stakeholders defined both old and new concepts together, taking into account emergent practice as well as research in the sector. It was important to be clear about their definitions to ensure that stakeholders understand what exactly the portfolio intends to measure about and learn about collectively, before getting into the ‘how’.

Another example is “capacity building”, quite a generic term, but also one that is central to the GPSA’s work (and also for many other TPA projects). Therefore, the GPSA observed its practice and developed its own framework to define types of necessary capacities for effective social accountability processes. It’s important we are clear about what these key concepts mean for a given portfolio or program.

Such definitions are important to support the potential transfer of key ideas to different project contexts more easily, and to ensure we are monitoring similar dynamics (i.e., construct validity). These are also important to help avoid situations where actors are (perhaps unknowingly) talking past each other when reflecting about the work — e.g., equating capacity development to the top-down transfer of expert knowledge as opposed to the development of capacities to learn by doing with and from others. USAID also just launched a document explaining what they mean by locally-led development. So, we aren’t alone in this endeavor.

Definitions can be a purposive tool for a living process that enables us to find compromises to produce harmony among the different components of a ‘school of music’ (see below). Their codification is just a step to support MEL that can be revisited over time, in light of emergent practice. In the TPA sector, this isn’t meant to prescribe or proscribe, but rather, to ensure we’re talking about the same things, so we can also work to measure and learn about them consistently. We find that there are no perfect definitions and concepts can evolve with learning and practice. Therefore we are striving for ‘good enough’, while also being careful about conceptual stretching.

A ToC can be a useful instrument to prioritize concepts and indicators. But a portfolio-level ToC is often written in a way that does not speak to the specificities of concrete projects (see below). This is why it is useful to “localize” the ToC and associated Results Framework (RF) indicators into project-level ones early on. Coherent nested ToCs; defined core concepts; a priority set of indicators; and a general approach to coding and scoring them might help us learn about key features of a portfolio/program.

It’s also equally critical to be explicit about what is not necessary to measure. We believe all stakeholders should avoid the urge to add too many indicators, and to add ‘extras’ with caution. The aim here is twofold:

a) focus attention on a manageable number of priority areas (theory, questions, indicators, learning activities) for the different stakeholders of a given ‘school of music’; and

b) avoid projects with too many indicators, while offering limited value added to your priority areas. The latter seems to be a pervasive problem in TPA projects that we know. The GPSA team reviewed 1,000s of funding applications. Over the years, one of us curated MEL jam sessions with grant partners. Developing fit-for-purpose TPA indicators which can provide the meaningful evidence we seek is rarely easy and can contribute to ‘overdoing it’ on the indicators. However, we suspect that if agencies really measured all the indicators identified in their proposals, sometimes over 70, they would not have time and money left to do the actual work! It can also dilute the quality of indicator measurement across the board, when the resources are spread too thin in this way (particularly with small teams).

This may sound like cause for concern, a risk of putting diverse grants into a “MEL straitjacket,” promoting check-box isomorphic mimicry, or promoting undue standardization. This is not our intent. So, to allow for diversity, one option is to use “functional equivalents,” an approach used in comparative social science and law as well as in the TPA field (e.g., in OECD Anti-bribery convention and Global Integrity’s legendary reports and indicators). In practical terms, this is about determining the function of key aspects of TPA projects, rather than focusing on the form (i.e., name or label).

For example, many TPA portfolios seek to bring actors from civil society, public sector, and citizens to engage collaboratively on joint problem solving to address specific service delivery or policy failure problems. These processes can take many forms with different labels — from school-based management bodies to community health committees to higher level engagement structures between policy makers and CSO coalitions. They can be formal or informal, rigid or loose.

The importance is the function that they play and that there are the appropriate group of stakeholders engaged in them to effectively address the identified problems. If these processes are critical to the joint purpose of a ‘school of music’, then each member should also ideally track an indicator focused on the activities and engagement of multi-stakeholder compacts, platforms or interfaces. However, it should be up to each member to determine what that looks like and how that works in practice (i.e., the functional equivalent). Our preference is a balance that promotes localization and appropriate contextual fit but does not simply propose that a thousand flowers (and indicators) bloom wild; this can lead to cacophony and may well be counterproductive to collective learning over the medium and long-term.

We suspect that the use of functional equivalent indicators may enable better comparison of processes with similar aims (they are part of the same overall program/portfolio) but that are not just copycats (i.e., they represent locally-led, contextually-tailored versions of the collective ‘music’). We hope this will help us to learn more about what works and does not, under which conditions. For example, how and why different contextual factors can sometimes lead to the same results or alternatively, how seemingly similar situations produce different results.

Over time, you can build a useful map of the portfolio, which can help to identify opportunities for structured comparisons. Ad-hoc case selection strategies, so common in TPA evaluations, can work well for communications, but less so for transparent, adaptable and accountable ‘schools of music’. They also get in the way of social learning, in the sense of grappling together with what we know and the uncertainty of the work.

Organizations should be prepared for course correction on the choice of core concepts and indicators until they reach the ‘Goldilocks’ balance.

We recognize that there’s a risk of too much standardization, as practitioners know well. However, we have seen in some TPA portfolios and programs that not having enough relevant data in many cases or datasets that cannot be meaningfully compared. It may take time to get the mix of concepts and indicators ‘right’, but it’s worth the effort in our view.

From a MEL practitioner perspective, the focus should be balance and compromise. Unrealistic zero-sum debates across extremes do not help us to move forward nor help us to ‘learn by doing’. As Kathy Bain picked up from our previous posts, “If we cannot support the scale up and learning from the many disparate but rather small scale success stories we all know about, we are falling short. Candid discussions and more purposeful experimentation on how best to do this, while learning from each other, is urgently needed”.

2) Build theory collectively, yet with boundaries

Nested mid-level theories of change (with boundaries) can help provide focus and build political compromises. We have shared some of our experiences on the benefits of mid-level theory for field and portfolio MEL. Mapping assumptions can help prioritize change pathways within a portfolio. Being explicit can help us interrogate the validity of these assumptions, as well as recognize other pathways that co-exist beyond our portfolio (i.e., other genres of music that exist in other schools of music and may be in harmony or discord with our work).

In this way, when we talk theory, we are not thinking about only lobbying for our preferred musical genre as universal, but the possible benefits of alternative paths and what their tradeoffs might be given organizational and contextual circumstances. When portfolio ToCs are only made with strong normative assumptions for advocacy, fundraising or other objectives, they may inadvertently undermine the quality and effectiveness of MEL, reinforcing our discursive “existential threat.”

We also advise to explicitly ask about the funder or fund manager’s contribution and comparative advantage: Even when ‘schools of music’ increasingly let ‘local musicians’ lead, there is much to learn from their common thread. We need to learn about whether they add value to the symphony or the transaction costs, organizational dynamics and/or other factors turn lofty goals into a symphony or a cacophony. If we inquire rather than assume local partner coordination (i.e. an orchestra) and a funder or fund manager’s role in it, we can learn how to better support the work.

For example, the Fund for Transparent Slovakia’s (FpTS) evaluation found that incentivizing joint projects among NGO partners did not pay off in their context, but supplementing grants with informal dialogues, including but not limited to partners, added value to the system (see p.61 here for a glimpse). However, it is important to note that much of the FpTS administrators’ staff time, responsible for the fund’s value-add beyond direct grants, was not covered by its administrative costs.

Clarifying a funder’s role in a given sector or context, where there are other actors trying different approaches, often also requires asking: what is our unique contribution to funding change within a system? Also, what ‘musicians’ and ‘musical genres’ are well suited to support and which ones are a better fit for another’s niche or specialty? In the case of the FpTS that means grappling with the opportunities and constraints of working with funders from the local private sector. For the GPSA, the institutional home at the World Bank cannot be overlooked. For many others, those opportunities and constraints will be shaped by the link with a government’s foreign policy, a founder, management and/or a board member, whose influence practitioners working at portfolio level know well.

3) Identify an appropriate time horizon for impact (and be realistic!)

Target conscientiously: Looping back to our first post reflecting on the feedback that the Hewlett Foundation received on its strategy: Telling a narrative of progress is about showing results that stretch us collectively, but does not ‘throw the baby out with the bathwater’. We recognize that TPA often is messy and takes patient investment, but the process can lead to success if funders and other stakeholders keep at it together, as Louise Cord, the Global Director for Social Sustainability and Inclusion at the World Bank, put it.

For example, in the short- and medium term, you can set targets for the journey that are doable but not easy to achieve (e.g., other actors’ uptake and adaptation of interventions’ lessons — i.e., embeddedness) rather than expect unrealistic ones (e.g. wholesale scale-up through the adoption and implementation of an intervention exactly as one designed it). This way we may avoid feeding the discursive existential crisis of the next decade.

Connect MEL across strategy cycles: We often say that TPA work is the story of a marathon, rather than a sprint. So, we can see it as a type of relay race between the conductors in a school of music. Strategy cycles start and end, often informed by path dependence and, hopefully, learning from predecessors. The challenge is that we often forget to talk about the baton passed between the conductors in this relay race.

Evidence suggests that scaling up innovations takes a decade and translating policy change to implementation tends to take at least five years. So, it’s not realistic to expect high-level impact for communities within only a couple of years, as is often expected. For this reason, we should seek measurement of impact and progress in 5, 10, and even 20-year periods. For the medium-term, we could use more in-depth reflections of iterations across those cycles — did the interpretations of the lessons from the last funding cycle that inform our current actions hold up over time, or not? For example, the Partnership to Engage, Reform and Learn (PERL) program is giving a series of webinars in October 2021 to share lessons from 20 years of different UK governance programs in Nigeria.

Similarly, the World Bank’s approach to social accountability in the Dominican Republic since the 1990s finds that local and global contextual shifts, and TPA and sectoral lessons, all informed changes in the Bank’s approach in country and elsewhere. All too often, those long-term histories are reserved for a selected few, doing the reflections, having the benefit of inter-generational reflection or learning from their predecessors, and/or shifting strategies from one approach to the next. In other words, we could use more comparisons to tell the story of these relays and the learning generated across strategy cycles over time.

4) Gather data purposefully and aggregate appropriately

Build filters before gathering data: The challenges of monitoring a portfolio’s work relate to its scope – the sheer volume of information and transaction costs associated with working with so many people and actors. The challenge is not so much around generating information as organizing it, constructing filters, and developing the systems to apply them so that the right information and indicators are available and consistently applied across project grants. As Clay Shirky (in Juskalian, 2008) asserts, “Without clear guidance, long qualitative narratives may be so variable that analysis, particularly comparative analyses, becomes extremely difficult”.

Aggregate data appropriately: An additional challenge with the portfolio structure may be demands for inappropriate aggregation, combining dissimilar investments, projects or outcomes to present several diverse projects in a simplified narrative. This is a challenge for TPA professionals as many of us have faced requests to aggregate index scores (e.g., civic space) and results, or perhaps find ways to add contributions to women’s reproductive health and sanitation — without considering whether we are adding apples and oranges and/or ignoring interaction (negative) effects. An over-simplified approach risks ignoring important differences and nuances that may be valuable for learning, while ignoring contextual factors and over-generalizing.

The use of a set of functional equivalent indicators with the same units of measurement, based on common conceptual definitions within a portfolio of grants, can ease aggregation processes downstream. That is, the approach may help us to transfer meaningful, targeted information from implementers to managers, and subsequently use this knowledge to inform higher-level decision making and governance structures (e.g., Boards; Steering Committees; funders; and public officials funding specific civil society work).

5) Strive for utility to all: implementers, funders and the field as a whole

Construct a MEL system that works for implementation: There is a craft to portfolio management, including its MEL. It’s hard work and requires financial and management support. Failing to purposefully invest in sufficiently resourced MEL systems can backfire. It feeds and potentially deepens the field’s supposed “existential crisis”. We know (and experience ourselves) that organizations that have very limited human capacity, financial resources, short time horizons. Therefore the political space may find some of these examples unhelpful. We believe that it’s important to design MEL systems that are possible to actually implement, which also means making well-informed trade-offs and compromises.

In our MEL choices, we prioritize questions and concrete contextual features. Prioritization, given scarce resources, entails trade-offs, and short, medium and long-term risks which we can manage or sweep under the rug. We learned that we should prioritize collaboratively and do internal advocacy to open space to create, course-correct and sustain the implementation of compromised solutions over time that travel across decision-making levels of the portfolio. This is often difficult because we are working with organizational restrictions, limited resources, technical criteria and, often, shifting politics within organizations and the systems in which they work.

In the face of these challenges, we should be transparent and manage risk, rather than set ourselves up for failure with unrealistic expectations. With an eye towards the portfolio-level narrative, funders and intermediary staff can help to frame and/or co-produce with partners, strategic questions that are broadly relevant to most stakeholders. They can also help us to identify lessons that may be applicable and could be tracked across multiple interventions to tell a collective story.

Prioritize focus areas for monitoring and learning, with key evaluation questions to apply across the portfolio (i.e., create filters and frames): As Al Kags argues, “The question of fostering active citizenship and indeed responsive government, is a complex one with a kaleidoscope of nuances. It is more like a set of puzzles, each of which have layers of contexts.”

For example, how do we tackle common challenges such as increasing the likelihood of scale up of TPA work? These are often areas of interest for many actors across the portfolio — from project managers to funders. The documentation of these common areas isn’t detailed enough for implementers, but it can help them identify counterparts within a ‘school of music’ from whom they can learn and with whom they might collaborate and go deeper into common areas of interest. Then one can assess whether setting a top-down common approach, method, or broader parameters to answer the question has more advantages or disadvantages than letting each team define the methodology on its own.

Coda

For now, we’re pleased with the thoughtful dialogue generated so far on rethinking the TPA sector’s narratives of success and failure and the role that funders and other portfolio managers might play, as they design and implement their own MEL systems, policies and practices. Others are engaging in this discussion offering their own perspectives. We recently conducted a webinar on building mid-level theory. Participants showed a surprising level of appetite for a candid conversation about how to build this in practice for the TPA field for organizations of different sizes and types. In a recent webinar on the future of anticorruption work convened by USAID’s Achraf Aouadi (I-Watch Tunisia) put the issues (from his perspective) on the table, as did Ambassador Power (from hers). So, this isn’t just the view of three consultants. Rather than dismiss TPA portfolios as too ‘hard to measure’, let’s rise to the challenge and learn from each other. We encourage others to join in the discussion and let us know how you are managing these issues (and others) in your TPA programs.

We pose a polite challenge to funders out there: in addition to investing in your improved portfolio MEL systems in this new strategy cycle, you can also help the field by supporting relevant TPA actors to “play their MEL music together”, to facilitate collective strategic thinking and exchanging of tricks of the trade.

Autores: Tom Aston, Florencia Guerzovich and Alix Wadeson

As we illustrated in the previous blog in this series, funders, fund manager organisations and implementing organisations in the Transparency, Participation and Accountability (TPA) sector are wrestling with the challenge to move beyond piecemeal project-level MEL to evidencing more cohesive programme and portfolio-level results which are greater than the sum of their parts. This is the holy grail, as Sam Waldock of the UK Foreign, Commonwealth and Development Office (FCDO) put it in the Twitter discussion.

Some grounds for optimism

While we agree that portfolio MEL is challenging, it’s not impossible. Some efforts in the wider international development sector give us reasons to be optimistic when there is political commitment at the right level. As CARE International’s Ximena Echeverria and Jay Goulden explain, it’s possible to demonstrate contributions to the Sustainable Development Goals (SDGs) across an organisation with over 1,000 projects per year. With standard indicators, capable staff, and serious effort, you can assess progress at a considerable scale. But there are also ways to break such enormous portfolios down into more manageable chunks.

One of us conducted a review of CARE’s advocacy and influencing across 31 initiatives (sampled from 208 initiatives overall) that were relevant to just one of CARE’s 25 global indicators up to 2020. This was based on a Reporting Tool adapted from Outcome Harvesting (a method which is increasingly popular in the TPA sector). CARE has continued to assess this in subsequent years, looking at 89 cases last year because it saw the value in the exercise. As advocacy and influencing constitutes roughly half of their impact globally, it’s obviously worth evaluating and assessing whether trends of what worked changed over time. Oxfam was also able to do something similar for its advocacy and influencing work, building off 24 effectiveness reviews (which relied on a Process Tracing Protocol; if you’ve read some of our other blogs, you’ll know we’re fans of this method).

Both reviews, also similar to the Department for International Development’s (DFID) empowerment and accountability portfolio review of 50 projects, were Qualitative Comparative Analyses (QCA) — or fuzzy-set QCA. We believe that QCA is a helpful approach to find potential necessary and/or sufficient conditions, but such conditions are not always forthcoming (as DFID’s review showed); they also rely heavily on the quality of within-case evidence. We’re often searching for, but not quite finding, necessary and/or sufficient conditions. For this reason, there are limits to what QCA can do, and without adequate theory it can sometimes be premature.

Realist syntheses (or realist-informed syntheses) also provide a helpful option to assess what worked for whom, where and how. The “for whom” question should be particularly of interest for the Hewlett Foundation’s new strategy refresh as well as to all colleagues who strive to design portfolio MEL systems that are useful to its different stakeholders and decision making needs. One great benefit of the approach is that it assumes contingent pathways to change (i.e., diverse mechanisms), emphasizing that context is a fundamental part of that change rather than something to be explained away. A more contingent realist perspective thus builds in a limited range of application for interventions (x will work under y conditions) rather than assuming the same tool, method, or strategy will work the same everywhere (the fatally flawed assumption that brought the sector to crisis point).

The Movement for Community-led Development (MCLD) faced a similar “existential threat” as the TPA sector — after another hot debate about the mixed results of Community Driven Development (CDD) in evaluations. Colleagues from 70 INGOs and hundreds of local CSOs from around the world, started a collaborative research including a rapid realist review of 56 programmes, to understand the principles, processes, and impact of their work. They counted on leadership and some external funding, but often relied on voluntary time and good will. The process and the results include several relevant overlaps and insights for monitoring and evaluating portfolios of TPA work. The full report is due to be published on the 6th October, 2021.

As observers and participants in the MCLD process, we would like to underscore a factor that distinguishes this effort from many others — don’t ignore the “L” (learning). Regular calls among the group were made to engage in a kind of “social learning,” resembling a learning endeavour which is described by the Wenger-Trayners as an ongoing process which creates a space for actors to mutually exchange and negotiate at the edge of their knowledge, reflecting on what is known from practice as well as engaging with issues of uncertainty.

In the TPA space, where practice often entails advocacy grounded in expertise and/or values, including to set research and learning agendas, creating this learning space requires, at a minimum, changing the mindset and challenging normative assumptions and apparent agreements that constrain social learning. For another call to put “L” front and centre, see Alan Hudson’s feedback to the Hewlett Foundation’s strategy refresh.

Therefore, with sufficient political commitment and flexibility, the appropriate methods and processes are out there to assess and support social learning in as few as 5 to as many as 1,000 cases or initiatives. But there also seems to be a more meaningful sweet spot somewhere between assessing a group of 5 and 50 initiatives.

How can portfolio evaluation contribute useful knowledge to the field ? The GPSA’s journey

We’ll discuss an example of a funder the three of us know well (but do not speak for), one that has been unusually open about its complex MEL Journey. The Global Partnership for Social Accountability (GPSA) started grant-making in 2013; by 2014 it had already designed 20 flexible grants tackling a broad range of development challenges (health, education, public financial management, etc.) in diverse contexts — from Mongolia and Tajikistan to Ghana and the Dominican Republic. The common thread for these grants was that, a priori, they were all cases that it was judged would benefit from politically smart collaboration between public sector and civil society actors. The prioritisation of a problem and the precise approach and tools to tackle challenges themselves was localised.

In its infancy, the GPSA lacked a comprehensive MEL system. The programme prioritized grant partners’ ability to determine on their own appropriate approaches to MEL and aggregation of results was less pressing. “Learning by doing” is something of a mantra at the GPSA whereby both the GPSA and its grant partners are expected to embrace this approach. It encourages (uncommon levels of) openness about failures and allows ample room for iteration of collaborative social accountability processes undertaken by diverse grant partners across over 30 countries (to date). To pick up Bowman et al.’s typology, by 2016, MEL was a full-blown “jam session” among grant partners that resemble other decentralised systems (see the typology below).

Bowman, et al. 2016

One practical challenge was that the GPSA had questions about the aggregation of results. The onus, as affirmed by former World Bank President Jim Kim, was to show that this “miracle” has real outcomes. Delivering on this challenge could go in different ways. As elsewhere, finding and implementing a compromise among decision-makers needs across the system is easier said than done or resourced.

Over the years, the GPSA explored the implications of adaptive learning as a fundamental portfolio approach while responding to this demand, it intentionally experimented over time for gradual design, implementation, and course correction. A careful dance while fitting in World Bank rules, processes, and shifting approaches and norms.

A key part of this journey was about transforming the free flow ‘jam session’ into a more harmonious school of music. The latter is an entity that includes a wide range of activities (instruments) implemented by multiple, diverse grant partners (music school members) that have a loosely defined common theory of action to work in diverse contexts to tackle a range of development problems (different musical genres and sheet music). The GPSA, along with funding partners and World Bank management sector and country staff, makes initial choices in this regard — which is why strategy refresh season is important. They also provide financial and non-financial support and also can facilitate information exchange and compile activity reports for higher ups. Yet, the portfolio’s day-to-day work is often decentralised, and line-managed by people in other organisations (grant partners). The GPSA, like other funders or fund managers, often has limited authority to direct the behavior of partners — but it’s not innocuous and thus should not be omitted or obviated from the story.

The GPSA is building upon layers of ‘jam session’ lessons to shape a school of music-type portfolio that can start to play more harmoniously under the direction of a MEL system developed over time. This has been a multi-year process in which “L”, in the sense described above, was front and centre in the GPSA’s criteria for funding applications, annual reports, revised theory of action and indicators in the results framework. It’s also more explicitly emphasised in GPSA’s annual grant partner forums — GPSA alumni participate in these events which enables retrospective insights and facilitates that ongoing engagement and support across civil society can be maintained over the longer-term, beyond the duration of a grant. The GPSA also hosts other virtual events and bilateral conversations.

Connecting the dots across these MEL tools, in 2016, the GPSA began to articulate key concepts ‘for the practice from the practice,’ such as collaborative social accountability. The GPSA needed a new common language to talk about and understand the work — what looks like yet another label, was an effort to avoid cacophony. In 2018, the GPSA agreed to “test” key evaluation questions of interest to different stakeholders, with a small number of evaluations (Expert Grup took the plunge first). Then, the GPSA learned that the priority of these questions resonated with grant partners as well. In 2019, grant partners watched presentations of 4 evaluations and came back to one of us saying: “now I understand what you mean that an evaluation can be useful, I want one like that!” Part of the trick here is that those select evaluation questions were also useful to management, and a broader group of Global Partners.

Social learning within the school of music informed the next iteration of MEL tools, including refining the evaluation questions, and others that can support us to better grasp the elusive question of ‘what does sustainability and scale of social accountability look like in practice?’ All this MEL work and social learning with partners about practice precedes the current GPSA Manager, Jeff Thindwa’s, call for the field to take stock, match narratives to the work, and write the next chapter for social accountability. We agree that this kind of social learning is key to exploring TPA theories of action and change, develop tools to assess them, and hopefully will help us to challenge the doom and gloom narrative. In a slightly different guise, social learning underscores the importance that the principle of utilisation-focused evaluation should be a key priority (rather that its specific methodological incarnations, per se), which has been reasserted in recent evaluation discourse (for instance, as a central theme at the Canadian Evaluation Society’s 2021 annual conference).

In the next post of this series, we’ll share more practical insights about the design of MEL systems emerging from our work across different portfolios. These are insights that we believe are useful for different types of schools of music as well as for colleagues who privilege any of the methods discussed above and/or support the idea of a bricolaging them.