New Paper: Relational Rubrics for MEL Complex Processes.

[Florencia Guerzovich]

Dear All, 

A key challenge amongst us Monitoring, Evaluation & Learning (MEL) folks is assessing whether and how the scale-up of complex interventions contribute to long-term sustainability and actual system change. Without a crystal ball of the future, we only have a partial view when monitoring and evaluating programs. And yet, we are asked to make meaningful assessments about the sustainability of benefits, including scale-up.

In a new paper, published by the Global Partnership for Social Accountability/The World Bank, Alix Wadeson and I develop and test a prospective and  relational rubric to square this circle. 

Paper  https://documents.worldbank.org/en/publication/documents-reports/documentdetail/099248202082451403/idu143be23531a0f714f561b91515c596de86102 

Blog Post (including CODA)  https://medium.com/@florcig/addressing-the-evaluation-of-sustainability-paradox-a-relational-rubric-for-evidencing-001683a36398

Florencia 

Florencia Guerzovich, PhD

Independent Consultant 

https://www.linkedin.com/in/m-florencia-guerzovich-9b3ab74/

[Jindra Cekan]

Florencia, It’s interesting to see this as a prospective study and certainly I agree. Relationships are key as work on exit and expost (yes!) that I and others have done show this.

RecentlyI took 12 years of research on 91 expost evaluations (

https://valuingvoices.com/catalysts-2/

) +2 excellent exit studies that I didn’t see referenced in your paper (INTRAC and CDA). I produced nine checklists based on research on actual expost 2 to 30 years post-closure. More here:

https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/875

Cheers, 

Jindra čekan/ová, PhD

Valuing Voices: Sustainable Solutions for Excellent Impacts

and Development Finance + Development Aid for Sustained Impacts

CEKAN CONSULTING LLC

www.ValuingVoices.com

www.LesyCekanova.cz 

www.Jindracekan.com 

President/ Catalyst

"This beautiful, bounteous, life-giving planet we call Earth has given birth to each one of us, and each one of us carries the Earth within every cell of our body.... 

We need to change our way of thinking and seeing things. We need to realize that the Earth is not just our environment. The Earth is not something outside of us. Breathing with mindfulness and contemplating your body, you realize that you are the Earth. "

― Thích Nhất Hạnh, Zen Master, on the Climate 

--
You received this message because you are subscribed to the Google Groups "Adaptive Development | #AdaptDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adaptdev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adaptdev/CAHdRuNgzzwCz9tpgXwGkYRyWhxYq8agch2LS_KKvFB0c%3DJTMjA%40mail.gmail.com.

[Florencia Guerzovich]

Dear Jindra,

 

Many thanks for your interest in our paper! It is great to know that work coming from a different angle (ex post  and exit evaluations vs. development of rubrics to complete/retrofit a complex portfolio's results framework) reach similar substantive conclusions. We believe that it is this substantive point that should shine through, especially for colleagues who grapple with MEL systems and resources that don't fit with their realities and practices. Whether in WB social accountability program (our sample in this case) or in the different universe of projects that your work covers.

 

Lastly, we should note that we began writing our paper a long time ago  and finished it  well over a year ago. The necessary but lengthy process for World Bank's expert peer review, editorial, design, communications and multi-level approvals, meant that the time of publication in early 2024 does not accurately reflect the actual timing of evidence review, writing and the window to include other relevant evidence. Your paper was published in 2023, which was after we had submitted our paper and that process had commenced.  

 

We look forward to collaboration with passionate MEL folks like yourself to hopefully trigger a robust conversation in this list and elsewhere about practical ways to close the gap between existing and emerging MEL methods and practice in order to make the most effective use of the different tools in our collective toolboxes.

Thanks again for your interest and reaching out.

 

Warm regards,

 

Florencia and Alix

[Jindra Cekan]

Thanks for your nice email Floriencia. 

FYI the exit studies existed back in the 1990s and early 2000s but that's the problem with our rich-in-knowledg but silo'd esectors, so much learning out there, but not knowing about what the other silos are doing like I knew little of your references too...

FYI have you found any ex-posts at the Bank that had fieldwork? Our research in the publicly available Bank ex-posts were not encouraging: https://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-the-world-bank/

but at least many of the indications of likely sustainability could be found, even if not verified: 

https://valuingvoices.com/can-we-assume-sustained-impact-verifying-the-sustainability-of-climate-change-mitigation-results-reposting-a-book-chapter/

Happy to keep connected, 

Jindra

[alixwadeso...@gmail.com]

Hi Jindra,

 

Thanks for your interest and exchange on the paper/blog! I am the co-author jumping in to respond to your question on ex-posts for WB. In our time/contracts with Global Partnership for Social Accountability (GPSA) program of the Bank, we didn’t come across any ex-posts. That said, we focus solely on the GPSA program which is a much smaller scope and range of documents compared to the entirety of the Bank’s programming! So, we cannot speak to that other massive/majority of the Bank’s work. Hope that helps with your question though!

 

Otherwise, my personal experience as an evaluator is that I personally haven’t had or even seen many opportunities advertised for ex-post evaluations within the range of NGO/civil society and funders that I engage – mostly in the broad sectors of humanitarian aid and international development. There is one exception of an evaluation I conducted in 2022 for the International Institute for Environment and Development (IIED) which is published here and a blog I wrote with my co-evaluator here on my Medium account.

 

It was a very interesting and unique opportunity, so I hope to have more chances to do ex-posts! I agree they are very valuable for learning and offer insights that other types of evaluations do not have the scope or advantage of time passed to be able to address. These opportunities have been limited, at least within my field of focus at least, for many reasons including very constrained budgets and timelines of programming and funder interests and requirements. Not sure if that echoes other MEL folks experience in the sectors I work and/or others though.

 

Anyway, thanks again!

 

Warmly,

Alix

To view this discussion on the web visit https://groups.google.com/d/msgid/adaptdev/CAKPUvB%2BBxz13CRJ-xYWvuwBLi1mS8f0973i6CDnqaAxPn%3DUOGw%40mail.gmail.com.

[Jindra Cekan]

Thanks a lot Alix for following up. 

Interesting about the ripple effect of research grants.

We have found 91 ex-posts where evaluators went to ask participants and partners (not just desk studies which I've often found at the Bank) tho' JICA is amazingly good too.

Warmly, Jindra