Experimental use of AI to summarise the description of use of MSC in published papers

[rick davies]

Hi all

As you may know, there is an online bibliography of papers that have been published and which describe the use of MSC, which can be see here: https://www.zotero.org/groups/266453/most_significant_change_technique/library

Often the description of how MSC was used is embeded within the description of wider processes and results. This can take time to read and extract

Today I made a trial attemtpt to get AI (Claude AI) to extract this type of information. I chose this recent paper: Akatukwasa C, Kemigisha E, Achen D, et al. (2023) Narratives of most significant change to explore experiences of caregivers in a caregiver-young adolescent sexual and reproductive health communication intervention in rural south-western Uganda. PLOS ONE 18(5). Public Library of Science: e0286319. DOI: 10.1371/journal.pone.0286319.

This is the text of the prompt I used

Please describe how MSC was used in this study. Include any evaluative judgements made about its advantages and disadvantages. Include any assessments of how it compares to other methods, including other ways of using MSC. Only use information available in this document. Not other sources

This is the response from Clause AI

Based on the information provided in the research article, here are the key points about how the Most Significant Change (MSC) technique was used and evaluated in this study:

• The purpose of using MSC was to explore the experiences of caregivers who participated in an intervention aimed at improving caregiver-young adolescent sexual and reproductive health communication. MSC was used as a qualitative evaluation method.
• The typical MSC process was adapted for this study. Four domains of change were pre-defined based on the project's objectives. The domains were: a) caregiver-adolescent SRH communication, b) caregiver knowledge and attitudes, c) parenting skills, d) personal life and family.
• 30 caregivers were selected to provide their stories of change. The stories were collected one month after the intervention using a narrative interview approach. The stories were audio-recorded, transcribed, and translated to English.
• Thematic analysis was used to analyze the stories and generate themes deductively from the pre-defined domains as well as inductively to identify emerging themes.
• The authors stated that MSC helped identify underlying and unexpected changes beyond the pre-defined domains of change. It allowed them to capture open and spontaneous expressions of the participants' experiences.
• Limitations acknowledged include: stories from 30 participants may not represent all participants, social desirability bias due to rapport building during the project, stories only capture short-term effects.
• No comparisons were made to other evaluation methods. The authors stated using MSC augments the quantitative results from the same study.

In summary, the key advantages highlighted were capturing open-ended experiences, identifying unexpected changes, and complementing quantitative data. The limitations were the small non-representative sample and potential biases. Overall, MSC provided qualitative insights into the intervention's impacts.

====================================

On reflection, for my next sumamrising trial I think the text of the prompt could be improved in a number of ways. But before I share those ideas, why dont you also make some suggestions.

There is also the possiblity of asking follow-on questions, seeking additional information about the paper that has just been sumarised. What follow-on questions do you think i should ask?

[rick davies]

One other advantage of these AI summaries is that they will appear in the online bibliography, under the Notes tab. Whereas it is not possible to share the original pdf in publicly accessible versions of Zotero libraries (for copyright reasons) 

regards, rick

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites:  https://richardjdavies.wordpress.com/ | https://parevo.org | https://www.mande.co.uk | Twitter: @MandE_NEWS | rick....@gmail.com Skype: rickjdavies

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/c03ebc2c-809d-4a9e-bf3e-83231439d952n%40googlegroups.com.

[Rob Assels]

From my perspective, your original prompt gets at the heart of it. However, you asked, so I might be interested in knowing where the MSC data diverged from the quantitative and, if it did, how the divergence was interpreted. And by divergent, I don't mean unexpected changes; I'm thinking conflicting findings, not supplementary ones.

Rob Assels

Evaluation Manager

M 902.759.1878  |  rob.a...@researchns.ca 

            

---

From: mostsignificantchang...@googlegroups.com <mostsignificantchang...@googlegroups.com> on behalf of rick davies <rick....@gmail.com>
Sent: August 16, 2023 9:14 AM
To: MostSignificantChange (MSC) email list <mostsignificantchang...@googlegroups.com>
Subject: [MostSignificantChange] Experimental use of AI to summarise the description of use of MSC in published papers

 

--

[Fiona Kotvojs]

Hi Rick,

 

To me, it looks really impressive as a summary. The question is how accurate a reader thinks it is – if it is good, then on a separate note I see a lot of applications for this in my evaluation work. How soon will it also replace me as an evaluator?

 

But, back to your question. As the evaluation findings aren’t the focus, just the methodology, I am interested that it didn’t pick up the MSC steps (which the AI doesn’t know). So from the summary, we can’t tell what was different about this evaluation (using MSC as the method) to a ‘normal’ qualitative analysis. So I think it may need specific direction around the steps to determine whether a panel was used (and the number of levels) and feedback provided (how and to whom). These are the two elements that differentiate MSC from other qualitative methods and they are not addressed. If they weren’t used and it is just a case of asking an ‘MSC question’, it isn’t MSC.

 

Thanks

 

Fiona

 

Dr Fiona Kotvojs GAICD
Monitoring and Evaluation Specialist
Kurrajong Hill Pty Ltd

 fi...@kurrajonghill.com.au
Phone: 0448 453 422

 

 

 

 

From: mostsignificantchang...@googlegroups.com <mostsignificantchang...@googlegroups.com> On Behalf Of rick davies
Sent: Wednesday, August 16, 2023 10:15 PM
To: MostSignificantChange (MSC) email list <mostsignificantchang...@googlegroups.com>
Subject: [MostSignificantChange] Experimental use of AI to summarise the description of use of MSC in published papers

 

Hi all

--

[Fiona Kotvojs]

Hi Paul,

 

I just tried the AI link on this to see how it went. As I am outside UK and USA, I can’t test it. I thought you may be interested.

 

Fiona

 

From: mostsignificantchang...@googlegroups.com <mostsignificantchang...@googlegroups.com> On Behalf Of rick davies
Sent: Wednesday, August 16, 2023 10:15 PM
To: MostSignificantChange (MSC) email list <mostsignificantchang...@googlegroups.com>
Subject: [MostSignificantChange] Experimental use of AI to summarise the description of use of MSC in published papers

 

Hi all

--

[rick davies]

Hi Fiona, and all

It's important to note that the task I set Claude AI was not to evaluate the use of MSC in that paper. Rather, it was to summarise the description of the use of MSC in that paper.  However that summary did, I think, include evaluative judgements made by the authors themselves

If I wanted the AI to evaluate the use of MSC in the paper I would have also uploaded a copy of the MSC guide and asked the AI to use that as a basis for evaluating the use of MSC in the reference paper. Of course, I could have provided other documents as well or in place of the MSC guide, so long as the total size of these documents was within the overall uploading limits.

It was interesting nevertheless to see that one of the comments by the AI, in the absence of any instruction to evaluate the use, was nevertheless evaluative. This was in respect to a summary of another paper by Sorcher et al, also in the bibliography. Here are two paragraphs from the AI's response at the end of that summary:

But it does not mention any further selection or filtering of the most significant changes reported by these participants. All the MSC data seems to have been coded and analyzed to identify themes.

The lack of a selection process means there was likely no prioritization or convergence on the most significant changes from the full set of stories collected. Without selection, the MSC implementation focused on capturing a breadth of perspectives rather than determining the most significant changes according to specified criteria.

I then responded Regarding this statement "The lack of a selection process means there was likely no prioritization or convergence on the most significant changes from the full set of stories collected." Is this a conclusion given or implied by the contents of the paper. or, something that can be concluded based on your other prior knowledge of the use of MSC

AI then responded You're right, that statement about the lack of prioritization or convergence goes beyond what is directly stated or implied in the paper. It reflects my own inference based on knowledge of how MSC is typically implemented, rather than being derived solely from the information provided in this particular paper. The paper does not explicitly state whether there was any prioritization or convergence from the MSC data collected. My statement about the likely lack of prioritization assumes that if selection had been done, the authors would have mentioned it, since selection is a standard part of MSC implementation. But that assumption goes beyond what is written in the paper itself. Thank you for pointing that out - I will be more careful about sticking to only what is directly presented in the source when summarizing research.

This was despite my initial prompt including the following sentence: Only use information available in this document. Not other sources

What was happening here was some form of leakage from the AI's wider pre-existing body of knowledge into the current task.  It may be to some degree inevitable, because the very ability to respond to my queries is dependent on the AI understanding what I'm saying, and there is a huge amount of training that is going on beforehand to enable that to happen, using a huge range of data sources.

The most immediate lesson is that you have to be continually wary about the content that is generated by the AI.  Not rejecting it out of hand but not accepting it as gospel either.  More practically, it implies the frequent need to ask follow-up questions.  I will talk more about this in another email in response to Rob's posting yesterday

regards, rick

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites:  https://richardjdavies.wordpress.com/ | https://parevo.org | https://www.mande.co.uk | Twitter: @MandE_NEWS | rick....@gmail.com Skype: rickjdavies

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/SYCPR01MB49925599361C80C663A3EAC59215A%40SYCPR01MB4992.ausprd01.prod.outlook.com.

[rick davies]

Hi Rob

I think your following question makes a lot of sense. There are two options here. One is to revise and expand the initial prompt to include sub- questions like this. The other is to devise follow-on questions which focus in on particular issues such as the one you have raised there

I have set up a Google document page here, with some ideas on initial prompts and follow up questions.  You and other readers would be most welcome to comment on these..

I will continue my experiments with other recently included references in the zotero bibliography.. 

You can now see the attached Notes, which include the full text of the Claude AI queries, on each of the following recently referenced papers (click on the paper name to make the Notes panel visible)

 Fernandes B, Neelakantan L, Shah H, et al. (2023) Evidencing the Impact of Web-Based Coproduction With Youth on Mental Health Research: Qualitative Findings From the MindKind Study. JMIR Public Health and Surveillance 9(1): e42963. DOI: 10.2196/42963.

Musser R (2023) Generative Beauty: How Engaging with Beauty Sparks Transformation. Honors Theses. Available at: https://scholarship.richmond.edu/honors-theses/1667.

Sorcher R, Cloete E, Salazar AD, et al. (2023) Life has become more colourful: the social and economic outcomes of community-led sanitation initiatives in the rural Philippines. Journal of Water, Sanitation and Hygiene for Development 13(6): 464–473. DOI: 10.2166/washdev.2023.042.

Wescott H, Ferri D and MacLachlan M (2023) Participation, Legal Capacity, and Gender: Reflections from the United Nations Partnership on the Rights of Persons with Disabilities Project in Serbia. Disabilities 3(1). 1. Multidisciplinary Digital Publishing Institute: 129–146. DOI: 10.3390/disabilities3010010.

In two of these you will see there are separate notes for queries made with Claude AI and ChatGPT4.  I would be interested here peoples reflections on the differences between these summaries

regards, rick

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites:  https://richardjdavies.wordpress.com/ | https://parevo.org | https://www.mande.co.uk | Twitter: @MandE_NEWS | rick....@gmail.com Skype: rickjdavies

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/YQXPR01MB65899A2E94942DFB4468B8609815A%40YQXPR01MB6589.CANPRD01.PROD.OUTLOOK.COM.

[Jaqui Goldin]

Dear Rick

I would love to access the manual that you have produced but can’t find it on the link in Agnes Marsan’s email.

Could you send me a copy of the manual.

Best regards

Jaqui

[Engaged Scholarship Evaluation]

HI Rick,

 

I’m writing to say the same thing as Jaqui. May I please access the manual, too?

 

Thank you.

Annalisa

 

 

-- 

This is a tranquil mailbox. It is checked once in the morning and once in the afternoon. Breathe.

 

Annalisa L. Raymer, PhD

Global Development | Education | CALS | Cornell University

https://cals.cornell.edu/annalisa-raymer

 

Director, Community Learning and Service Partnerships, CLASP

Cornell’s Adult Education Program for Mutual Learning

CLASP phone: 607.255.7633

https://clasp.cornell.edu

 

"I never teach my students; I only attempt to provide the conditions in which they can learn." ― Albert Einstein

 

https://app.cheerity.com/yIXP5zvYx/main/p/06MOZBw2-

 

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/5BD781DD-6ABB-488B-908B-EEE849D97838%40gmail.com.