MSC for monitoring

[Jane Dowden]

Hello, 

Just to say thank you for this group. I have recently joined as I am interested in finding out more about Most Significant Change approach. I have not used MSC before. I have read some guidance and information elsewhere but I am still not sure whether the approach can be adapted for our needs. 

I found the following posts in this forum particularly useful in clarifying the principles of MSC and what should be prioritised/avoided:

Asking about negative change in MSC interviews 7/01/22

Seeking the most typical change 17/03/21

Quality of Evidence and MSC stories 27/11/20

I am particularly interested in using MSC for monitoring although we would use this to inform evaluation too. I have seen lots of examples of it being used for evaluation but not so much for monitoring.

Our current quantitative approach to measuring outcomes is not giving reliable results and although we are looking at developing a new quantitative framework, there are so many different pathways each with a different weighting to the different outcomes that a qualitative approach might be simpler and more useful. However, we are dealing with around 4500 beneficiaries each year. Each one is awarded a small grant. The programme runs on a rolling basis so these are fairly evenly spread across the year. Although the applications are made by support workers, because of the numbers involved (1000s) it isn't possible to provide training for them. We currently only have contact via the support workers not directly with beneficiaries although we might consider collecting feedback directly from beneficiaries in future.

• How important is training for story collectors? I think this might be the weakest link for us and might mean we can only use this approach for evaluation with trained collectors rather than for monitoring. Has anyone tried this approach with just written instructions for story collectors - did it work?
• If we have too many stories to use in a panel meeting, could stories be pre-sifted and a sample taken? What might the criteria be or should it be random?
• It is interesting to see ideas for both positive and negative change stories. What about stories of 'no change'? Do these come out too?

Thanks,

Jane Dowden

Impact Manager

St Martins Charity

[rick davies]

Hi Jane

I hope you get some interesting responses

Before then, I want to check if you are aware of these two resources:

1. The 2005 MSC Guide, available here  https://www.researchgate.net/publication/275409002_The_'Most_Significant_Change'_MSC_Technique_A_Guide_to_Its_Use

2. The Online bibliography of papers written about MSC, available here: https://www.zotero.org/groups/266453/most_significant_change_technique/library

Re Q 2, is there any sort of mediating structure of other people between the 1000s of support workers and yourself? If so, could they be involved in intermediary MSC selection processes

A random sampling of support workers would be one option. but given the numbers involved, I think you will still need some intermediary selection processes

Stories of "no change" dont officially exist :-) Things are changing all the time, just some are " more significant" than others, to the observers present. But I can understand why you would ask this, because the most important thing that happened sometimes is that expectations of change were not realised. I would be interested to hear what other say on this topic, do they include stories of no change? Actually, when I think about it, there is probably change going on, in the observers mnds, they are probably adjusting their expectations or understanding of the circumstances. I think that I what I would ask people to write about in those circumstances

regards, rick davies

[Jane Dowden]

Thanks Rick,

Yes, that first resource was the first thing read and was really helpful. I have found the bibliography and had a quick search for monitoring. I haven't read everything in it yet but certainly a great resource!

Thanks for your response. With regard to a mediating structure, there isn't one as part of the grants scheme but we have another programme which is about building local networks so I think we could find people in similar roles to the support workers who could be part of panels to choose stories and this would help to link the two programmes. I was thinking it would also be useful to include people with lived experience of homelessness on the panels too. 

For a bit of context, the grants are to help people access accommodation or prevent eviction (rent deposits, rent in advance, rent arrears etc). The support workers applying either work for 3rd sector advice services, local government or they work for housing associations which are often the landlord of the beneficiary. This makes it problematic to rely on them as the story collector because they often have their own vested interest. In addition, the relationships, context and types of support provided are all very different. For monitoring purposes we can't cover the cost of extra people to do story collection for every grant unless we only focused on a very small sample.

I was thinking we could have two levels of decision making (e.g. three panels which reduce 27 stories to 9 them a fourth panel which chooses 3 from these nine). If we did this every quarter which I think is just about feasible in terms of time and resource, we'd need to find a way to select 27 stories from around 1000 grants. I can see how we could use this for evaluation but to make it work for monitoring I need to be able to convince managers and our funders that we are monitoring the outcomes effectively. Is 27/1000 enough? Is there a way of using a 3rd party story collector or someone from our team to collect 27 stories directly from beneficiaries every quarter? I think we need to pilot it and see how it works first. But I also I want to be sure I have understood the method correctly. 

Due to our funding structure there isn't much pressure to measure outcomes or report in a particular way so that means there is scope to take a more experimental approach to monitoring outcomes but it needs to be convincing. 

We are currently getting 60% return on the qualitative outcomes monitoring but we know from recent evaluation that the reliability and accuracy of this data is likely to be low. The monitoring data comes from a feedback form completed by the support worker, supposedly in conversation with the individual but that is not always the case. Completion rate and accuracy of outcomes data is something that we are not alone in the sector in struggling with. 

I think we might decide to try MSC as a small scale pilot using trained evaluators as story collectors and then see whether it is feasible to apply the approach to monitoring our grants or whether we only use it for evaluation. 

I'd welcome any feedback, advice or critique of this plan.

Thanks 

Jane

[rick davies]

Hi Jane

Wow...lots of interetsing issues here

1. Piloting is almost always a good idea

2. However you organise the selection process, think carefully about whose voices you want to hear, via the selection process. You alerady seem to be doing that 

3. The proposed structure of 3 bottom panels (9>3, 9>3, 9>3) and then a top panel (of 9>3) seems workable. I am assuming the bottom panels would all have the same type of members?

4. Random samply of 27 from 1000 _might _ be okay of the population was either very homogenous or very diverse. But if it is lumpy i.e has different groups of people a different approach might be better. Can you classify the 1000 into any meaningful groups, whose distinct voices you would like to hear? Then could you prioritise these groups into those whose voices you might like to hear sooner versus late for some reason? Could you sample a different group each different monitoring period?

Re sorting people into groups, i am a fan of participatory card/pile sorting methods.. see more here https://mande.co.uk/special-issues/hierarchical-card-sorting-hcs/  and here https://mande.co.uk/special-issues/hierarchical-card-sorting-hcs/#resources

regards, rick

[Fiona Kotvojs]

Hi Jane,

 

Sorry for the delay in my comments on your questions.

 

I have used MSC for almost 20 years in a wide range of situations. I prefer to use it for monitoring as it has a real strength there. From my perspective, as a monitoring tool, MSC provides an ‘early warning’ system for negatives – emerging issues. When I ask what good and bad changes have occurred (and I specifically say “good and bad”) people will identify emerging problems we need to know and address before they become a problem. It also helps identify: stakeholder groups that aren’t getting benefit and we can refocus work to address; unexpected changes that may provide opportunities to improve achievement of outcomes; and most importantly for me, helps promote change – as a monitoring tool it supports change management. This is a real positive. If I am not using it as a change management tool, I don’t use it for monitoring. Instead I simply ask the same questions and don’t conduct a panel – then it is not MSC – it is a semi structured interview.

 

From my experience, the story collectors must be trained; without training you are likely to get data that can’t really be used or isn’t good quality. It is like any form of evaluation – quality data underpins everything -  rubbish in, rubbish out. I would also say that simply asking people to provide a written response to a series of questions doesn’t provide quality data (stories). I have tried several times and no longer do – it has been a waste each time.

 

Start small, you don’t need to interview everyone. Qualitative is looking for richness in the data. Quantitative is looking for (put simply) a yes/no answer. So we use them for different things and this is reflected in the sample size for qualitative and quantitative being determined differently. With quantitative there is a mathematical basis, with qualitative you interview until you keep hearing the same data – you aren’t really collecting new data. So, start small. You might want to try 10 interviews with a 1 panel to start with – just to get the feel for what its like. You could even go and collect these stories yourself to really get the understanding. Then plan your application of MSC.

 

Where I have large MSC (and here I am talking about 300 stories a year – not 1000’s) I generally don’t send stories of no change to the panel. I will analyse these separately (though I would tell the panel that of the 30 collected, 10 had no change). If there are still lots, on a few occasions we have grouped stories (sometimes using hierarchical card sorting but only to 3 levels) with a group of stakeholders, and then chosen the MSC ‘most reflective’ of that group to go to the panel. The HCS process also gives really useful information for the monitoring or evaluation.

Yes, I do get “no change”. I put these in a separate pile and analyse. I also do quantitative analysis of MSC which gives other useful information for both M&E. I use the most significant change database (https://www.mostsignificantchange.com/) to manage my data and support both quantitative and qualitative analysis of the data. Personally I would never do a large MSC without this (in reality I always use the database if I have 3 or more panels or 2 levels).  

Also remember the stories selected as the MSC by a panel aren’t the key – it is the discussion. The discussion is the same as analysis in a qualitative evaluation. So taking 9 stories to a panel and selecting 3 probably isn’t going to get the level of discussion you really want. Aim to select just 1 and accept 2 where they aren’t going to reduce further. But the discussion is what matters – document this, it is the data analysis step.  

Happy to discuss on zoom or what’s app if you would like.

 

All the best

Fiona

 

Dr Fiona Kotvojs GAICD
Monitoring and Evaluation Specialist
Kurrajong Hill Pty Ltd

 fi...@kurrajonghill.com.au
Phone: 0448 453 422

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/e4b2dd53-2095-4f07-b311-9a696ab7a3c8n%40googlegroups.com.