Sorting algorithms and MSC selection

[rick davies]

Hi all

In the field of computer science there is a body of algorithms (procedures) known as sorting algorithms. These are designed to enable the sorting of a disordered list of objects in to an ordered list e.g. 2 7 3 4  6 9 8 1  5 into 1 2 3 4 5 6 7  8 9. The main aim of these algorithms is to complete a sort as quickly as possible.

Selecting the most significant MSC story from a set of MSC stories is also a sorting process. In one respect it is a bit simpler: we only want to find the most significant, i.e. the top ranking story, we don't want to rank all the stories in terms of their significance. On the other hand the comparison process, where one story is compared to another, is more complex because there are multiple dimensions of significance to consider, not just a "is the number bigger or smaller" type of comparison, as in the number sorting example above.

Voting-discussion-voting is one common procedure for finding the most significant MSC story. That  does involve full ranking by aggregating individual preferences re the most significant story. In other circumstances the collective selection process can be less structured and more dialogic. This may involve a wide ranging discussion or many if not all the MSC stories, comparing them in various ways, until the panel feels they have identified the most significant story. 

I am now interested in an alternative, inspired by sorting algorithms, which retains the dialogic element, but is more systematically structured. Consider a set of MSC stories, randomly ordered: B D, A, F, E, C. The panel members then compare B and D, to identify the most significant. They choose B. They then compare B  to the next story in the random list - A. They select A. They then compare A to F. They select A. They then compare F to E. They then select E. They then compare E to C. They select E. That is their final choice. There were 5 comparisons (N-1). Every story does not have to be compared to every other story because we are not looking for a full ranking, just the top most ranking item.

Any thoughts on this possible approach? Could it mean that selection choices are better thought through and better  documented? Would it matter that some comparisons are not made e.g that F and B in the above example, are not compared? Would it necessarily be more or less time consuming that either of the other two methods described above (vote-dialogue-vote, or open unstructured dialogue)?

regards, rick davies

[Bob]

I’m wondering if that is a bit ‘rational’.  Our comprehension of the different dimensions of significance will alter as we go through the list.  Thus the E to C comparison may bring up a dimension or criterion or nuance of a criterion that was not being used for the original A-F comparison.  If it had been used, then maybe F would have been chosen and not A.  I can think of a variety of different ways of managing this issue - but none of them come near to regarding ABCDEF as independent one-dimensional components.  Sometimes the complex is just complex.

On 5/10/2021, at 10:20 PM, rick davies <rick....@gmail.com> wrote:

Any thoughts on this possible approach? Could it mean that selection choices are better thought through and better  documented? Would it matter that some comparisons are not made e.g that F and B in the above example, are not compared? Would it necessarily be more or less time consuming that either of the other two methods described above (vote-dialogue-vote, or open unstructured dialogue)?

Cheers

Bob

BOB WILLIAMS
b...@bobwilliams.co.nz
http://www.bobwilliams.co.nz
Mobile +64 21 254 8983
Skype : bobwill
Twitter : @boblwil

"Only Connect ..... "  E. M. Forster

System Diagrams: A Practical Guide

https://gum.co/systemdiagrams

Systemic Evaluation Design 
https://gum.co/evaldesign

Video -https://www.youtube.com/watch?v=VUTqG5j21mU

Wicked Solutions: A systems approach to complex problems.
https://gum.co/wicked

[rick davies]

Hi Bob

Thanks for these thoughts

 Re " I can think of a variety of different ways of managing this issue", please tell us more

I don't think the proposed method  is "regarding ABCDEF as independent one-dimensional components" Each binary comparison would be multidimensional. We are just looking at a sequence of such comparisons

That said, I agree that "Our comprehension of the different dimensions of significance will alter as we go through the list.  Thus the E to C comparison may bring up a dimension or criterion or nuance of a criterion that was not being used for the original A-F comparison. "

regards, rick

[Steve Powell]

Hi Rick

that's interesting. 

I think Bob's right that there's more at stake here than if you were e.g. just asking people to sort something objective like the length of rods: each pair will bring to mind different salient features. From that point of view, the procedure you suggest gives more weight to the ones you offer first as they get more chance for their salient features to be considered, don't they? So if B is obviously better than E but in particular because this story is so much about youth empowerment, then youth empowerment has more chance to carry forward as a salient feature doesn't it? 

And are there anchoring effects (re Kahneman) here? - i.e. given a decision where the criteria are unclear, people are going to grasp at anything, and if you (even accidentally) offer them a comparison which seems to suggest youth empowerment, people will take that and run with it, no? I think MSC done properly will try to counteract that by getting people to reflect on this very process, but reflecting on it doesn't mean eliminating it. 

Also, even if we were asking people to sort the length of rods, the procedure you suggest is always perfect to pick the longest rod, but only as long as people make perfect judgements each time, no? As soon as you introduce a bit of noise / random error, which in fact would be the case even with judgements about objective properties like length, then your procedure isn't optimal I believe - a more thorough procedure would introduce more steps to counteract random error. And the presence of noise is another reason your procedure is biassed in favour of stories earlier in the chain, isn't it? 

I think having said all that, it's exactly this kind of openness which can make MSC so interesting in practice, in dialogue with participants. 

Best wishes,

Steve

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/ca0e95de-a745-4a2d-b723-ca657a5815e6n%40googlegroups.com.

--

Causal Map: Identify and visualise causal connections in speech and writing

_____________________________________________________________

independent social researcher

skype: stevepowell99

mobile: +44 75 1088 1300

pogol.net

[rick davies]

Thanks Steve... so is there any other type of pair comparison process that people can imagine might be more useful? Serious question

[Steve Powell]

Rick, do you want a procedure which tries to surface and make explicit individual dimensions/ categories for comparison, or one which always leaves them open? For the former, there are procedures like PAPRIKA https://en.wikipedia.org/wiki/Potentially_all_pairwise_rankings_of_all_possible_alternatives but for the latter I'd say "ask Rick Davies, that's a special feature of MSC" ;-)

Best wishes,

Steve

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/5656b78e-85b6-43c4-9e2f-1a42d39be309n%40googlegroups.com.

[rick davies]

Hi Steve

Comparisons of all possible pairings  was also used by PRA people in the 90s, but it is the most time consuming approach possible My interest is is in other sorting algorithms that will do the same task but more time efficiently - while minimising the loss of information that could be lost in the process.

Whether the process is done by making sorting criteria explicit before hand, or emergent during the process is a second independent dimension. My interest of course is in the latter. This would be a constant, the variations I am interested in are the "ranking" methods that can be used with the latter approach

Though they may seem antagonistic, I don't think the use of algorithms and open-ended forms of discussion are necessarily so. Algorithms can benefit from  "relaxed constraints" and dialogue processes can benefit from some form of constraint

regards,  rick

[Fiona Kotvojs]

Hi Rick,

 

I have never completed MSC in the way suggested, and I think it would take longer than a more general discussion where people identify which they believe is the MSC and why.

 

From my perspective the value of MSC is in the discussions around selecting the MSC which surface values. I am not sure whether this approach would promote or reduce this.

 

Fiona

 

Dr Fiona Kotvojs GAICD

Monitoring and Evaluation Specialist

Kurrajong Hill Pty Ltd

 

fi...@kurrajonghill.com.au

Phone: 0448 453 422

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/CANum532tvp%2BaY0c4Gh27jbgxfGkbMGegosvnANX3O%3D-PJoCjrg%40mail.gmail.com.

[rick davies]

Hi Fiona

I am keen to retain the value you refer to, while seeing if it is possible to systematise the process of comparison that are usually involved in one form or another. Sorting methods struck me as one possible source  of ideas that might be worth exploring.

I agree the jury is still out re whether the method I proposed yesterday  would be both effective and effecient

regards, rick

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/SYCPR01MB5328BD6B4701E77A9459C21292B29%40SYCPR01MB5328.ausprd01.prod.outlook.com.

[Orlando José Leite de Castro]

Dear Rick,

Your alternative is logically perfect, considering the single objective stated.

If you have composed points to consider about each story, when comparing each pair, you might have to include a set of rules to stablish the winning conditions between each two, but the overall method remains correct. 

In this case, you have to achieve a result to each story, expressing its final score and, if needed, some exceptional cases of uncertainty, that would maintain both on the search for the best, turning more complex the algorithm, without violating the general rule.

Hoping to be useful, I send you my

Best regards

Orlando

[Bob]

Rick have you ever tried the final check comparing E with B? Although I recognise that the psychological pressure will be to select E anyway, it is possible that the criteria for selection (or at least the qualitative measure) may shift as the sort continues, so that by the end (admitted of a relatively small list), the criteria may have undergone a subtle shift in emphasis.    And have you tried it where people know all the stories before the sort vs reading the stories for the first time when comparing?  

On 12/04/2022, at 12:21 PM, Orlando José Leite de Castro <ojca...@gmail.com> wrote:

I am now interested in an alternative, inspired by sorting algorithms, which retains the dialogic element, but is more systematically structured. Consider a set of MSC stories, randomly ordered: B D, A, F, E, C. The panel members then compare B and D, to identify the most significant. They choose B. They then compare B  to the next story in the random list - A. They select A. They then compare A to F. They select A. They then compare F to E. They then select E. They then compare E to C. They select E. That is their final choice. There were 5 comparisons (N-1). Every story does not have to be compared to every other story because we are not looking for a full ranking, just the top most ranking item.

Cheers

Bob

BOB WILLIAMS
b...@bobwilliams.co.nz
http://www.bobwilliams.co.nz

Mobile Aotearoa +64 21 254 8983

Mobile Argentina +54 9 11 3602 2284

Skype : bobwill
Twitter : @boblwil

"Only Connect ..... "  E. M. Forster

[rick davies]

Hi Bob

Up to now this has only been a sort of thought experiment, I have not tried it in real life

But your suggestion is worthwhile

It is quite likely that there will be some shift in criteria being used, as the comparison process proceeds, especially if one item does not consistently win in the pair comparisons

regards, rick

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites: http://www.mande.co.uk  and http://richardjdavies.wordpress.com/ | Twitter: @MandE_NEWS | rick....@gmail.com Skype: rickjdavies

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/686BA3DD-143A-4DA0-9E7A-94FCC1493881%40bobwilliams.co.nz.