The website and in-app help have always been clear on this -- Sets C and D are well-matched and the other sets (E, F and the G/H ones we've had in-house) are easier. One reason for the other ones being easier is that when we're looking at groups with impairments, things can hit the floor. Now, you can always score the bins separately or plot curves as we've done in our work, but if half of your lures have no chance of ever getting a "similar" response, it's not optimal. Hence, the easier sets.
So, if you're doing a lot of repeat testing and have had to span across these, though, we've got an issue. How do you equate performance across sets? Since we've got a lot of data in-house, I've been able to work through the issue. The standard solution is just to use z-scores to say something like "you've got a z-score of 1.5 in Set E ... what is the LDI score in Set C that is at z=1.5?" That's a standard way of going about it and if all you've got is the LDI, that's what you'll be able to do.
But, here we've got performance across lure bins. By making Set E "easier", what this means is that Set E's "lure bin 1" is really more like Set C's "lure bin 1.6" Fortunately, with enough data, we can figure these out. What I've got in the picture here is the p("Similar"|Lure) for each of the sets both before (left) and after (right) the matching. What you can see is that we've got shifts going on and, really, the best way to think of this is that the Set E-H lines have been shifted "left" from where they should be. That is, they need to move "right" so that their first data point isn't "lure bin 1" but more like "lure bin 1.5".
In addition, the spacing between the bins isn't always spot-on. Since they weren't constructed specifically to match C/D, there's no reason why they should. When we made C/D we actually took A/B (that were similar but not perfect) and shuffled stimuli after rank-ordering them. So, we went C, D, D, C, C, D, D, C, ... thereby ensuring that C and D were nicely matched to each other without either "shift" or "stretch" differences along the lure-bin axis.
To cast E-H into the same "space" as C/D, we just need to shift and stretch the lure bin axis appropriately. I'll spare you the details here, but the upshot is that a simple 2nd order polynomial transform of the lure-bin axis for E-H nicely puts them all into the lure-bin axis space of C/D. In essence, you know you sampled bins 1.6, 3.1, 4.2, ... and then resample back to what bins 1, 2, 3, ... would have been. So far, as you can see from the plot on the right, it's working well.
If anyone has C/D and E/F performance on the same people, it'd be great to try this out. We've got some data here that we'll be working on, but independent labs' data is always good. Drop me a line and I'll send you the "beta" spreadsheet to try out.
Craig
