How are the levels calculated?

[Tina Lund]

Hello,

I have recently asked my students to complete three of the Dialang tests in English (Reading, Structure and Vocabulary) and am now in the process of analyzing their results. I am particulary interested in seeing how closely their CEF level in Reading is linked to the levels they have achieved in the Structure end Vocabulary tests. I have a couple of questions that I hope you can help me with with regard to how the levels have been 'calculated'.

For an example, I have two students who have similar scores in the placement tests (375 and 360), so they are in the same bracket. However, one of them is more confident in her reading abilities and has ticked off more of the boxes in the self assessment questionaire. This means that one has been given the easy test and the other one has been given the intermediate test. When it comes to correct answers, they both have 21/30 in Reading, but the student who has taken the easy test achieves level A2, whereas the other student achieves level B2.

What I am wondering is whether there is an inbuilt limit to the level they can achieve with each test - would a student who manages a 30/30 score in the easy test be given a C2 result? 

Furthermore, it seems that for the Structure test it only takes one mistake to go from C2 to C1 and only a further mistake to go to B2, whereas the Reading tests have different cut off points. Is there some sort of overview over the different tests that shows where these cutt off points are and whether all types of mistakes count the same?

I look forward to hearing from you.

Kind regards, Tina Lund

[Adrian Fish]

Hi Tina,

I'm going to have to pass that on to one of the academics who actually designed the algorithm. It's a bit out of my league, to be honest. I'd assume that you just can't achieve a C2 when taking the easy test. The levels are supposed to be absolute measures of ability, so it would make sense.

Cheers,

Adrian.

[Adrian Fish]

Hi Tina,

Here we go ...

Each test consists of a set of items which were calibrated individually and the 3 tests for each language/skill are made up of these. There's no guarantee of an easy test in English being as easy as the easy test in French, although such parity was attempted. There's no overview of these differences, there's just a set of tables with the discrimination values for each individual item, and these values are what are used to provide the final CoE level. You'd have to run some comparative analysis across the dataset.

So, to summarise: you can get a C2 in an easy test in the case of a language/skill that just happens have a set of relatively difficult items. That's a valid scenario. There is no absolute relationship between test difficulties between languages or skills.

I hope that helps. If you want to delve further into the theoretical underpinnings of the whole thing I could hook you up with on of the academics who worked on DIALANG's item system.

Cheers,

Adrian. 

[alexanderv...@gmail.com]

Dear Adrian,

My employer is very interested in the algorithms behind DIALANG, as we would like to develop an application of our own to guage the language proficiency of candidate helpdesk operators that ought to be deployed on a variety of projects. Do you think you could hook me up with the relevant academics? Would s/he or he be interested in a joint effort, or perhaps the sharing of the dataset?

Thank you in advance for your time and attention, Alexander

[Adrian Fish]

Hi Alexander,

Can't they just take the test online rather than reinvent the wheel? You can hook Dialang web up to any LTI consumer now, such as Sakai, Moodle or Blackboard, among others, and, when you do this, you can download reports of student scores against the CoE framework. Won't that do the job?

If not, then the linguists at Lancaster may well be interested in some kind of collaboration, especially if that collaboration results in some growth of the item banks.

Cheers,

Adrian.

[Hussein Abdelfatah]