RELY

[Karin Edlund]

Hi,

What is the accurate command for RELY and computing percentage agreement on a dependant tier (i.e. %cod)?

I have tried 

rely +d sample.cha samplea.cha +t%cod -t*

Also, what is the command for calculating Cohen's kappa on %cod tier?

I have tried with +dN, but I can't get it to work unfortunately.

Happy to receive advice on how to solve this.

Best Regards,

Karin Edlund

[Brian MacWhinney]

Dear Karin,

    You need to provide a number for N, based on the number of categories in your coding system.  I am assuming that you are using RELY as described in the second function in the section on RELY in the CLAN manual.  In the manual, the suggested command uses +d1 which doesn't make sense.  If you have two categories, you need +d2.  Then, for that example in the manual. you get Cohen's kappa of 0.92.  Of course, all of this depends on the actual shape of your coding system, and this little example can't illustrate that.  However, I will at least fix that error in the manual.

--Brian MacWhinney

--
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2xW3-jbmfqNXyCVLAwZOh8dpCccdTeEWqujDsJQqTgH%3DQ%40mail.gmail.com.

[Brian MacWhinney]

Dear Karin,

    Looking now at that section of the CLAN manual, I see that it had not been properly updated to include both the +dm1 and +dN switches.  Here is the revised section:

The second function of the RELY command is to evaluate the correctness of a student transcript against a master transcript. The student is given a version of the complete master transcript without the coding tier and the goal is to add codes. In this case, it is assumed that the coding in the master transcript is fully correct. The question is whether the student's codes are correct. To specify that the first file given is the master, you use the +dm1 switch.  In addition, to compute Cohen's kappa you need to add the +dN switch. In this example, we can say that there are only two levels to the category, making the +dN switch to be +d2.  However, the number used would depend on the actual levels of the coding system.

rely +d2 +dm1 sample.cha samplea.cha +t%spa

The output from this command provides two counts: precision and accuracy.  These are the two dimensions used in fields such as computational linguistics to evaluate correctness.  Precision is the percentage of fields in the master file that are matched by the student. Accuracy is the percentage of fields in the student file that are matched to the master. A failure in precision occurs when the student misses a code in the master file.  A failure in accuracy occurs when the student inserts a code that is not found in the master. If a researcher is interested in going beyond these two scores, they can also compute the F score which is the harmonic mean of precision and accuracy.  By adding the +d2 switch, the command also produces Cohen's kappa.

--Brian MacWhinney

On Jun 11, 2020, at 5:17 AM, Karin Edlund <karin.ev...@gmail.com> wrote:

[Karin Edlund]

Hi Brian,

Thank you for your input on the RELY function.

What I would like to do is the following:

1) Check reliability on percentage agreement for word and utterance level on each of 15 transcriptions comprising 5 min each.

I was thinking of using the third function of RELY - the overall match on the main line between two transcripts - rely +d sample.cha samplea.cha

This would be only to check the transcription, not the coding. 

I would say the two transcribers are on the same level regarding transcribing, thus perhaps it is not a case of checking correctness of a student against a master transcript. 

Or is the relation master-student more at matter of what data counts as the primary data?

2) Compute Cohen's kappa for the coding. Two coders. I have five different categories for coding. I am not using %spa, but %cod.

Would this be the right code for that:

rely +d5 sample.cha samplea.cha +t%cod 

what does dm1 stand for in the RELY code you suggested?

Perhaps I am mixing things up, but overall I am trying to check percentage agreement for word and utterance level in the transcriptions between two transcribers and secondly I am computing cohen's kappa for the coding. 

Seeking for the best RELY function and its respective code to do this. 

Best Regards,

Karin Edlund

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/C233654B-250D-4B81-B2DC-F2005E936177%40andrew.cmu.edu.

[Brian MacWhinney]

On Jun 13, 2020, at 6:34 AM, Karin Edlund <karin.ev...@gmail.com> wrote:

Hi Brian,

Thank you for your input on the RELY function.

What I would like to do is the following:

1) Check reliability on percentage agreement for word and utterance level on each of 15 transcriptions comprising 5 min each.

I was thinking of using the third function of RELY - the overall match on the main line between two transcripts - rely +d sample.cha samplea.cha

This would be only to check the transcription, not the coding. 

I would say the two transcribers are on the same level regarding transcribing, thus perhaps it is not a case of checking correctness of a student against a master transcript. 

Or is the relation master-student more at matter of what data counts as the primary data?

Yes, that will work for that comparison.

2) Compute Cohen's kappa for the coding. Two coders. I have five different categories for coding. I am not using %spa, but %cod.

Would this be the right code for that:

rely +d5 sample.cha samplea.cha +t%cod 

what does dm1 stand for in the RELY code you suggested?

You still need +dm1 to tell CLAN that you consider the first file as the master.  

--Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2y-xD6aGFLzh26e%2Bvx185OJwvSKdAnB4j_%2BTGfmTVSJ_w%40mail.gmail.com.

[Karin Edlund]

Brian, 

Thank you so much again! I will try my best to get the commands to work.

Best Regards,

Karin Edlund

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/116DA890-299A-4EBA-B7C1-DFFB5C22AD1D%40andrew.cmu.edu.

[Karin Edlund]

Hi,

When accounting the overall percentage of agreement for overall match between two transcripts on the main line (function 3 of RELY), are Tier names included in that? When running the RELY function (3) in CLAN I got many "Tier names do not match" due to the fact that the transcribers use one set each of names of the participants. 

If Tier names are included in the accounting of overall agreement of the transcription, is there a way of coming around this (provided we can keep our own sets of names for the participants)?

Best Regards,

Karin Edlund

[Brian MacWhinney]

Dear Karin,

     I don't think that RELY should try to get around this problem.  Instead, I would recommend making sure that your transcribers use the same set of tier names.

--Brian MacWhinney

--
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2zECn_n3DWDUv20P38HYCfzufK6onPhbdPwf9PuJvhVHg%40mail.gmail.com.

[Karin Edlund]

Hi Brian,

Thank you for your quick answer.

I can understand that this is the case. We need to keep the participants blinded to the second coder and this is why we chose to work with separate coding systems for the participants. 

We are mostly interested in overall match on actual word and utterance level, thus the tier names seem not to be so interesting to include for that calculation?

Best Regards,

Karin Edlund

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/9EE985BB-D778-4BDE-A0CE-C4581D19E5F4%40andrew.cmu.edu.

[Brian MacWhinney]

Dear Karin,

    It should be pretty easy to change your tier names using the CHSTRING command or just the REPLACE dialog in CLAN.  I guess you only need to change one of the two files.

-- Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2xEGX-uAou19zrpX0K_bS-L_8BFP%2B9hxWFR9KG%2BZqrLFw%40mail.gmail.com.

[Karin Edlund]

Hi,

Yes, we will do that. Thank you again!

Best regards,

Karin

To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/F1DB467F-5630-4C69-AD40-F2924ABD93AB%40andrew.cmu.edu.