Rely command - assessing reliability for each code separately

[Marissa Castellana]

Hello,

I am trying to assess inter-rater reliability using the RELY command for each of the four main codes (lowq, refq, recq, and infq) on the dependent tier (%spa) in our transcripts. Each code was also denoted with a "-s", "-m", or "-e" depending on the language used. For example, an utterance may be tagged as "lowq-s", "lowq-m", or "lowq-e".

However, my code (rely +c  -t* +t%spa +t*MOT +d) seems to capture overall reliability instead of reliability per code. I have tried adding +s"lowq" and also attempted adding the more specific sub-code +s"lowq-e", but these are not producing the intended results. Does anyone have any advice or potential solutions so that I can gather reliability per code rather than overall?

Thank you!

Best,

Marissa

[Leonid Spektor]

Hi Marissa,

I am not sure if I understand correctly what you want. I have made two sample files. Here is the first one:

*MOT: child's word.

%spa: lowq-s

*MOT: something other .

%spa: lowq-s

*MOT: child's word.

%spa: refq-s

*MOT: something other .

%spa: refq-e

and here is the second one:

*MOT: child's word.

%spa: lowq-s

*MOT: something other .

%spa: refq-e

*MOT: child's word.

%spa: lowq-s

*MOT: something other .

%spa: refq-m

-----------------------------------

if I ran the following command, then I get result:

> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha

Marissa1.cha: # utterances examined: 4

Marissa2.cha: # utterances examined: 4

% of all utterances examined with matching codes %: 100.0000

Marissa1.cha: # words examined: 4

Marissa2.cha: # words examined: 4

# words (tokens, codes) matched: 3

% of all words examined with matching codes %: 75.0000

I believe it is a correct result, because all codes except for "refq-s" and "refq-m" match. Please note that the match is for the whole file and not based on what speaker utterances those code are associated with.

-----------------------------------

if I ran the following command, then I get result:

> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +slowq-s

Marissa1.cha: total lowq-s: 2

Marissa2.cha: total lowq-s: 2

total lowq-s matched: 2

total lowq-s %: 100.0000

In this case RELY only looks at "lowq-s" codes and those two codes per file match exactly between two files.

-----------------------------------

if I ran the following command, then I get result:

> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +s*-s

Marissa1.cha: total *-s: 3

Marissa2.cha: total *-s: 2

total *-s matched: 2

total *-s %: 66.6667

 Again there are three codes with -s in the first file  and there are only two codes in the second file.

Leonid.

-----------------------------------

I am attaching my two sample files here. If this is not what you wanted, then please explain in more details and maybe give me an example of what you are trying to achieve. Please email to me directly at spe...@andrew.cmu.edu.

Leonid.

[Marissa Castellana]

Hi Leonid,

Thank you so much for this detailed response! 

Yes, that code worked! This code (rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +slowq-s) captures reliability for the lowq-s code, and I added all lowq codes (rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +slowq-s +lowq-m + slowq-e) to gather reliability for lowq overall.

Thanks!

Best,

Marissa

--
You received this message because you are subscribed to a topic in the Google Groups "chibolts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chibolts/NTR16qI-RIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/EACA6EF9-AE62-48CE-9FB4-7CB28F239511%40andrew.cmu.edu.

--
You received this message because you are subscribed to a topic in the Google Groups "chibolts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chibolts/NTR16qI-RIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/EACA6EF9-AE62-48CE-9FB4-7CB28F239511%40andrew.cmu.edu.

--
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/db4a7ef9-4541-4d8d-900a-d4c49d973ba7n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "chibolts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chibolts/NTR16qI-RIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/EACA6EF9-AE62-48CE-9FB4-7CB28F239511%40andrew.cmu.edu.