Inter-Rater Agreement with just the words we are coding for

[Anya Montgomery]

Good Afternoon,

We are currently coding transcripts and there are two
different students coding these transcripts. We would like to achieve
80% conformity by our transcripts. We previously running the RELY +d command, however we realized that the program was accounting for all of the words in the transcript as opposed to checking for conformity between only the words that we coded for. Does anybody know of a way that we can run a test that tells use the inter-rater agreement or percentage of reliability based on the words that are coded for?

thank you for your help,

Anya Montgomery 

[Leonid Spektor]

Anya,

If your students are coding on specific dependent tier only, then you can use +t option to tell RELY to look at that dependent tier only. You might try adding +c option too.

Leonid.

--
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/f92bed6c-25fc-4075-ad56-b3890b4e17ea%40googlegroups.com.

[Anya Montgomery]

Thank you so much, Leonid, for responding. We attempted to use the +t option and the +c option but we were unable to determine inter-rater agreement. What specific command should I run? 

On Wednesday, August 28, 2019 at 2:36:29 PM UTC-4, Leonid Spektor wrote:

Anya,

If your students are coding on specific dependent tier only, then you can use +t option to tell RELY to look at that dependent tier only. You might try adding +c option too.

Leonid.

On Aug 28, 2019, at 14:25, Anya Montgomery <amon...@fiu.edu> wrote:

Good Afternoon,

We are currently coding transcripts and there are two
different students coding these transcripts. We would like to achieve
80% conformity by our transcripts. We previously running the RELY +d command, however we realized that the program was accounting for all of the words in the transcript as opposed to checking for conformity between only the words that we coded for. Does anybody know of a way that we can run a test that tells use the inter-rater agreement or percentage of reliability based on the words that are coded for?

thank you for your help,

Anya Montgomery 

--
You received this message because you are subscribed to the Google Groups "chibolts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chib...@googlegroups.com.

[Leonid Spektor]

Anya,

The command that you used before plus +t%dep_tiers that you want to compare is all that it should need. Perhaps I don't understand what exactly you want. Could you email to me directly two sample files you want to compare and tell me what words and on what tiers you want to check inter-rater agreement for.

Also, make sure you have newer CLAN version installed.

Leonid.

To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/2ff0386e-a5a1-45cc-9a4a-6b263d0413ee%40googlegroups.com.