We wrote a paper that examines turn level dialogue annotations (
https://aclanthology.org/2023.findings-acl.780). In order to study the annotations, we created an annotation protocol. Our protocol is specifically focusing on the preparation, annotation, and evaluation stage of the annotation pipeline.
We are now in the process of extending our prior work to solely focus on studying the protocol (see figure 4 in the paper). In addition to this we are exploring LLMs in the pipeline.
The main information we'll be discussing is in the appendix.