CFP- Crowdsourcing Special Session at QoMEX 2021

[bnad...@gmail.com]

[Apologies if you receive multiple copies of this CFP]

 

Special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021 | 14 - 17 June 2021 | Virtual

13th International Conference on Quality of Multimedia Experiences

https://qomex2021.itec.aau.at/special-session-4

==========================================================================================

 

 

Important Dates:

- 20th January 2021: Grant application (optional, see below)

- 21st January 2021: Grant acceptance notice
- 16th February 2021: Full paper submission

- 28th March 2021: Full paper notification

 

PAPERS SUBMISSION : Please check out the QoMEX 2021 timeline and guidance, as the papers should be submitted following the conference instructions.

Publication: IEEE

 

Grant: $ 200 financial support provided by Microworkers.com for paper ideas who want use the Microworkers.com for their experiment and their grant application is accepted. See below for details.

 

Dear Colleagues:

We organize a special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021.

Laboratory studies are an established and essential tool for Quality of Experience (QoE) and User Experience (UX) research. However, they required well-equipped test rooms and personnel for the supervision of the test participants. Therefore, they are often cost- and time-intensive. Further, the number of test candidates is often limited due to the sparse laboratory space and need for the test taskers' physical presence in the test environment. In the past months, the pandemic situation made it even more challenging to conduct laboratory studies by increasing the organizational overhead and limiting potential participants. Two possibilities to overcome the current situation are crowdsourcing and remote user studies.

In this context, the special session aims to foster contributions following the traditional way of optimizing and designing crowdsourced subjective studies for Quality of Experience and User Experience research and additionally wants to raise awareness and foster research in a new research direction, the Quality of Experience and User Experience of crowdsourcing workers. The special session also seeks to encourage researchers to exchange their experiences in remote user studies, with non-anonymous test takers, how best practices from crowdsourcing studies can be applied in this context, and discuss which new challenges arise.

 

The topics of interest include but are not limited to the following:

• Crowdsourcing for subjective studies
• Novel applications 
• Limitations of current crowdsourcing systems
• Quality control mechanism and reliability metrics 
• Large scale crowdsourcing studies and diversity of participants

·        Reproducibility of results

• Reproducibility and cross platform studies 
• Assessment of hidden influence factors / Impact of hidden influence factors
• Bias estimation and bias reduction
• Automation and workflows

·        Standardization of crowdsourcing test methods

• Usability and User Experience of crowdsourcing tasks
  • Optimization of task interfaces and task workflows
  • Relation to result quality and worker motivation
  • Enhancing workers’ UX (e.g., by means of gamification of tasks)
  • Quality of complex crowdsourcing workflows (e.g. combination of AI and Crowds)
• Interconnection of crowdsourcing and lab-based tests, 
  • Studies comparing results from lab and crowdsourcing
  • Adaptations of established lab-test standards to the crowdsourcing environment
• Remote user studies
  • Supervised remote user studies
  • Remote studies with non-anonymous users
  • (Crowdsourcing) best practices for remote user studies

 

Authors are kindly invited to submit their full papers by the deadline. All submissions are handled through the QoMEX2021 submission process: https://qomex2021.itec.aau.at/papersubmission/

For any query, Please write mail to babak.naderi@tu-berlin and matthia...@tu-ilmenau.de.

Sincerely Yours,

Matthias Hirth, Institute of Media Technology, TU Ilmenau,

Babak Naderi, Quality and Usability Lab, Technical University of Berlin

 

Grant details:

In line to support research on the QoE using crowdsourcing, Microworkers.com offers $200 grants for selected ideas who want to conduct an experiment on their platform and submit a paper to this special session. The applicants have to send a research abstract and a rough cost plan how the grand will be spend to the to the session organizers.  All applications will be forwarded to Microworkers.com where a company representative will make the final funding decision. Note, the  funds cannot be withdrawn from Microworkers and can only be used for research purposes.

Please send your 1)research abstract, 2) cost plan by20th January (AOE) to matthia...@tu-ilmenau.de and babak....@tu-berlin.de. The selected candidates will be announced on 21th of January.