psiturk without mturk?

[stephm...@gmail.com]

Hello,

I created a behavioral task with psiturk and I am currently hosting it using OpenShift. I was wondering if it is possible to collect task data without going through mechanical turk? I am hoping to send the OpenShift domain name to potential participants and connect them directly to the task.

Thanks,

Stephanie

[Todd Gureckis]

Yes, just make sure the server is running and send people manual links where you set the Worker ID and Assignment Id stuff by hand...

e.g.

http://MYIPADDRESS:22362/exp?hitId=MYEXP&assignmentId=MYEXPVERSION1&workerId=SUBJECT100&mode=debug

I believe mode=debug is the best option now, but there are grumblings about making a "lab" mode
https://github.com/NYUCCL/psiTurk/issues/95

> --
> You received this message because you are subscribed to the Google Groups "PsiTurk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to psiturk+u...@googlegroups.com.
> To post to this group, send email to psi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/psiturk.
> For more options, visit https://groups.google.com/d/optout.

[yani...@gmail.com]

Is a lab-mode available? I would like to try my experiment outside of mturk, and possibly use turkprime.com for the mturk microbatching feature..
Can this be done? does someone have any experience with turkprime?

Thanks,

Yaniv

[todd gureckis]

yeah all this works… we often use it in non-mturk settings.

> Visit this group at https://groups.google.com/group/psiturk.

[yani...@gmail.com]

Is there a way to make lab mode the default if no mode query string is provided? TurkPrime cannot provide a mode query string, but I'd like to use it for it's automatic splitting of the HIT to HITs of under 10 participants.

Alternatively - is there a script that can do that? automatically create 10 participant hits until my participants target is met?

thanks,

yaniv

[todd gureckis]

you’d have to hack it to do that (not that difficult). however, psiturk can be used to create hits with less than 10 assigments
see the manual for ‘hit create’

[yani...@gmail.com]

Thanks! that was useful

On Tuesday, 22 December 2015 08:24:42 UTC+2, Todd Gureckis wrote:
> you’d have to hack it to do that (not that difficult). however, psiturk can be used to create hits with less than 10 assigments
> see the manual for ‘hit create’
>

[yani...@gmail.com]

If I create multiple small HITs for the same experiment, will psiturk still prevent a worker from completing the experiment more than once? How is this implemented?

[Paul Feitzinger]

On Tue, Dec 22, 2015 at 08:01:24AM -0800, yani...@gmail.com wrote:
> If I create multiple small HITs for the same experiment, will
> psiturk still prevent a worker from completing the experiment more
> than once? How is this implemented?

I believe as long as you keep the `database_url` and `table_name`
options in config.txt the same, workers will be blocked from running
your experiment more than once, within and across hits.

The check is performed in psiturk.experiment.py:advertisement

https://github.com/NYUCCL/psiTurk/blob/2.1.2/psiturk/experiment.py#L242

and the sqlalchemy ORM is configured to use the `database_url` and
`table_name` config vars here:

https://github.com/NYUCCL/psiTurk/blob/2.1.2/psiturk/db.py#L11
https://github.com/NYUCCL/psiTurk/blob/2.1.2/psiturk/models.py#L12

[Elisa Celis]

Hi All,

Were you able to get psiturk working with turkprime? I want to do the same, also for the automated microbatch feature. (I know psiturk can hit create in small batches, but I have 1000s of HITs so am trying to find an automated solution).

Thanks,
-e

[Dave Eargle]

I have this on my todo list to be possible from psiturk. You can crontab a script that will in turn invoke `psiturk -s` on a file which runs `hit create <params>`, and which also keeps track of how many assignments have been created via reading and writing a number to a simple textfile or something. You mentioned you're on my openshift cartridge. If you're using the psiturk ad server (the default is yes), you'd have to do `server on` (even though the cartridge doesn't use that server) so that `hit create` won't complain about the server not running.

I haven't tried it yet, but should work.

If you don't want to use `psiturk -s`, you can access the internal methods  of psiturk for use in your own scripts by importing them, like in this gist

[Dave Eargle]

I just created a new branch that should make batch creating hits much much easier. I'll merge it later after I have a working example to put into the docs... but here's the branch.

https://github.com/NYUCCL/psiTurk/tree/execute_one_cmd

With this, you can run `psiturk -e 'hit create 9 1.00 1'` via a batch script, something like this:

```

while [ : ]

do

    `psiturk -e 'hit create 9 1.00 1'`

    sleep 1h

done

Except instead of while [ : ] you implement some logic that keeps track of how many you've created so far, and that stops once you've hit your desired level. You'd repeat `psiturk -e 'hit create'` as many times within one loop as you needed. This script would be run in the same dir as your config.txt.

Or you can implement the loop and sleep logic in python and just call a shell script via `from subprocess import call`, shouldn't make a difference.

[todd gureckis]

This is cool…  One idea for the future is to make psiturk export some of its functionality as a regular python library as well.

so if you have psiturk installed you could do something like

import psiturk as pt

for i in range(10):

pt.hit_create()

…

there is a devious plan behind doing this which to eventually make building up a GUI dashboard to control your psiturk

process and monitor the data.  it was a feature a long time ago but we switched to the command line interface because

it was easier to add/subtract functionality without corresponding GUI changes.  now that things are more stable with

new command line functions we could revisit the GUI interface making psiturk much easier for new users.

T

[Dave Eargle]

@todd that sounds intimidating

Okay here's a customizable bash script that batch-posts hits to mturk using your config.txt settings. So if you want to post these hits live, change `launch_in_sandbox_mode = false` in your config.txt

https://gist.github.com/deargle/f454a156d3a72adf62869e2ef09101d8

Disclaimer: I'm not ultra confident with bash nor with non-GNU envs so this might not be the most portable script. But hopefully it'll be an example for what you can do on your own in python or whatever if you just can't tweak it to work.

I'll pull the corresponding branch into @master... eventually.

[qian...@gmail.com]

在 2016年8月3日星期三 UTC+1下午9:36:33，Dave Eargle写道：

Hi Dave,

I just notice this feature (batcher) and I have checked the code provided in github (psiturk_batcher.sh). In my testing experiment, I created an HIT which has 20 assignments. However, I saw there are 56 accept (including 8 complete) in dashboard. Some of them received 1015 error (and 1008, 1009, 1010). I wonder could I use this feature to avoid 1015 error? In my understanding, the total_assignments I need to set in my experiment is 20, right? Many thanks!

Best,
Qian

[Dave Eargle]

Hi Quian, the script just calls 'hit create' as if you typed it yourself in a psiturk instance launched in the directory on the server where you're running the script, so if you're seeing errors, skip the script and run 'hit create' manually. I suspect that the script can't find your psiturk api server credentials in the .psiturkconfig file that it would be checking -- that's what throws a 1015.

[Dave Eargle]

> I suspect that the script can't find your psiturk api server credentials in the .psiturkconfig file that it would be checking -- that's what throws a 1015.

Sorry, I misspoke here. It's not the script that can't find your credentials, it's the running psiturk server instance that can't. Where are you running your psiturk server? openshift?

[qian...@gmail.com]

在 2016年10月26日星期三 UTC+1下午3:51:01，Dave Eargle写道：

Hi Dave,

I run psiturk on our university server. I released an HIT which has 20 assignments on AMT (live mode). Some workers reported me that they met error 1015. I didn't get your meaning that my psiturk server can't find the credentials. Could you please explain it further? And as I mentioned, the maximum assignments I set is 20, why there are over 90 workers can accept the assignment?

Best,
Qian

[Dave Eargle]

> Some workers reported me that they met error 1015. I didn't get your meaning that my psiturk server can't find the credentials. Could you please explain it further?

The only place in the code that a 1015 can be thrown is here. 1015 is "api_server_not_reachable." The error will be thrown if your psiturk api username and password are wrong in the loaded CONFIG. The config is loaded when you run `psiturk` on the command line (and then after you run `psiturk`, you run `server on` to launch a server with the loaded config). If you change the contents of ~/.psiturkconfig, you'll need to restart that server. My batch script invokes its own `psiturk` process, so it will pick up changes to that file even if older running processes don't. That can lead to situations where the hit is created successfully via `hit create`, yet when participants go to perform the hit, the older running process can't access the api server when they access /exp (that's when that linked code above is called). 

> And as I mentioned, the maximum assignments I set is 20, why there are over 90 workers can accept the assignment?

TOTAL_ASSIGNMENTS=20 should limit it to 20 total. Not sure why there are 90. 

* Did you change anything else? 

* run `./psiturk_batcher.sh` and show me the output. 

* Also, paste in your settings for these variables.

* what is your output for the script showing?

For the record, I and at least one other person has used the script successfully.

[qian...@gmail.com]

在 2016年10月26日星期三 UTC+1下午4:27:10，Dave Eargle写道：

Hi Dave,

Many thanks for your detailed explanation.

I didn't use batcher in my previous experiment and workers got 1015 error when did the experiment without using batcher feature.

Because I saw there are over 90 accepts while I set the maximum assignment as 20, I think batcher.sh may help me to solve this problem. I will try this feature following the instruction. btw, do you have any idea about why 'accepts' >> maximum assignment without using batcher?

Best,
Qian

[Dave Eargle]

> Because I saw there are over 90 accepts while I set the maximum assignment as 20, I think batcher.sh may help me to solve this problem. I will try this feature following the instruction.

The script won't help if there's some other problem with your api creds somewhere along the line.

> btw, do you have any idea about why 'accepts' >> maximum assignment without using batcher?

Where are you setting the "maxiumum assignment" number? In `hit create`? If so, it's possible that you have the order of parameters wrong. Each hit will only allow the maxiumum number of assignments that you specify in the first parameter to `hit create`.

[qian...@gmail.com]

在 2016年10月27日星期四 UTC+1上午1:01:50，Dave Eargle写道：

I create hit using the command like 'hit create 20 0.20 10' the first number is the assignment I set, the second is rewards for single assignment, the third is the HIT duration. I think this is correct. I'm not sure what' wrong with my api...

[qian...@gmail.com]

在 2016年10月27日星期四 UTC+1上午1:01:50，Dave Eargle写道：

Hi Dave,

I noticed that the 'number of participants' when creating an HIT means the number of successfully submitted assignments. If yes, it can explain my problems. Although my experiment have 90+ accepts but only about 10 completes the experiment (status==3 or 4). So workers can still accept the assignments. Is my understanding correct?

Best,
Qian

[Dave Eargle]

I'm a bit fuzzy on how mturk defines it's terminology. where are you seeing the number of 'accepts'? in psiturk? on mturk.com? 

if psiturk, here's the two places that are responsible for querying mturk and printing the hit data. That's all we get back, so however mturk defines them is what we show.

[qian...@gmail.com]

在 2016年10月27日星期四 UTC+1下午3:06:51，Dave Eargle写道：

Hi Dave,

I have tested that by creating a basic version of my experiment. In my case, I set the 'number of participants' as 3. And I used my 3 own registered worker accounts to test, 2 complete the task and 1 quit earlier. Then I log in amt using the fourth account. The task was still accessible but it disappeared once I finished the task (so there were 3 completed assignments).

Best,
Qian