Problem exporting large experiment from local Jatos server

[matt Anderson]

Hi,

I have a large psychophysical experiment that works just fine in my local Jatos installation. The entire folder containing all my assets is 4gb in size (it contains a large number of images). When I try to export my study, it seems to be working, and generates a .jzip file to upload to the server installation. However, a quick look at the downloaded .jzip file reveals that something went wrong: it's only 4 bytes in size. Of course, if I try to upload this file to the server installation of jatos, I get an error (Failed to load resource: net::ERR_FILE_NOT_FOUND). 

I wonder if anyone can shed any light on this? Given that the .jzip file takes a while to download (30+ mins), I'm repeatedly prompted to log in due to session time-out. Perhaps this is part of the issue? If I try using a smaller version of the experiment (<1gb) it works perfectly. 

So, any ideas? 

Best,

Matt

[Kristian Lange]

Hi Matt,

Interesting problem. I can't remember ever having seen such a big study.

The reason might be that the internal buffers are limited to 2GB by default. But there are two parameters that can go into production.conf that might help here. If your study is 4GB in size you should put them a bit higher, maybe 5GB. So, on both, your local and your remote JATOS you should set in conf/production.conf: 

play.http.parser.maxDiskBuffer = 5GB
play.http.parser.maxMemoryBuffer = 5GB

Just put them anywhere, e.g. the end of the config file. And don't forget to restart JATOS afterwards.

I hope this helped already.

Best,

Kristian

[matt Anderson]

Hi Kristian,

Yep - I need labels for a large number of images, so there's no way of reducing the size of this experiment. 

I gave your suggestion a go but it doesn't seem to work. My local jatos still exports tiny files (4 bytes) that I can't unzip. 

To be clear about what I tried, I simply added the following two lines to the end of the production.conf file:

#play.http.parser.maxDiskBuffer = 5GB

#play.http.parser.maxMemoryBuffer = 5GB

then launched jatos again and exported the same study (I also tried restarting my computer). 

Any other ideas about what might work? 

Thanks for your help so far!

Matt

[Kristian Lange]

Hi Matt,

Just to be sure, you added those two lines without the # in front, didn't you? The # would turn them off.

Best,

Kristian

[matt Anderson]

Yes - that's right (I accidentally copy-pasted the commented version). Here's the .conf file:

Matt

[Kristian Lange]

Hi Matt,

I tried on my JATOS and I can confirm: larger study folders (I used a 4.5 GB one) can't be exported. I tried out some things but in the end it didn't work.

But on the positive side, there might be an easy way around it. JATOS' exported studies are just zip compressed folders. You could just export your study in JATOS without the 4GB of image files and then unzip the .jzip file, put the audio files into the folder along your other study assets, and zip it again. This zip you can import like usual to your remote JATOS. And one hint: many zip tools wrap everything you want to zip into an extra folder. JATOS can't handle this extra folder. To make sure that your re-zipped study is fine: If you peek into it, a '.jas' file has to be in the root.

Another way would be to upload your image files directly to your JATOS server with scp. All study asset files are stored in a folder (by default study_assets_root in your JATOS folder)  and you can add files there manually.

Hope this helps. Sorry that JATOS doesn't handle 4GB studies (yet)!

Best,

Kristian

[Kristian Lange]

Another idea that just came to my mind: You don't necessarily have to load your static image files via JATOS - any file storage service that one can request via HTTP can do this job. The one that comes to my mind is AWS S3, but Google Drive or Dropbox might also work. This way you can keep your study jzip smaller and therefore easier to handle. 
Kristian

[matt Anderson]

Thank you for your help Kristian! It seems like the easiest option may be to unzip the .jzip file and then put the files in before re-zipping, so I'll have that a shot and see what happens. If not, I can get the server admin to import my files with scp. 

Best Wishes,

Matt

[Kristian Lange]

I'm curious if it works out in the end.

Best,

Kristian

[Chi Zhang]

Hi Kristian,

Sorry to interrupt. I just happened to see this email. Would you please be a bit more specific about how the storage service would be working with JATOS? Thank you!

Best,

Chi

--
You received this message because you are subscribed to the Google Groups "JATOS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jatos+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jatos/5329c99e-fc02-4f1f-b1d5-a0bb0d51bfe5n%40googlegroups.com.

[Kristian Lange]

Hi Chi!

I'm not sure I understand you. This thread was about uploading of huge study jzips (4 GB) to a JATOS server. It's not about a storage service - unless you see JATOS as a storage service. Can you specify your question a bit more?

Best,

Kristian

[Kristian Lange]

Hi Chi!

I think I understand you better now. You want to know more about those external storage services one can use to provide files (e.g. images, videos) in your experiment's web page.

But there is not much to say. The files you load in your experiment don't have to come from the JATOS server. Just an example: I have this images here on my Google Drive: https://drive.google.com/file/d/0B62IPJoUI5wmcS1qSGppdERaVjg. I made it public so everyone can access it. In your experiment's HTML you could use it with:

<img src="https://drive.google.com/uc?export=view&id=0B62IPJoUI5wmcS1qSGppdERaVjg">

Although I'm not sure that to load files from Google Drive is a good idea. I'm not even sure whether they allow this use in their EULA.

With Dropbox I did something similar but now they do not have permanent links anymore - a Dropbox link lasts (at least for me) only for one day. 

The more professional setup would be to use AWS S3 or similar services. I'm sure Alibaba, Google Cloud or Azure have something similar - but I just know of S3. With S3 you need an AWS account, upload your files into a 'bucket', make them public and put the link into your HTML. The downside is that S3 and the others cloud services cost money. But if it's just a couple GB then it's really very cheap.

Hope this helps,

Kristian 

[Chi Zhang]

Hi Kristian,

This is exactly what I mean. Thank you so much for the answer!

Best regards,

Chi

To view this discussion on the web visit https://groups.google.com/d/msgid/jatos/5d3942f7-730f-4205-85dc-9f17f6c946b3n%40googlegroups.com.