Dataverse configuration to use AWS S3 storage not working

97 views
Skip to first unread message

Yash Gupta

unread,
Oct 20, 2017, 8:13:14 AM10/20/17
to Dataverse Users Community
Hi everyone,

I'm configuring my local Dataverse instance to work with Amazon S3 storage for storing my datasets. I was following the standard configuration steps as mentioned under the Dataverse installation guide:

I've correctly updated the storage driver environment variables for S3 storage & provided the AWS Access Key ID & Secret Access Key under AWS Configuration step.

But after configuring everything (& restarting GlassFish server), when I uploaded my dataset on the Dataverse dashboard, it isn't showing up in my S3 bucket.
Additionally, when I tried to download that dataset back to my machine, it showed the following error:

Internal Server Error- An unexpected error was encountered, no more information is available.

Can someone help me as in where I'm going wrong?

Thanks in advance.

Philip Durbin

unread,
Oct 20, 2017, 9:01:00 AM10/20/17
to dataverse...@googlegroups.com
Hi Yash,

I'm sorry to hear you're having trouble. I tried the S3 feature myself the other week and it seemed to work fine for me. It doesn't sound like you're doing anything wrong. It sounds like our Installation Guide might be unclear.

Whenever you see an Internal Server Error (hopefully rarely!) the place to look for more information is the Glassfish server log. If you accepted the defaults when you installed Dataverse, this file will be located at /usr/local/glassfish4/glas​sfish/domains/domain1/logs/server.log . We mention this at http://guides.dataverse.org/en/4.8/admin/troubleshooting.html but it's non-obvious from the error you see in a web browser.

Here's what would be most helpful.

- Attempt to upload a file (You say it isn't showing up on S3.).
- Attempt to download a file (I'm confused about how how are able to attempt to download the file from the Dataverse interface because I would hope that if the file failed to upload to S3 it would also fail to appear in the Dataverse interface. If it appears in the Dataverse interface, that means it has been saved to the database.).
- Attach a copy of server.log in an email to sup...@dataverse.org with a note about which version of Dataverse you are using (this is important so that developers can compare line numbers in the stack traces in the logs to lines of code for this or that version of Dataverse). This will create a support ticket. You could also reply on this thread with the file if you're not concerned about revealing the contents of the file publicly.

For extra credit, please go ahead and open an issue at https://github.com/IQSS/dataverse/issues/new because, again, it sounds like something is unclear in the Installation Guide.

Thanks,

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e8bf3511-4622-4de4-b3d5-4713aa1e4505%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

danny...@g.harvard.edu

unread,
Oct 20, 2017, 9:40:21 AM10/20/17
to Dataverse Users Community
Hi Yash,

It sounds like Phil has you covered on troubleshooting, but it's great to hear that you're investigating S3 as an option. You're one of the first, so thanks in advance for any feedback that you can provide! 

- Danny

Yash Gupta

unread,
Oct 20, 2017, 10:10:38 AM10/20/17
to Dataverse Users Community
Hi Phil,

As per your suggestions,

1. I tried to upload a sample datafile in a dataset to my local Dataverse instance, & it successfully got uploaded.
2. I tried to download that dataset, and after this step I faced the issue of "Internal Server Error".
3. I've mailed the GlassFish server log file to the recommended mail ID with all the details. 
And, my attempt for working with S3 storage was part of a test analysis that I was supposed to do on DataVerse, so I was working on a sample dataset that I grabbed from the Internet. And hence, no privacy issues ;) Hereby attaching the server log file with the thread.

And, I guess I'll probably proceed to open an issue for the same on your Github community as I'm in quick need of a solution.

Thanks for the quick response.

On Friday, October 20, 2017 at 6:31:00 PM UTC+5:30, Philip Durbin wrote:
Hi Yash,

I'm sorry to hear you're having trouble. I tried the S3 feature myself the other week and it seemed to work fine for me. It doesn't sound like you're doing anything wrong. It sounds like our Installation Guide might be unclear.

Whenever you see an Internal Server Error (hopefully rarely!) the place to look for more information is the Glassfish server log. If you accepted the defaults when you installed Dataverse, this file will be located at /usr/local/glassfish4/glas​sfish/domains/domain1/logs/server.log . We mention this at http://guides.dataverse.org/en/4.8/admin/troubleshooting.html but it's non-obvious from the error you see in a web browser.

Here's what would be most helpful.

- Attempt to upload a file (You say it isn't showing up on S3.).
- Attempt to download a file (I'm confused about how how are able to attempt to download the file from the Dataverse interface because I would hope that if the file failed to upload to S3 it would also fail to appear in the Dataverse interface. If it appears in the Dataverse interface, that means it has been saved to the database.).
- Attach a copy of server.log in an email to sup..@dataverse.org with a note about which version of Dataverse you are using (this is important so that developers can compare line numbers in the stack traces in the logs to lines of code for this or that version of Dataverse). This will create a support ticket. You could also reply on this thread with the file if you're not concerned about revealing the contents of the file publicly.

For extra credit, please go ahead and open an issue at https://github.com/IQSS/dataverse/issues/new because, again, it sounds like something is unclear in the Installation Guide.

Thanks,

Phil

On Fri, Oct 20, 2017 at 8:13 AM, Yash Gupta <yash.gu...@gmail.com> wrote:
Hi everyone,

I'm configuring my local Dataverse instance to work with Amazon S3 storage for storing my datasets. I was following the standard configuration steps as mentioned under the Dataverse installation guide:

I've correctly updated the storage driver environment variables for S3 storage & provided the AWS Access Key ID & Secret Access Key under AWS Configuration step.

But after configuring everything (& restarting GlassFish server), when I uploaded my dataset on the Dataverse dashboard, it isn't showing up in my S3 bucket.
Additionally, when I tried to download that dataset back to my machine, it showed the following error:

Internal Server Error- An unexpected error was encountered, no more information is available.

Can someone help me as in where I'm going wrong?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
server.log

Yash Gupta

unread,
Oct 20, 2017, 10:14:09 AM10/20/17
to Dataverse Users Community
Hi Danny,

It's a pleasure for me to be one of the first guys to be working on DataVerse S3 storage service. And, yeah I'll keep updating the thread now and then regarding the issues for the betterment of the community.

Thanks, 
Yash

Philip Durbin

unread,
Oct 20, 2017, 10:48:46 AM10/20/17
to dataverse...@googlegroups.com
Thanks for creating https://github.com/IQSS/dataverse/issues/4223 . I just left a comment for you there with more questions.

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/2c0c6dfe-73e7-4862-8651-8e667ea4065f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages