AWS/S3 configuration question about files.directory

56 views
Skip to first unread message

Jamie Jamison

unread,
Sep 14, 2018, 2:29:17 PM9/14/18
to Dataverse Users Community
Hi,

I'm running into some trouble with an AWS/S3 installation, using 4.9.2 on a RHEL AWS sewrver.

I've followed the directions in the "Configure Dataverse to use AWS/S3" Everything seems to work but when files are uploaded they promptly disappear from the display. 

The jvm options are set:
<jvm-options>-Ddataverse.files.storage-driver-id=s3</jvm-options>
 <jvm-options>-Ddataverse.files.s3-bucket-name=dataverse-test-site</jvm-options>

But I noticed that directory.files options is:
<jvm-options>-Ddataverse.files.directory=/usr/local/dvn/data</jvm-options>


Could that be part of the problem?

Errors below if that is helpful - it looks like it is trying to save to the local directory rather than the S3 bucket:
[#|2018-09-12T21:50:36.766+0000|WARNING|glassfish 4.1|javax.enterprise.system.container.web.com.sun.web.security|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789036766;_LevelValue=900;|
  Context path from ServletContext:  differs from path from bundle: /|#]

[#|2018-09-12T21:50:36.823+0000|INFO|glassfish 4.1||_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789036823;_LevelValue=800;|
  Loaded [1] org.ocpsoft.rewrite.spi.GlobalParameterProvider [org.ocpsoft.rewrite.instance.WildcardParameterProvider<0>]|#]

[#|2018-09-12T21:50:36.861+0000|WARNING|glassfish 4.1|org.primefaces.config.WebXmlParser|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789036861;_LevelValue=900;|
  DocumentBuilderFactory#setFeature not implemented. Skipping...|#]

[#|2018-09-12T21:50:37.206+0000|WARNING|glassfish 4.1|org.primefaces.config.WebXmlParser|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789037206;_LevelValue=900;|
  DocumentBuilderFactory#setFeature not implemented. Skipping...|#]

[#|2018-09-12T21:50:37.642+0000|INFO|glassfish 4.1|edu.harvard.iq.dataverse.authorization.groups.GroupServiceBean|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789037642;_LevelValue=800;|
  |#]

[#|2018-09-12T21:51:04.786+0000|INFO|glassfish 4.1|javax.enterprise.resource.webcontainer.jsf.faclets.facelet|_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789064786;_LevelValue=800;|
  Facelet[/dataset.xhtml] was modified @ 9:51:04 PM, flushing component applied @ 9:50:53 PM|#]

[#|2018-09-12T21:51:06.604+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789066604;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=dcterms&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:06.604+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789066604;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=ddi&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:06.605+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789066605;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=dataverse_json&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:06.605+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789066605;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=schema.org&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:06.684+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789066684;_LevelValue=900;|
  Response has already been committed, and further write operations are not permitted. This may result in an IllegalStateException being triggered by the underlying application. To avoid this situation, consider adding a Rule `.when(Direction.isInbound().and(Response.isCommitted())).perform(Lifecycle.abort())`, or figure out where the response is being incorrectly committed and correct the bug in the offending code.|#]

[#|2018-09-12T21:51:12.628+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789072628;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=dcterms&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:12.629+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789072629;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=ddi&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:12.629+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789072629;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=dataverse_json&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:12.629+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789072629;_LevelValue=900;|
  Skipping outbound rewriting of invalid URL: http://54.67.118.35 :8080/api/datasets/export?exporter=schema.org&persistentId=doi:10.5072/FK2/A6DT9H|#]

[#|2018-09-12T21:51:12.666+0000|WARNING|glassfish 4.1||_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789072666;_LevelValue=900;|
  Response has already been committed, and further write operations are not permitted. This may result in an IllegalStateException being triggered by the underlying application. To avoid this situation, consider adding a Rule `.when(Direction.isInbound().and(Response.isCommitted())).perform(Lifecycle.abort())`, or figure out where the response is being incorrectly committed and correct the bug in the offending code.|#]

[#|2018-09-12T21:51:20.408+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536789080408;_LevelValue=900;|
  Failed to process and/or save the file DTA900.file4787.ANIMALS.R2DAT.TXT; /usr/local/dvn/data/temp/tmp119958731845467819upload|#]

[#|2018-09-12T21:51:20.543+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=28;_ThreadName=http-listener-1(3);_TimeMillis=1536789080543;_LevelValue=900;|
  Failed to process and/or save the file DTA900.file4788.ANIMALS.R2.PDF; /usr/local/dvn/data/temp/tmp4140614355669249559upload|#]

[#|2018-09-12T21:51:20.680+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536789080680;_LevelValue=900;|
  Failed to process and/or save the file DTA900.file4789.PLANTS.R2DAT.TXT; /usr/local/dvn/data/temp/tmp4009080308446761643upload|#]

[#|2018-09-12T21:51:20.799+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=29;_ThreadName=http-listener-1(4);_TimeMillis=1536789080799;_LevelValue=900;|
  Failed to process and/or save the file DTA900.file4790.PLANTS.R2.PDF; /usr/local/dvn/data/temp/tmp4442003893027527855upload|#]


Thank you,

Jamie

Jamie Jamison

unread,
Sep 14, 2018, 3:51:01 PM9/14/18
to Dataverse Users Community
Someone kindly pointed out my typo:   invalid URL: http://54.67.118.35 :8080/api/datasets/export.    Found that and fixed though the problem remains.   Looks like the files are being written to the local directory /usr/local/dvn/data/temp/[files]  rather than the s3 bucket.

------------------------------------------------------- 

|2018-09-14T19:44:45.479+0000|WARNING|glassfish 4.1||_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536954285479;_LevelValue=900;|

  Response has already been committed, and further write operations are not permitted. This may result in an IllegalStateException being triggered by the underlying application. To avoid this situation, consider adding a Rule `.when(Direction.isInbound().and(Response.isCommitted())).perform(Lifecycle.abort())`, or figure out where the response is being incorrectly committed and correct the bug in the offending code.|#]



[#|2018-09-14T19:44:57.005+0000|WARNING|glassfish 4.1||_ThreadID=28;_ThreadName=http-listener-1(3);_TimeMillis=1536954297005;_LevelValue=900;|

  Response has already been committed, and further write operations are not permitted. This may result in an IllegalStateException being triggered by the underlying application. To avoid this situation, consider adding a Rule `.when(Direction.isInbound().and(Response.isCommitted())).perform(Lifecycle.abort())`, or figure out where the response is being incorrectly committed and correct the bug in the offending code.|#]



[#|2018-09-14T19:45:06.856+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=26;_ThreadName=http-listener-1(1);_TimeMillis=1536954306856;_LevelValue=900;|

  Failed to process and/or save the file DTA900.file4787.ANIMALS.R2DAT.TXT; /usr/local/dvn/data/temp/tmp6468067426457018456upload|#]



[#|2018-09-14T19:45:06.980+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=29;_ThreadName=http-listener-1(4);_TimeMillis=1536954306980;_LevelValue=900;|

  Failed to process and/or save the file DTA900.file4788.ANIMALS.R2.PDF; /usr/local/dvn/data/temp/tmp6145672697769131904upload|#]



[#|2018-09-14T19:45:07.215+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=28;_ThreadName=http-listener-1(3);_TimeMillis=1536954307215;_LevelValue=900;|

  Failed to process and/or save the file DTA900.file4789.PLANTS.R2DAT.TXT; /usr/local/dvn/data/temp/tmp171046977344108765upload|#]



[#|2018-09-14T19:45:07.378+0000|WARNING|glassfish 4.1|edu.harvard.iq.dataverse.EditDatafilesPage|_ThreadID=27;_ThreadName=http-listener-1(2);_TimeMillis=1536954307378;_LevelValue=900;|

  Failed to process and/or save the file DTA900.file4790.PLANTS.R2.PDF; /usr/local/dvn/data/temp/tmp1701295237670241692upload|#]

 

Jamie Jamison

unread,
Sep 14, 2018, 8:20:44 PM9/14/18
to Dataverse Users Community
The problem appears to have been solved by removing the jvm option: Ddataverse.files.directory=/usr/local/dvn/data  Although I don't know if it should have been replaced with something else.  

So far the problem seems to have been solved.

Jim Myers

unread,
Sep 18, 2018, 9:19:30 AM9/18/18
to Dataverse Users Community
FWIW: TDL is running a test instance using S3 with dataverse.files.directory defined. The S3 IO provider does use this property if defined to decide where to place temp files. With it defined, they go to <dataverse.files.directory>/temp and without they go to /tmp/files/temp.  If setting dataverse.files.directory causes trouble, I might suspect a permission problem on that dir or a missing dir, etc. S3 storage always creates temp files before sending data (uploads, unzipped files in a zip upload, etc.) on to the bucket (since the file size is needed) .

Jamie Jamison

unread,
Sep 18, 2018, 1:41:59 PM9/18/18
to Dataverse Users Community
What I'm not sure of is how to configure the temp directory for S3.  I've noticed is all the files seeming stored out side of the doi+shoulder directory structure.   Comparing this with our test that uses local storage - in that latter case, has the jvm file directory option - files starting with "tmp" are stored in temp and non files outside the directory structure.  Because I haven't defined the temp directory that is why all the files outside of the directory.

As it is probably obvious I'm still finding my way around AWS/S3. 
s3bucket-example.png

Jamie Jamison

unread,
Sep 18, 2018, 2:50:44 PM9/18/18
to Dataverse Users Community
Also, I am trying by trial-and-errors to code the dataverse.files.directory for s3.   So far no luck.  Is this documented anywhere?  Thanks


On Tuesday, September 18, 2018 at 6:19:30 AM UTC-7, Jim Myers wrote:

Jim Myers

unread,
Sep 18, 2018, 3:15:23 PM9/18/18
to Dataverse Users Community
I don't think it matters as long as it exists and the user running glassfish (usually the glassfish unix user) can write there. 
-Ddataverse.files.directory=/usr/local/dvn/data should work if that's the case (it exists and is writable). (Not having it set works because glassfish can write to /tmp/files/temp ...)
You could use something like 
-Ddataverse.files.directory=/usr/local/glassfish4/glassfish/domains/domain1/files
where the directory is within the glassfish domain too.

-- Jim
Reply all
Reply to author
Forward
0 new messages