--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hello Sankar,The problem seems to be that the /tmp/data already exists in the root UFS. For instance, if your root UFS is the local filesystem and /tmp/data exists in the local filesystem, you might see this error.Could you try using a path that does not exist in the root UFS?Best,
On Fri, Sep 30, 2016 at 11:29 PM, <sankar.mittapally@creditvidya.com> wrote:Hi,--I am trying to mount my s3 bucket and getting failed with following error. Please tell me how to fix this.Alluxio - 1.2.0 versionJDK - 1.7./bin/alluxio fs mount /tmp/data/ s3n://sankarThriftIOException(message:Mount path /tmp/data shadows an existing path in the default underlying filesystem)
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Jiří Šimša
Sure, I am trying to mount S3 data to alluxio for fast response to Spark Cluster.ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"alluxio.creditvidya.com"}ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"1335MB"}ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/mnt/ramdisk"}ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"${ALLUXIO_HOME}/underFSStorage/"}export ALLUXIO_JAVA_OPTS+="-Dfs.s3n.awsAccessKeyId=$AWS_ACCESS_KEY-Dfs.s3n.awsSecretAccessKey=$AWS_SECRET_KEY"[root@alluxio conf]#I exported the AWS_ACCESS_KEY and AWS_SECRET_KEY in bashI used ./bin/alluxio fs mount /s3 s3n://sankarRegards
SankarOn Tue, Oct 4, 2016 at 8:20 PM, Jiří Šimša <ji...@alluxio.com> wrote:Hi Sankar,Can you share your alluxio-env.sh, the exact commands that you are running, and the errors you are seeing?On a related note, could you describe the goal you are trying to accomplish?Best,
On Tue, Oct 4, 2016 at 7:21 AM, Sankar Mittapally <sankar.mittapally@creditvidya.com> wrote:
Hi Jiri,I am able to mount S3 but some how it is not in visible state, showing 404 error even if I use the correct key.
+usersHi Sankar,So the "./bin/alluxio fs mount /s3 s3n://sankar" command was successful, but "./bin/alluxio fs ls /s3" is not showing the contents? Could you paste the output of those commands here?Alternatively, if you do not need to use the the local file system as UFS, you could try replacingALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"${ALLUXIO_HOME}/underFSStorage/"}withALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"s3n://sankar"}and restart Alluxio using the -f flag (this will format Alluxio)?Best,
Can you attach your master.log and user.log?
Please find latest logs.user_root.log----------------2016-10-04 15:54:56,913 INFO logger.type (Format.java:formatFolder) - Formatting JOURNAL_FOLDER:/home/alluxio/alluxio-1.2.0/journal/2016-10-04 15:54:56,936 INFO logger.type (Format.java:formatFolder) - Formatting BlockMaster_JOURNAL_FOLDER:/home/alluxio/alluxio-1.2.0/journal/BlockMaster2016-10-04 15:54:56,945 INFO logger.type (Format.java:formatFolder) - Formatting FileSystemMaster_JOURNAL_FOLDER:/home/alluxio/alluxio-1.2.0/journal/FileSystemMaster2016-10-04 15:54:56,947 INFO logger.type (Format.java:formatFolder) - Formatting LineageMaster_JOURNAL_FOLDER:/home/alluxio/alluxio-1.2.0/journal/LineageMaster2016-10-04 15:55:20,154 INFO reflections.Reflections (Reflections.java:scan) - Reflections took 356 ms to scan 1 urls, producing 5 keys and 75 values2016-10-04 15:55:20,201 INFO logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with FileSystemMasterClient master @ alluxio.creditvidya.com/172.31.27.102:199982016-10-04 15:55:20,214 INFO logger.type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ alluxio.creditvidya.com/172.31.27.102:199982016-10-04 15:55:20,318 INFO logger.type (BaseFileSystem.java:mount) - Mount /s3 to /master.log-----------------2016-10-04 15:54:57,601 INFO logger.type (MountTable.java:add) - Mounting /home/alluxio/alluxio-1.2.0/underFSStorage at /2016-10-04 15:54:57,602 INFO logger.type (AbstractMaster.java:start) - FileSystemMaster: Starting leader master.2016-10-04 15:54:57,602 INFO logger.type (JournalWriter.java:completeAllLogs) - Marking all logs as complete.2016-10-04 15:54:57,602 INFO logger.type (AbstractMaster.java:start) - FileSystemMaster: journal checkpoint does not exist, nothing to process.2016-10-04 15:54:57,602 INFO logger.type (JournalWriter.java:getCheckpointOutputStream) - Creating tmp checkpoint file: /home/alluxio/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp2016-10-04 15:54:57,604 INFO logger.type (JournalWriter.java:getCheckpointOutputStream) - Latest journal sequence number: 0 Next journal sequence number: 12016-10-04 15:54:57,611 INFO logger.type (JournalWriter.java:close) - Successfully created tmp checkpoint file: /home/alluxio/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp2016-10-04 15:54:57,611 INFO logger.type (JournalWriter.java:close) - Renamed checkpoint file /home/alluxio/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp to /home/alluxio/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data2016-10-04 15:54:57,611 INFO logger.type (JournalWriter.java:deleteCompletedLogs) - Deleting all completed log files...2016-10-04 15:54:57,611 INFO logger.type (JournalWriter.java:deleteCompletedLogs) - Finished deleting all completed log files.2016-10-04 15:54:57,878 INFO server.Server (Server.java:doStart) - jetty-7.x.y-SNAPSHOT2016-10-04 15:54:57,898 WARN http.MimeTypes (MimeTypes.java:<clinit>) - java.util.MissingResourceException: Can't find bundle for base name org/eclipse/jetty/http/mime, locale en_US2016-10-04 15:54:57,900 INFO handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.s.ServletContextHandler{/metrics/json,null}2016-10-04 15:54:58,169 INFO handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.w.WebAppContext{/,file:/home/alluxio/alluxio-1.2.0/core/server/src/main/webapp/},/home/alluxio/alluxio-1.2.0/core/server/src/main/webapp2016-10-04 15:55:06,639 INFO server.AbstractConnector (AbstractConnector.java:doStart) - Started SelectChann...@0.0.0.0:199992016-10-04 15:55:06,639 INFO logger.type (UIWebServer.java:startWebServer) - Alluxio Master Web service started @ 0.0.0.0/0.0.0.0:199992016-10-04 15:55:06,640 INFO logger.type (AlluxioMaster.java:startServing) - Alluxio master version 1.2.0 started @ alluxio.creditvidya.com/172.31.27.102:199982016-10-04 15:55:20,310 INFO logger.type (MountTable.java:add) - Mounting s3n://sankar/ at /s32016-10-04 15:55:20,317 INFO logger.type (JournalWriter.java:openCurrentLog) - Opened current log file: /home/alluxio/alluxio-1.2.0/journal/FileSystemMaster/log.out
[root@alluxio bin]# ./alluxio fs ls /s3[root@alluxio bin]#
On Tue, Oct 4, 2016 at 9:17 PM, Jiří Šimša <ji...@alluxio.com> wrote:
It looks like you ran a lot of mount and unmount commands and it is hard for me to understand the state your system is in.Would it be possible for you to:1) Remove all the log files2) Restart the system using the -f flag:"./bin/alluxio-start.sh all -f"3) Mount your S3 folder again:"./bin/alluxio fs mount /s3 s3n://sankar"4) List the folder using:"./bin/alluxio fs ls /s3"5) Attach the logs
+users
It seems like your S3 folder is mounted correctly. Did you also run the ls command? I would expect the user.log to contain more information if that was the case.
2016-10-04 09:27:08,117 WARN httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, ...
which helped me diagnose the issue. The ufs path that you are mounting needs to be a folder in your bucket. Please try mounting a folder under s3n://sankar instead.
Best,
Hi Sankar,
When I tried to reproduce your issue, I saw the additional entries in my master.log:2016-10-04 09:27:08,117 WARN httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, ...
which helped me diagnose the issue. The ufs path that you are mounting needs to be a folder in your bucket. Please try mounting a folder under s3n://sankar instead.
Best,
Instead of:./bin/alluxio fs mount /s3 s3n://sankardo:./bin/alluxio fs mount /s3 s3n://sankar/some_folderBest,
Well, the path has to exist.
Hi Jiri,I am able to mount S3 without issues. Thank you for your support, Do we have any documents on alluxio on spark cluster stand alone? If we have please provide me link.Thanks
Hi Jiri,I am able to mount S3 without issues. Thank you for your support, Do we have any documents on alluxio on spark cluster stand alone? If we have please provide me link.Thanks
On Tue, Oct 4, 2016 at 10:17 PM, Sankar Mittapally <sankar.m...@creditvidya.com> wrote:
That is there in S3 bucket :)
On Tue, Oct 4, 2016 at 10:17 PM, Jiří Šimša <ji...@alluxio.com> wrote:
Well, the path has to exist.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Jiří Šimša
--Regards
Sankar MittapallySenior Software Engineer
--Regards
Sankar MittapallySenior Software Engineer
--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
--Regards
Sankar MittapallySenior Software Engineer
--Regards
Sankar MittapallySenior Software Engineer