Problem mounting hcp-openaccess using s3fs-fuse

[Zhenqi Liu]

Dear HCP team,

I was trying to mount hcp-openaccess s3 storage using s3fs-fuse but failed with "invalid credentials" (aws cli works fine).

I compiled s3fs-fuse github (v1.90+) and ran on linux server without root using commands like these,
s3fs hcp-openaccess ~/s3fs/hcp-openaccess -o profile=hcp
s3fs hcp-openaccess:/HCP_1200 ~/s3fs/hcp-openaccess -o profile=hcp
which failed with "s3fs.cpp:s3fs_check_service(3590): invalid credentials(host=https://s3.amazonaws.com) - result of checking service".

Adding "-o curldbg -d -d -f" to print the debug info shows that aws host returns 403 Forbidden / Access Denied / invalid credentials.

Additional details
- The official aws cli works fine for me "aws s3 ls s3://hcp-openaccess/ --profile=hcp" by listing the contents.
- I also tried s3fs-fuse with password file "-o passwd_file=.passwd-s3fs-hcp", same error..
- A possible related thread a few years ago seems to mention top-level folder object keys: https://www.mail-archive.com/hcp-...@humanconnectome.org/msg06623.html
- Using goofys (https://github.com/kahing/goofys) also result permission denied for me, there was an old open issue (https://github.com/kahing/goofys/issues/222)

Could you help me with this?

Thank you!

Zhen-Qi Liu

[Hodge, Michael]

Hi Zhenqui,

S3fs doesn't use the same credentials file as the AWS cli.  Maybe that's the problem?  It uses ~/.passwd-s3fs.  You'll want to make sure the access key in your .passwd-s3fs file is the same one that you're using in your AWS CLI credentials file.

Here's the command I used to successfully mount the HCP_1200 directory on my machine, with my current access keys stored in ~/.passwd-s3fs:

s3fs -o allow_other,uid=1000,gid=1000,umask=227,use_cache=/tmp,ro hcp-openaccess:/HCP_1200 /mnt/s3/HCP_1200 -d -d -f

Regards,

Mike

---

From: Zhenqi Liu <liuzh...@gmail.com>
Sent: Tuesday, September 14, 2021 2:49 PM
To: HCP-Users <hcp-...@humanconnectome.org>
Subject: [hcp-users] Problem mounting hcp-openaccess using s3fs-fuse

 

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/ef38251b-e827-43de-ba47-edca7b263fa4n%40humanconnectome.org.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Zhenqi Liu]

Thank you for the reply.

After some digging, it seems the error I met was caused by this open issue.

https://github.com/s3fs-fuse/s3fs-fuse/issues/1745

Best,

Zhen-Qi Liu