FS is mounted but gives odd responses

[Matthew Halder]

I've gotten this pretty far but I'm wonder if I have some errors in the configurations that need to be cleaned up.  There are two screen shots attached, one is of the keywhiz-fs connecting to the server and mounting the filesystem.  The other screenshot is of the directory itself which, as you can see, is mounted.  But the `ls` command gives some very odd output.
There aren't any secrets in the Keywhiz server at this time except for what would have come in by default.  Are there some test I can run or configs I can verify to ensure that this is setup correctly?  Haven't tried adding my own secrets into the server yet but these errors kind of threw me for a loop.  LMK what I can do tweak/test this, thank you.

-Matt

[Matthew McPherrin]

kwfs is unable to talk to the server.

The important error is in the second screenshot:  The cert is valid for 127.0.0.1, not 172.20.0.51

You must issue a cert valid for the name you're connecting to.

--
You received this message because you are subscribed to the Google Groups "keywhiz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keywhiz-user...@googlegroups.com.
To post to this group, send email to keywhi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/c1633a5a-c537-4b08-a49b-2a7fd79fd312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Halder]

So I've gotten past the TLS/certificate issue (screenshot attached) but now I'm not sure the process for adding secrets to the keywhiz-fs.  I've added the CN=www.example5.com to the groups and added a default secret to it from Keywhiz.  However, I'm still getting the same "I/O errors" and "???? ?" when navigating to the /secrets directory.  I feel like I'm super close right now but not sure how to continue to proceed.  Also, is the logging location for Keywhiz server and keywhiz-fs by default?
Thanks all for the help.

-Matt

[Matthew McPherrin]

It seems like kwfs still can't talk to the server.

Here's an equivalent "curl" command line to what kwfs is trying to do.  You'll want figure out making that work before anything else.  Is the hostname running the server really "intermediate"?

curl --cert path/www.example5.com.cert.pem --key path/to/www.example5.com.key.pem --cacert path/to/ca-chain.cert.pem https://intermediate:4444/secrets

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/491c3eec-157f-40c0-94f8-752785a3d58f%40googlegroups.com.

[Matthew Halder]

The results were a bit too long for a visible screenshot so I'll copy/paste the results of the curl command.  And, yes, I named the keywhiz server to intermediate since that was the CN name on the cert (its just added to the host file to get ths PoC working).

--------------------Begin_Curl_Output--------------------
root@ip-172-20-2-22:/tmp# curl -vvvvvv --tlsv1 /usr/local/ca/certs/www.example5.com.cert.pem --key /usr/local/ca/private/www.example5.com.key.pem --cacert /usr/local/ca/certs/ca-chain.cert.pem --capath /usr/local/ca/ https://intermediate:4444
* <url> malformed
* Closing connection -1
curl: (3) <url> malformed
* Rebuilt URL to: https://intermediate:4444/
* Hostname was NOT found in DNS cache
*   Trying 172.20.0.51...
* Connected to intermediate (172.20.0.51) port 4444 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/local/ca/certs/ca-chain.cert.pem
  CApath: /usr/local/ca/
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-SHA256
* Server certificate:
*      subject: C=US; O=KeyWhizTestCA; OU=platform; CN=intermediate
*      start date: 2016-08-01 22:28:01 GMT
*      expire date: 2026-07-30 22:28:01 GMT
*      common name: intermediate (matched)
*      issuer: C=US; L=Denver; O=KeyWhizTestCA; OU=platformTest; CN=platformTest
*      SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: intermediate:4444
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 02 Aug 2016 17:54:17 GMT
< X-Content-Security-Policy: default-src 'self'
< X-XSS-Protection: 0
< X-Content-Type-Options: nosniff
< Frame-Options: DENY
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Location: https://intermediate:4444/ui/
< Content-Length: 0
<
* Connection #0 to host intermediate left intact

--------------------End_Curl_Output--------------------

Any ideas, I'm still seeing the same odd output once mounted.  Is the mount location on the server correct as just <host>:<port>?
Thanks,

Matt

[Matthew McPherrin]

Can you try with the /secrets/ at the end of the curl command?

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/ff5ecd8d-a67e-4c92-a2b9-0a23269a1c9b%40googlegroups.com.

[Matthew Halder]

Sure, it appears my last reply was deleted for some reason, sorry if this is a duplicate.

--------------------Begin_Curl_Output--------------------
root@ip-172-20-2-22:/tmp# curl -vvvvvv --tlsv1 /usr/local/ca/certs/www.example5.com.cert.pem --key /usr/local/ca/private/www.example5.com.key.pem --cacert /usr/local/ca/certs/ca-chain.cert.pem --capath /usr/local/ca/ https://intermediate:4444/secrets/

* <url> malformed
* Closing connection -1
curl: (3) <url> malformed

> GET /secrets/ HTTP/1.1

> User-Agent: curl/7.35.0
> Host: intermediate:4444
> Accept: */*
>

< HTTP/1.1 401 Unauthorized
< Date: Tue, 02 Aug 2016 18:19:38 GMT

< X-Content-Security-Policy: default-src 'self'
< X-XSS-Protection: 0
< X-Content-Type-Options: nosniff
< Frame-Options: DENY
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000; includeSubDomains

< WWW-Authenticate: ClientCert not authorized as a Client
< Content-Type: text/html; charset=ISO-8859-1
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Length: 298
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /secrets/. Reason:
<pre>    Unauthorized</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

* Connection #0 to host intermediate left intact

--------------------End_Curl_Output--------------------

Thanks again for all the help

-Matt

[Matthew McPherrin]

(forgot to reply-all, re-sending)

The important error message here is:

WWW-Authenticate: ClientCert not authorized as a Client

This means the certificate you're passing to kwfs (and curl) isn't marked as being a client cert.  You'll have to figure out how to make your CA do that.

We should also more clearly log what's going wrong here: We don't indicate *why* server requests are failing, it seems.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/932b89c4-4ca5-4add-bbc7-c0d3cd6ba08d%40googlegroups.com.

[Matthew Halder]

I've created a cert with the usr_cert extension (as opposed to the server_cert from before) and signed with the CA intermediary and verified it on the keywhiz-fs system.  Then added the the CN hostname to the Keywhiz Admin console as a client.  From there I added a group to the client so that it would have secrets to access.  After all that I'm still getting the same WWW-Authentication error when attempting to mount the intermediate:4444/secrets/.  The intermediate:4444 mounting works fine but the secrets portion is still a mystery as to what exactly it's checking, any thing else I could check here to see why the server is disallowing the client to mount?  I don't know where else to go from here to have this mount process succeed.

V/r,

Matt

[Matthew McPherrin]

In the server's log, do you see a message like "

logger.warn("Certificate does not contain CN=xxx,...: {}", principal.getName());

?  

If you run on your certificate:

openssl x509 -in your.crt -noout -text

Make sure it has

        Subject: CN=some-client-name

Also:

            X509v3 Extended Key Usage:

                TLS Web Client Authentication

Also, you should make sure it's signed by a CA trusted by the server, as configured in the trustStorePath in the keywhiz configuration file.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/4cbb38af-d61c-4a97-8e45-697e6284c3c0%40googlegroups.com.

[Matthew Halder]

I don't see any logging location on the server (not in $KEYWHIZ_HOME or /var/log) but I'm pretty positive the cert is good, here's the x509 output:

root@ip-172-20-2-22:/secrets# openssl x509 -noout -text -CA /usr/local/ca/certs/ca-chain4.cert.pem -in /usr/local/ca/certs/www.example7.com.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4104 (0x1008)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=KeyWhizTestCA, OU=platform, CN=intermediate
        Validity
            Not Before: Aug  2 21:16:02 2016 GMT
            Not After : Aug  2 21:16:02 2017 GMT
        Subject: C=US, ST=CO, O=Alice Ltd, OU=platform, CN=www.example7.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f6:7e:ce:ac:f9:49:75:08:71:68:69:f1:1f:89:
                    74:81:f8:25:96:13:c8:ad:51:3e:e8:41:c4:d8:84:
                    fa:a0:c6:56:a4:22:38:b9:b7:a0:6c:f1:51:26:39:
                    17:f9:0f:dd:1e:90:2f:20:91:60:0a:c5:c1:f5:4a:
                    a8:72:1d:b1:bb:05:f9:e1:84:65:dc:fe:05:31:f2:
                    99:d6:04:2d:85:a7:f0:e2:a4:7c:ef:ed:2b:05:7d:
                    3f:86:e2:60:d6:c2:1a:3e:ef:6c:89:cb:b4:56:68:
                    30:56:f5:1e:4e:cf:71:0b:40:39:90:e2:ae:c3:cf:
                    1f:a7:31:95:10:4f:97:e1:51:63:f7:c1:e1:8d:ba:
                    f8:2c:ab:35:21:24:ae:3c:fc:f0:bb:40:74:e8:61:
                    76:1c:90:30:f0:bf:41:9a:d2:86:0a:5f:89:4d:89:
                    fa:6e:7f:f5:62:b1:f2:84:47:71:18:5d:bd:14:a5:
                    fd:0a:2f:af:ef:00:c3:9f:f9:b0:37:95:e3:46:27:
                    fa:64:0b:45:88:c2:32:1e:d4:c8:e0:ba:7a:15:b7:
                    11:0d:a7:8f:fd:9f:fa:9d:67:64:5e:bf:11:1c:9e:
                    18:74:cf:e1:ea:3c:be:3c:d4:40:57:3e:27:23:08:
                    89:74:59:5e:ee:b6:3e:ed:24:db:ee:0b:9c:fa:7f:
                    53:29
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generted Client Certificate
            X509v3 Subject Key Identifier:
                C9:BC:E9:CD:70:0B:8C:CE:D1:99:B9:20:06:83:49:D5:E9:65:1E:ED
            X509v3 Authority Key Identifier:
                keyid:9B:8F:75:7D:1A:FD:B1:52:AF:3E:1A:D5:D9:41:B1:1F:4A:4F:12:B4

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
    Signature Algorithm: sha256WithRSAEncryption
         7d:bf:89:8e:7e:45:54:cd:46:88:9b:15:b6:f7:b8:98:d2:bf:
         3c:75:20:f6:39:3c:bd:48:f8:66:bd:a6:16:cf:46:0e:33:1f:
         b2:da:a6:6d:d2:71:69:1c:2c:ca:34:2e:8a:41:a0:c0:5d:80:
         75:ae:84:1d:41:33:cc:76:b9:49:29:9c:48:29:8b:13:eb:f8:
         b5:4e:e9:9c:b8:9e:97:f0:bf:cb:0e:33:b5:9f:66:77:07:16:
         9c:eb:7b:ba:7a:61:f5:89:7d:5a:b9:17:ba:0c:0e:7b:4a:fb:
         46:52:50:aa:e8:34:4b:6a:27:59:00:b8:21:27:5b:0f:ac:97:
         43:76:64:13:83:ec:d5:6f:2c:74:95:09:5c:6b:eb:51:aa:da:
         98:22:2b:a4:07:65:cc:bc:5b:24:15:53:ca:2b:ea:fb:f4:7f:
         0f:e8:b1:b5:05:38:fd:8c:48:70:8e:e0:a8:f9:e7:ea:68:3c:
         fa:89:cd:d7:8d:93:f2:51:9d:f4:1f:f6:57:d6:7c:87:df:97:
         69:d5:74:22:a1:db:2e:1e:6f:de:4c:0a:bc:30:75:46:97:32:
         7d:15:55:40:2b:a6:3f:a2:4b:a7:c3:49:1c:6a:04:15:ba:61:
         2a:9f:96:54:40:32:8c:2c:a6:18:21:60:1d:4e:fd:a8:a8:e2:
         3a:d4:cb:34:8a:23:a1:72:02:5f:02:ca:a2:52:4e:ec:6f:3b:
         3a:76:56:06:24:4a:ce:af:77:f7:db:b9:67:2d:65:16:7e:69:
         8a:f9:bb:f6:4b:92:cd:87:08:c7:02:32:93:ea:b7:4a:cf:8f:
         dd:b5:c9:e1:8e:65:06:83:e0:e1:76:66:0c:2d:d2:0e:1e:bb:
         8d:9c:16:68:8a:67:af:f6:da:8d:f5:72:a5:84:22:f6:b3:5a:
         e5:fb:98:59:0d:38:60:24:48:41:b3:41:a9:ab:c7:a8:45:b7:
         3d:b9:ac:20:d4:fc:6d:66:20:fe:25:f2:ee:b4:47:c4:7f:b1:
         27:4b:5a:83:81:ea:5a:71:b7:65:c6:60:82:41:a2:29:ea:46:
         20:01:8d:1c:a4:6b:27:34:c9:ec:ce:4e:09:a2:d2:00:b9:35:
         29:51:5a:f2:5b:89:d1:b9:5b:ef:e4:ed:67:21:d1:9f:e3:e4:
         6e:81:84:06:71:d0:6d:0d:9b:85:0e:df:36:5b:e0:80:98:bd:
         1a:08:2d:94:43:84:64:2c:97:76:ee:69:69:f3:c6:5d:6e:a4:
         df:03:8a:df:36:8e:96:82:05:03:4d:53:52:6b:13:6b:82:9d:
         f4:61:5d:f7:b0:f5:18:06:91:c8:84:09:a4:89:1c:01:cb:53:
         42:2e:ae:a6:0b:af:1c:74


p12 for the keystore and the jks for the truststore were both created using the same root/intermediate certs that the ca-chain4 was used to create. 

[Matthew McPherrin]

Hmm, that looks about right.  Do you have a log of the commands used to create the certs?  Maybe I could try to recreate the situation you're in.

The sample Keywhiz config file in the repo is set to log to its standard out:

https://github.com/square/keywhiz/blob/master/server/src/main/resources/keywhiz-development.yaml.mysql#L52-L55

If you want to log to files or otherwise, check out instructions at 

http://www.dropwizard.io/0.8.4/docs/manual/configuration.html#logging

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/092385ea-f4fb-4c6c-a683-db3850254764%40googlegroups.com.

[Matthew Halder]

I've attached a screenshot of the cert that's served up via the Keywhiz servers web interface just to verify that it's got the correct information on the other end.

[Matthew Halder]

As I began putting together this list it occurred to me import the example7 cert in the java keystore was probably needed (great exercise in troubleshooting, thanks for the pointer).  So I got the connection via keywhiz-fs with this command and output:

172-20-2-22:/usr/local/ca# sudo /home/ubuntu/work/bin/keywhiz-fs --asuser="root" --group="root" --key=/usr/local/ca/private/www.example7.com.key.pem --ca=/usr/local/ca/certs/ca-chain4.cert.pem --cert=/usr/local/ca/certs/www.example7.com.cert.pem https://intermediate:4444/ /secrets/kwfs
INFO kwfs_client[/secrets/kwfs]: 2016/08/03 01:47:19 GET /secrets 200 869.671167ms

And this is what I get doing a curl against the keywhiz server now:
ubuntu@ip-172-20-2-22:~$ sudo curl --cert /usr/local/ca/certs/www.example7.com.cert.pem --key /usr/local/ca/private/www.example7.com.key.pem --cacert /usr/local/ca/certs/ca-chain4.cert.pem https://intermediate:4444/secrets -vvv

* Hostname was NOT found in DNS cache
*   Trying 172.20.0.51...
* Connected to intermediate (172.20.0.51) port 4444 (#0)
* successfully set certificate verify locations:

*   CAfile: /usr/local/ca/certs/ca-chain4.cert.pem
  CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS handshake, CERT verify (15):

* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-SHA256
* Server certificate:
*      subject: C=US; O=KeyWhizTestCA; OU=platform; CN=intermediate
*      start date: 2016-08-01 22:28:01 GMT
*      expire date: 2026-07-30 22:28:01 GMT
*      common name: intermediate (matched)
*      issuer: C=US; L=Denver; O=KeyWhizTestCA; OU=platformTest; CN=platformTest
*      SSL certificate verify ok.

> GET /secrets HTTP/1.1

> User-Agent: curl/7.35.0
> Host: intermediate:4444
> Accept: */*
>

< HTTP/1.1 200 OK
< Date: Wed, 03 Aug 2016 01:52:05 GMT

< X-Content-Security-Policy: default-src 'self'
< X-XSS-Protection: 0
< X-Content-Type-Options: nosniff
< Frame-Options: DENY
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000; includeSubDomains

< Content-Type: application/json
< Vary: Accept-Encoding
< Content-Length: 467

<
* Connection #0 to host intermediate left intact

[{"name":"Nobody_PgPass","secret":"","secretLength":0,"creationDate":"2011-09-29T15:46:00.000Z","mode":"0400","owner":"nobody"},{"name":"Database_Password","secret":"","secretLength":0,"creationDate":"2011-09-29T15:46:00.000Z"},{"name":"General_Password","secret":"","secretLength":0,"creationDate":"2011-09-29T15:46:00.000Z"},{"name":"NonexistentOwner_Pass","secret":"","secretLength":0,"creationDate":"2011-09-29T15:46:00.000Z","owner":"NonExistant","mode":"0400"}]ubuntu@ip-172-20-2-22:~$

However, the odd output of listing the /secrets/ and /secrets/kwfs still remains.  Here's what I see:

ubuntu@ip-172-20-2-22:~$ ls -l /secrets/
ls: cannot access /secrets/kwfs: Input/output error
total 0
-????????? ? ? ? ?            ? kwfs
ubuntu@ip-172-20-2-22:~$ ls -l /secrets/kwfs
ls: cannot access /secrets/kwfs: Input/output error
ubuntu@ip-172-20-2-22:~$

Any ideas on how I can access these defaults secrets from the mount point? 

Thanks for all the help, no way I would have gotten this far without this user group.

-Matt

[Matthew Halder]

This is the success message from the keywhiz server side:

INFO  [2016-08-03 01:47:27,116] keywhiz.service.resources.SecretsDeliveryResource: Client www.example7.com listed available secrets.
172.20.2.22 - - [03/Aug/2016:01:47:26 +0000] "GET /secrets HTTP/1.1" 200 - "-" "Go-http-client/1.1" 489
INFO  [2016-08-03 01:50:27,368] keywhiz.service.resources.SecretsDeliveryResource: Client www.example7.com listed available secrets.
172.20.2.22 - - [03/Aug/2016:01:50:27 +0000] "GET /secrets HTTP/1.1" 200 467 "-" "curl/7.35.0" 21
INFO  [2016-08-03 01:52:05,304] keywhiz.service.resources.SecretsDeliveryResource: Client www.example7.com listed available secrets.
172.20.2.22 - - [03/Aug/2016:01:52:05 +0000] "GET /secrets HTTP/1.1" 200 467 "-" "curl/7.35.0" 10

For more options, visit <a href="https://groups.g

[Matthew McPherrin]

Okay, with the curl listing secrets, we have the certs and server set up properly.  What's the command line you're running kwfs with? The server arg should be https://server:port -- no path component at all.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/1b583f17-9a62-4edd-8250-74775cd0d2aa%40googlegroups.com.

[Matthew Halder]

Correct, I've attached two screenshots.  One is of running the keywhiz-fs using https://host:port versus keywhiz-fs https://host:port/secrets; the /secrets gets a 404 as it appears /secrets is implied by the keywhiz-fs command already.  Once the keywhiz-fs with https://host:port succeeds, the second screenshot is of the listing of the secrets directory and the attempt to access the mounted file.  It seems that the OS isn't able to read or access the mounted file system.  This is the last line when running the mount command on the host: "keywhiz-fs on /secrets/kwfs type fuse.keywhiz-fs (rw,nosuid,nodev,default_permissions,allow_other)"

Using curl with the same certs against the https://host:port of the Keywhiz server gives a 302.  Doing the same curl with https://host:port/secrets as the target produces a 200 and appends a JSON blob of data to the output.
It seems like the keywhiz/kw-fs is working but the OS doesn't like the fuse file system.  I'm running Ubuntu 14.04.4 LTS, with 3.13.0-91-generic kernel as an instance on aws. 

-Matt

[Matthew Halder]

After doing some exploring, I'd bet that there is something missing that's not allowing the OS/users to view or access the mounted keywhiz-fs mount point.  I've tried adding permissions, changing owners, and mounting the filesystem elsewhere using mount.fuse with no luck.  Not sure if something is needed in fstab file or where to go next but I didn't see anything in the fusermount, fuse, mount.fuse man pages that lead to get through this issue.  Any suggestions on how other users are able to read out the secrets from a keywhiz-fs mounted file?  Any help would be much appreciated.

-Matt

[Matthew McPherrin]

Can you do an "ls -lah" in the mountpoint?  

If the FS is working properly, it should show something like this:  (this has no secrets -- those would show up as files if they did)

ls -lah

total 0

drwxr-xr-x 3 keywhiz keywhiz 4.0K Aug  3 20:54 .

drwxr-xr-x 8 root    root     127 Aug  3 20:54 ..

-r--r----- 1 keywhiz keywhiz    0 Aug  3 20:54 .clear_cache

drwx------ 3 keywhiz keywhiz 4.0K Aug  3 20:54 .json

-r--r--r-- 1 keywhiz keywhiz    9 Aug  3 20:54 .running

-r--r--r-- 1 keywhiz keywhiz    3 Aug  3 20:54 .version

Try reading the mount as the user kwfs is running as.  I think we have something in all our systems that looks like:

$ cat /etc/fuse.conf

# The following line was added by keywhiz-fs

user_allow_other

I don't remember what failures look like without that, but it prevents access to users that aren't the one running the FS

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/be7e6718-ae5b-4c5c-b5df-d8283ef0a84a%40googlegroups.com.

[Matthew Halder]

Currently this is the response when keywhiz-fs connecting the keywhiz server:
ubuntu@ip-172-20-2-22:~$ sudo /home/ubuntu/work/bin/keywhiz-fs --asuser="root" --group="root" --key=/usr/local/ca/private/www.example7.com.key.pem --ca=/usr/local/ca/certs/ca-chain4.cert.pem --cert=/usr/local/ca/certs/www.example7.com.cert.pem https://intermediate:4444 /key/kwfs
INFO kwfs_client[/key/kwfs]: 2016/08/03 23:09:19 GET /secrets 200 101.54482ms

This is what I see when trying to access the (tried both normal and root user):
ubuntu@ip-172-20-2-22:~$ ls -l /key/
ls: cannot access /key/kwfs: Input/output error

total 0
-????????? ? ? ? ?            ? kwfs

ubuntu@ip-172-20-2-22:~$ ls -l /key/kwfs
ls: cannot access /key/kwfs: Input/output error
ubuntu@ip-172-20-2-22:~$ sudo -i

And here's the /etc/fuse.conf file:

root@ip-172-20-2-22:~# ls -l /key/
ls: cannot access /key/kwfs: Input/output error

total 0
-????????? ? ? ? ?            ? kwfs

root@ip-172-20-2-22:~# ls -l /key/kwfs
ls: cannot access /key/kwfs: Input/output error
root@ip-172-20-2-22:~#

And here's the /etc/fuse.conf file contents:
root@ip-172-20-2-22:~# cat /etc/fuse.conf
# /etc/fuse.conf - Configuration file for Filesystem in Userspace (FUSE)

# Set the maximum number of FUSE mounts allowed to non-root users.
# The default is 1000.
#mount_max = 1000

# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other

I think I'll try to add a keywhiz user to the keywhiz-fs system and see if that gets me anywhere.  It seems like the connection between the keywhiz client and server is there but the access to the mounted file system itself is where the issue is.

[Matthew Halder]

Adding the keywhiz user didn't alleviate the situation.

Starting keywhiz-fs:
ubuntu@ip-172-20-2-22:~$ su - keywhiz
Password:
keywhiz@ip-172-20-2-22:~$ sudo /home/ubuntu/work/bin/keywhiz-fs --key=/usr/local/ca/private/www.example7.com.key.pem --ca=/usr/local/ca/certs/ca-chain4.cert.pem --cert=/usr/local/ca/certs/www.example7.com.cert.pem https://intermediate:4444 /key/kwfs
[sudo] password for keywhiz:
INFO kwfs_client[/key/kwfs]: 2016/08/03 23:23:11 GET /secrets 200 93.461903ms

Attempting access to the filesystem:
root@ip-172-20-2-22:/home/keywhiz# su - keywhiz
keywhiz@ip-172-20-2-22:~$ ls -l /key/

ls: cannot access /key/kwfs: Input/output error
total 0
-????????? ? ? ? ?            ? kwfs

keywhiz@ip-172-20-2-22:~$ ls -l /key/kwfs

ls: cannot access /key/kwfs: Input/output error

keywhiz@ip-172-20-2-22:~$ ls -l /key/kwfs

ls: cannot access /key/kwfs: Input/output error

keywhiz@ip-172-20-2-22:~$ ls -l /key/

ls: cannot access /key/kwfs: Input/output error
total 0
-????????? ? ? ? ?            ? kwfs

keywhiz@ip-172-20-2-22:~$

I'm at a loss on how to continue troubleshooting, online the '???' usually indicate a permissions or ownership issue on the directory in question but this seems unlikely since it seems like root would still be granted access. 

[Matthew Halder]

Is there a process for prepping a keywhiz-fs mountpoint before running keywhiz-fs and mounting?  I can't seem to get the mount point to be readable or accessible but I can still see that it mounted.  Attached a screen shot of the mount volumes and the accessing the mount point attempts.

On Monday, August 1, 2016 at 6:26:22 PM UTC-6, Matthew Halder wrote:

[Matthew McPherrin]

Your screenshot looks like it ought to be set up fine.  I haven't seen errors like that before, so I'm not sure what's wrong.

Are any errors printed in the keywhiz-fs output when you run ls -lah?

--

You received this message because you are subscribed to the Google Groups "keywhiz-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to keywhiz-users+unsubscribe@googlegroups.com.

To post to this group, send email to keywhi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/740f1f14-2e4e-45e0-bd50-e4a15cd7ddaa%40googlegroups.com.