HTTP Digest Auth with WebClient

[Vincent Olivier]

Hello,

I am using the Vertx WebClient for interfacing with the monero-wallet-rpc json-rpc server as below :

HttpClientOptions hco = new HttpClientOptions();
hco.setProxyOptions(new ProxyOptions()
.setType(ProxyType.SOCKS5).setHost(PROXY_HOST).setPort(PROXY_PORT));

HttpClient httpclient = vertx.createHttpClient(hco);
WebClient client = WebClient.wrap(httpclient);
client.head(PORT, HOST, URI).send(ar -> {
    if(ar.succeeded()) {
        HttpResponse<Buffer> response = ar.result();
        parseAuth(response.getHeader("WWW-Authenticate"));
        client.get(PORT, HOST, URI)
                .putHeader(HttpHeaders.AUTHORIZATION.toString(), getAuth())
                .send(ar2 -> {
            if (ar2.succeeded()) {
                HttpResponse<Buffer> response2 = ar2.result();
                System.out.println(response2.statusCode());
            }
        });
    }
});

This always prints "401" and the parseAuth() and getAuth() mimic exactly the behaviour of Firefox and curl. On Firefox and curl, I am not getting a "401", but a "200" and by debugging I can see that the getAuth() methods returns exactly the same string as Firefox and curl. So I figured either I'm using the client in the wrong way or there is a bug in it.

So for now, I just want to know if there is something obviously wrong with it.

Thanks,

Vincent

[Julien Viet]

Hi,

I don't see anything wrong, however perhaps the proxy config affects it ?

JUlien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/829b28db-de01-4bad-bb4a-eec122d8a765%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vincent Olivier]

Hi !

Thanks. Is there a way to log the raw headers exactly as they are sent ?

Vincent

[Julien Viet]

you can log the network activity : http://vertx.io/docs/vertx-core/java/#logging_network_activity

if you want to see what really happens on the wire I recommend to use wireshark.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/50a7e88e-7847-49ef-bb1e-1f7b4963e3d2%40googlegroups.com.

[Alexander Lehmann]

Could you post your parseAuth/getAuth code as well?

Digest Auth should work with the proxy code, but maybe something is wrong in the sequence of the requests (which could be a vertx issue, not in your code)

[Julien Viet]

the proxy is using Socks5 and it's auth against the server origin and not the proxy as far as I can tell.

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f2a5e039-5484-4b4b-a041-7e360d76d009%40googlegroups.com.

[Vincent Olivier]

Hi,

Here is the code. I highly suspect a WebClient bug at this point because I can dump the raw request from Firefox and curl (also using the same SOCKS5 proxy) and for a given server nonce, they are identical.

   private String nonce;
    private long nc;
    private static final Pattern AUTH_PATTERN = Pattern.compile("Digest qop=\"auth\",algorithm=MD5,realm=\"monero-rpc\",nonce=\"(?<nonce>.+)\",stale=false");
    private void parseAuth(String header) {
        Matcher m = AUTH_PATTERN.matcher(header);
        if(m.matches()) {
            nc = 0;
            nonce = m.group("nonce");
        }
    }
    private String getAuth() {
        if(nonce != null) {
            final byte[] cnonceBytes = new byte[8];
            RANDOM.nextBytes(cnonceBytes);
            String clientNonce = bytesToHex(cnonceBytes);
            String nonceCount = String.format("%08x", ++nc);


            return getAuth(nonce, METHOD, clientNonce, nonceCount);
        } else return null;
    }
    private String getAuth(String nonce, String method, String clientNonce, String nonceCount) {
        String ha1 = md5(USER + ":" + REALM + ":" + PASSWORD);
        String ha2 = md5(method + ":" + URI);
        String response = md5(ha1 + ":" + nonce + ":" + nonceCount + ":" + clientNonce + ":" + QOP + ":" + ha2);


        return "Digest username=\"" +
                USER + "\", realm=\"" +
                REALM + "\", nonce=\"" +
                nonce + "\", uri=\"" +
                URI + "\", algorithm=" +
                ALGORITHM + ", response=\"" +
                response + "\", qop=" +
                QOP + ", nc=" +
                nonceCount + ", cnonce=\"" +
                clientNonce + "\"";
    }
    private MessageDigest md5;
    private String md5(String payload) {
        return md5(payload.getBytes());
    }
    private String md5(byte[] payload) {
        md5.reset();
        return bytesToHex(md5.digest(payload));
    }
    private static final char[] HEXADECIMAL = "0123456789abcdef".toCharArray();
    private static String bytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for ( int j = 0; j < bytes.length; j++ ) {
            int v = bytes[j] & 0xFF;
            hexChars[j * 2] = HEXADECIMAL[v >>> 4];
            hexChars[j * 2 + 1] = HEXADECIMAL[v & 0x0F];
        }
        return new String(hexChars);
    }

ps: this code is very server-specific, I have hard-coded the server auth header. But this part works.

[Alexander Lehmann]

I will try to try your code this evening, it looks about right though.

Could you post an example request/response from curl -V, that would be helpful to mock a server that checks the authentication by setting all random values to fixed strings and run auth against this server.

[Alexander Lehmann]

When I configure apache with mod_auth_digest, the code can successfully authenticate with or without socks5 proxy, so I guess the error must happen somewhere.

It would probably be helpful to check the network traffic between the client and the proxy and if possible between the proxy and the server with wireshark or tcpdump.

[Vincent Olivier]

Hi ! Thanks for doing those tests! I will look into it (see what the server receives on the other end). The proxy is Tor and it works via the TorBrowser just fine. I'm using the alpha version for the new onion services (which I am using, so that is indeed something to investigate). I haven't tried another Java client yet, will do that too, if I can't find anything about Tor. Thanks. Will be back with results.

[Vincent Olivier]

Hello !

I tried directly through TCP without the proxy and I still had the problem. I looked at the raw request with Wireshark and I got this :

Frame 5869: 407 bytes on wire (3256 bits), 407 bytes captured (3256 bits) on interface 0
Ethernet II, Src: AsustekC_cc:74:ec (ac:22:0b:cc:74:ec), Dst: FreebsdF_00:75:97 (58:9c:fc:00:75:97)
Internet Protocol Version 4, Src: 192.168.1.33, Dst: 192.168.1.29
Transmission Control Protocol, Src Port: 57207, Dst Port: XXXXXX, Seq: 1, Ack: 1, Len: 341
Hypertext Transfer Protocol
GET /json_rpc HTTP/1.1\r\n
User-Agent: Vert.x-WebClient/3.4.2\r\n
[truncated]Authorization: Digest username="XXXXXXXXX", realm="monero-rpc", nonce="wJq8irY/W+OzkudqRavGUQ==", uri="/json_rpc", algorithm=MD5, response="0fb39587acb684f57d7b9afc1ee0d6f0", qop=auth, nc=00000
Host: docker:XXXXX\r\n
\r\n
[Full request URI: http://docker:XXXXX/json_rpc]
[HTTP request 1/1]
[Response in frame: 5875]

What is this "[truncated]" could it be that the header value is too long (the username is VERY long) an that it truncates the header ? Because what is sent from vert.x is clearly truncated indeed.

[Julien Viet]

is that a label added by wireshark or what actually goes on the wire ?

On 20 Oct 2017, at 17:53, Vincent Olivier <vncn...@gmail.com> wrote:

Hello !

I tried directly through TCP without the proxy and I still had the problem. I looked at the raw request with Wireshark and I got this :

Frame 5869: 407 bytes on wire (3256 bits), 407 bytes captured (3256 bits)on interface 0

Ethernet II, Src: AsustekC_cc:74:ec (ac:22:0b:cc:74:ec), Dst:FreebsdF_00:75:97 (58:9c:fc:00:75:97)
Internet Protocol Version 4, Src: 192.168.1.33, Dst: 192.168.1.29

Transmission Control Protocol, Src Port: 57207, Dst Port: XXXXXX, Seq: 1,Ack: 1, Len: 341

Hypertext Transfer Protocol
GET /json_rpc HTTP/1.1\r\n
User-Agent: Vert.x-WebClient/3.4.2\r\n

[truncated]Authorization: Digest username="XXXXXXXXX", realm="monero-rpc", nonce="wJq8irY/W+OzkudqRavGUQ==", uri="/json_rpc", algorithm=MD5,response="0fb39587acb684f57d7b9afc1ee0d6f0", qop=auth, nc=00000

Host: docker:XXXXX\r\n
\r\n
[Full request URI: http://docker:XXXXX/json_rpc]
[HTTP request 1/1]
[Response in frame: 5875]

What is this "[truncated]" could it be that the header value is too long (the username is VERY long) an that it truncates the header ? Because what is sent from vert.x is clearly truncated indeed.

On Wednesday, October 18, 2017 at 8:25:22 PM UTC-4, Vincent Olivier wrote:

Hi ! Thanks for doing those tests! I will look into it (see what the server receives on the other end). The proxy is Tor and it works via the TorBrowser just fine. I'm using the alpha version for the new onion services (which I am using, so that is indeed something to investigate). I haven't tried another Java client yet, will do that too, if I can't find anything about Tor. Thanks. Will be back with results.

On Wednesday, October 18, 2017 at 6:53:12 PM UTC-4, Alexander Lehmann wrote:

When I configure apache with mod_auth_digest, the code can successfully authenticate with or without socks5 proxy, so I guess the error must happen somewhere.

It would probably be helpful to check the network traffic between the client and the proxy and if possible between the proxy and the server with wireshark or tcpdump.


On Wednesday, October 18, 2017 at 1:11:53 PM UTC+2, Alexander Lehmann wrote:

I will try to try your code this evening, it looks about right though.

Could you post an example request/response from curl -V, that would be helpful to mock a server that checks the authentication by setting all random values to fixed strings and run auth against this server.



On Tuesday, October 17, 2017 at 9:45:26 PM UTC+2, Vincent Olivier wrote:

Hi,

Here is the code. I highly suspect a WebClient bug at this point because I can dump the raw request from Firefox and curl (also using the same SOCKS5 proxy) and for a given server nonce, they are identical.

   private String nonce;
    private long nc;

    private static final Pattern AUTH_PATTERN =Pattern.compile("Digest qop=\"auth\",algorithm=MD5,realm=\"monero-rpc\",nonce=\"(?<nonce>.+)\",stale=false");

    private void parseAuth(String header) {
        Matcher m = AUTH_PATTERN.matcher(header);
        if(m.matches()) {
            nc = 0;
            nonce = m.group("nonce");
        }
    }
    private String getAuth() {
        if(nonce != null) {
            final byte[] cnonceBytes = new byte[8];
            RANDOM.nextBytes(cnonceBytes);
            String clientNonce = bytesToHex(cnonceBytes);
            String nonceCount = String.format("%08x", ++nc);


            return getAuth(nonce, METHOD, clientNonce,nonceCount);
        } else return null;
    }

    private String getAuth(String nonce, String method, StringclientNonce, String nonceCount) {

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c5e5b9a9-ecb3-40a9-9cb8-5c83dfbf5731%40googlegroups.com.

[Vincent Olivier]

I copied it verbatim (except for the XXXX) from what Wireshark gives me.

[Vincent Olivier]

How do I make sure it's the raw thing ? I'm not familiar with Wireshark so much.

.setType(ProxyType.SOCKS5).setHost(PROXY_HOST).setPort<span style="margin:0px;padding:0px;

[Julien Viet]

I've never seen so long header in wireshark too :-)

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/482d6e5e-6abb-45f4-aea4-30fc4888d022%40googlegroups.com.

[Alexander Lehmann]

[truncated] probably means that the packet was truncated during capture, I don't think happens on the actual tcp connection.

[Vincent Olivier]

I followed this : https://www.wireshark.org/lists/wireshark-users/201003/msg00155.html

And I indeed got the full header untruncated.

I will try a shorter username just in case.

[Vincent Olivier]

Tried a shorter user name to no avail. I really don't know what else to do. Without password protection, it works just fine.

hco.setProxyOptions(new</span

[Julien Viet]

can you make a reproducer for us ?

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/dba8efe2-b81a-4328-a98f-9e33dafc2b7f%40googlegroups.com.

[Alexander Lehmann]

Could you please post an example request/response captured with wireshark where the login fails, with a simple user and password?

It may be interesting to compare the nonce and other values to see if something may be missing.

The only other thing I can think of to narrow this down, maybe you could implement a client using a normal java socket connection (without vert.x and netty) to check that this works based on the values calculated by the client code.

[Alexander Lehmann]

Actually I think we need the response headers from the HEAD request and the request/response headers from the GET request that fails, this should be sufficient to create a reproducer by setting a fixed cnonce.

e.g. from my test client https://gist.github.com/alexlehm/f67189f857068ba68a02826421162997

[Vincent Olivier]

Thanks so much for your help. Will do that and be back soon. Thanks again.

[Alexander Lehmann]

I am able to reproduce the issue when running the rpc server on the local machine, I have created a project that tries the digest auth against the server on localhost

https://github.com/alexlehm/digest-auth-issue

[Julien Viet]

Hi Alex,

this is great!

have you had the opportunity to investigate it ?

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/18ea57e0-e00b-4be4-a911-074c9c28b789%40googlegroups.com.

[Alexander Lehmann]

I did some investigation but that was inconclusive until now.

When accessing the service with the vert.x WebClient, I believe the auth reponse is correct (it matches the one calculated by curl when setting the same nonce/cnonce values). but the auth isn't accepted.

I am beginning to suspect that the issue is not directly caused by the vert.x http implementation as the same problem happens when I do the digest auth in a perl script or with a normal java socket without vert.x, maybe there is something wrong in the timing of the requests with the monero-rpc server, but I have no idea how to narrow that down further.

[Alexander Lehmann]

I have found the cause of the problem and its not really a vert.x issue. When doing the digest auth operation, the login only works when the request that fetches the nonce and the request that sends the Authorization request are sent in the same server connection with HTTP Keep-Alive. When the 2nd request is sent on a new connection, the auth fails, probably since the nonce is not known anymore.

The same behaviour can be simulated by adding --header "Connection: close" to the curl request, then the client will use a new connection and the auth fails as well.

I believe it is currently not possible to get the HttpClient or WebClient to use the same connection for both requests as we do not keep the connection open with a timeout in the client pool. When setting the pool max size to 1, it still doesn't work since the 2nd request is sent when the 1st one was just finished and the connection was closed.

So I would assume you have two options to fix this, first you could write a bug report for the monero project and wait for that to be fixed, the second possibility would be to implement a minimal http client using NetClient, so you have control over the connection and could use a single connection for two requests.