Using NGINX proxy for SSL Vertx bodyhandler is not reading the requestbody

[Patrick Conway]

I have a vertx application for uploading files to a server.

I had some ssl issues with vertx only got a B rating on ssl labs... ios clients complaining about not secure enough... 

Since have experience with NGINX and SSL I am trying to use that as a proxy infront of vertx to provide the SSL

But my vertx application is not reading the body of the request coming from NGINX ... req.bodyHandler is not getting called in the code below. 

route.handler(routingContext -> {


            response = routingContext.response();
           
            HttpServerRequest req = routingContext.request();


            req.bodyHandler(h -> {


                logger.info("Handle body upload of size "+h.length());


            });


            response.end();

        });

   }

}

Anyone have any ideas on how to 

A ... solve the body issue 

B ... have a good guide on how to get vertx configured with ssl so that it can get a A+ rating on https://www.ssllabs.com/ssltest/analyze.html

I tried to remove the ciphers with the low rating but it caused a pain because in dev environment (mac) ...I found a set of ciphers that worked ....then I deployed to ec2 and those ciphers didn't work.

NGINX is much simpler as I can just give it a list of allowed ciphers

 

[Michel Guillet]

Patrick,

Regarding your SSL configuration : https://mozilla.github.io/server-side-tls/ssl-config-generator/

This is the configuration template I’m using in production.

Your vert.x code seems correct to me. You’ll need to show us more for us to help you. The full vertx http server instantiation and the nginx configuration would help.

Cheers,

Michel

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5417dc8b-1ac5-4f8b-921f-3bebd2e34b0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Patrick Conway]

here is the server portion of my nginx configuration

Its a default config other than this

upstream cloudbackup{

 

                server 127.0.0.1:8443;

        }



 

server {


 

        listen                  443;

        server_name             NAME_REMOVED;

        add_header Strict-Transport-Security "max-age=31536000; includeSubdomains" always;


 

        ssl on;

        ssl_certificate         /etc/nginx/certs/mycert.crt;

        ssl_certificate_key     /etc/nginx/certs/mykey.key;


 

        ssl_session_timeout  5m;

        ssl_prefer_server_ciphers On;


 

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;


 

        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';


 

        ssl_dhparam /usr/share/nginx/dhparams.pem;

client_max_body_size 0;

proxy_pass_request_headers on;

        location /{ proxy_pass https://cloudbackup;}

        }

My main verticle

package com.inhance.cloud.verticle;

import com.inhance.cloud.config.ConfigHelper;

import com.inhance.cloud.handler.BackupListHandler;

import com.inhance.cloud.handler.HMACAuthHandler;

import com.inhance.cloud.handler.Handler;

import com.inhance.cloud.handler.download.DownloadHandler;

import com.inhance.cloud.handler.download.ThumbnailHandler;

import com.inhance.cloud.handler.upload.UploadHandler;

import io.vertx.core.AbstractVerticle;

import io.vertx.core.http.HttpMethod;

import io.vertx.core.http.HttpServerOptions;

import io.vertx.core.logging.Logger;

import io.vertx.core.logging.LoggerFactory;

import io.vertx.core.net.PfxOptions;

import io.vertx.ext.web.Router;

import java.time.LocalDateTime;

import java.time.format.DateTimeFormatter;

/**

 * Created by patrick on 17/05/2016.

 */

public class HttpVerticle extends AbstractVerticle {

    Logger logger= LoggerFactory.getLogger(this.getClass());

    @Override

    public void start() throws Exception {

      

        Router router = Router.router(vertx);

   

        Handler authHandler=new HMACAuthHandler(router.route());

        authHandler.register();

        Handler uploadHandler=new UploadHandler(router,"/:tagcode", HttpMethod.POST);

        uploadHandler.register();

        vertx.createHttpServer().requestHandler(router::accept)

                .listen(ConfigHelper.httpPort());

        logger.info("Server Started at "+ LocalDateTime.now().format(DateTimeFormatter.BASIC_ISO_DATE));

        

       

    }

    

}

My Upload Handler

package com.inhance.cloud.handler.upload;

import com.inhance.cloud.handler.AbstractHandler;

import io.vertx.core.MultiMap;
import io.vertx.core.http.HttpMethod;
import io.vertx.core.http.HttpServerRequest;
import io.vertx.ext.web.Router;

/**
 * Created by patrick on 25/05/2016.
 */
public class UploadHandler extends AbstractHandler {

    public UploadHandler(Router router, String path, HttpMethod method) {
        super(router, path,method);
    }

    @Override
    public void register() {

        route.handler(routingContext -> {

            response = routingContext.response();

            String tagcode=routingContext.request().getParam("tagcode");
       




            MultiMap headers = routingContext.request().headers();

            HttpServerRequest req = routingContext.request();






            req.bodyHandler(h -> {

                logger.info("Handle body upload of size "+h.length());

                


            });

            response.end();

        });
    }
}

Without NGINX vertx handles the upload ok...

The problem I have with Vertx is around the cipher suites and the dhparam

Without setting those correctly I can't get IOS to accept the server as secure enough  (mainly cipher suites)

[Michel Guillet]

> location /{ proxy_pass https://cloudbackup;}

Try without https ?

Michel

[Patrick Conway]

tried without https

and tried different body types,
JSON
and
streaming bytes

same result… bodyhandler code not being called

> --
> You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/eXHU1Edj4k0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

> Visit this group at https://groups.google.com/group/vertx.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/B8E8AD23-8CBC-4CBF-9626-E985C80B86DD%40gmail.com.

[Inhance]

Also all the ciphers used with nginx and added them with opts.addEnabledCipher

this caused Exceptions re cipher not supported

So I went through the whole list eleminating unsupported ones

But when I deployed to ec2 the handful of ciphers supported on my mac were not supported on ec2

Sent from my iPhone

On 21 Jun 2016, at 17:02, Michel Guillet <michel....@gmail.com> wrote:

> --
> You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/eXHU1Edj4k0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

> Visit this group at https://groups.google.com/group/vertx.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/B8E8AD23-8CBC-4CBF-9626-E985C80B86DD%40gmail.com.

[Michel Guillet]

Does your class HMACAuthHandler call context.next() when its done ? If you’re doing some async treatment in it, don’t forget to pause the request and restart it when you’re done (context.request().pause()).

Also you’re using a custom class AbstractHandler that could be the cause of the issue. You might want to show us your code.

You should also keep an eye on nginx access and error logs.

I’m using nginx as a reverse proxy for vert.x in production for almost a year now, there is no reason we won’t find what’s wrong here.

Cheers,

Michel

> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

> Visit this group at https://groups.google.com/group/vertx.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/11029179-CF95-4A67-815D-B7F75AC82281%40inhancetechnology.com.

[Inhance]

Thanks Michel
The abstract handler is just providing some common methods

Ill have a look at the authhandler.
im checking auth headers and if the check passes I call context.next

Sent from my iPhone

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/9FBC5EDC-A940-4FA3-BFC3-ACE85717315A%40gmail.com.

[Patrick Conway]

package com.inhance.cloud.handler;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import io.vertx.core.http.HttpMethod;
import io.vertx.core.http.HttpServerResponse;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.ext.web.Route;
import io.vertx.ext.web.Router;

import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;

/**
 * Created by patrick on 19/05/2016.
 */
public abstract class AbstractHandler implements Handler {
    protected Logger logger= LoggerFactory.getLogger(this.getClass());

    protected HttpServerResponse response = null ;
    protected Route route=null;
    protected String path=null;

    public AbstractHandler(Router router, String path, HttpMethod method) {
        super();
        route=router.route(path).method(method);


    }

    public AbstractHandler(Route route) {
        super();
        this.route=route;
    }

    protected String decodeKey(String param) {
        String key="";
        try {
            key= URLDecoder.decode(param,"UTF8");
        } catch (UnsupportedEncodingException e) {
            logger.error(e.getMessage());
        }
        return key;
    }

    @Override
    public abstract void register();

    protected void writeJson(Object o) {
        try {


            String json;

            if(o instanceof String){
                json=(String)o;
            }
            else {
                ObjectMapper mapper = new ObjectMapper();
                mapper.enable(SerializationFeature.INDENT_OUTPUT);
                json = mapper.writeValueAsString(o);
            }


            response.headers().add("Content-Length",Integer.toString(json.length()));
            response.headers().add("Content-Type","application/json");
            response.write(json);
        } catch (JsonProcessingException e) {
            logger.error(e.getMessage(),e);
            response.setStatusCode(500);
        }
    }
}

This is the AbstractHandler

[raphae...@gmail.com]

Since you set client_max_body_size 0;, you probably expect large uploads. 

In this case, make sure to set proxy_request_buffering off; otherwise nginx will buffer the entire uploaded body before contacting the back-end server.

If this doesn't fix your issue, a simple way to check if it's nginx or vert.x misbehaving is to use http in your proxy_pass and start a nc -l 80 as your backend (or openssl s_server -msg -www if you want to debug with a SSL backend). If nc/openssl don't receive your requests, you definitely have an issue with nginx config.

-- raphael

[Patrick Conway]

Hi Raphael

Correct...I am expecting some uploads to be video files which will be large.

I have set request buffering off but it did not resolve the issue..

Also I have just tried nginx proxied to nc as the backend and I can see the request body coming through

[Michel Guillet]

Patrick, 

I’m running out of obvious reasons why this is not working.

The best thing to do is probably to gather a small git repo that you can share including the nginx config you’re using so that we can reproduce your issue locally and try to fix it.

Cheers,

Michel

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/78475634-01bd-48d1-99a0-ac0aacc771ae%40googlegroups.com.

[Raphaël Luta]

So to summarize:
- Vert.x works fine on its own and properly manages the uploads
- Nginx works fine and correctly proxies requests
- you don't have any error or warnings in you nginx logs
- you've already tested for the most obvious interop issues: disabling backend SSL, content-type mismatch, etc...

Next in line for me would be testing for some incorrect handling of "Expect: 100-continue" if present in request or maybe some issue caused by using HTTP/1.0 instead of HTTP/1.1 on the backend (set proxy_http_version 1.1; to test with a HTTP/1.1 backend connection)

If everything fails, disable SSL on the backend and take a full network trace during a small upload (like tcpdump -s 1500 -w /tmp/trace.pcap -i any port 443 or port 8443), this should give you some insight on the issue.

-- raphael

> --
> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
> Visit this group at https://groups.google.com/group/vertx.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/78475634-01bd-48d1-99a0-ac0aacc771ae%40googlegroups.com.

[Patrick Conway]

Thanks Raphael / Michel

So I've been doing some more testing...and the problem is related to the file size...

here is a github of the basic setup I am using

https://github.com/pconway11/vertx_nginx

I havent found the exact limit but a 1k file doesnt get into the body handler

But a 512 byte file does?

[Patrick Conway]

The magic number is 815 bytes

815 works  816 doesnt

[Paulo Lopes]

Could it be because you're finishing the response and not waiting for the body to be completly parsed?

I see that you set a body handler and right after you send some response and end the request... maybe you should do that once the body has been complete parsed right?

[Jez P]

You should definitely, as Paulo points out, be calling end() within your body handler, not before it has a chance to complete (where you have it at the moment). 

[Patrick Conway]

possibly, but I would have thought that should affect the direct call also?

When I send the file direct to vertx on port 8080 the full file is read...

When I send the file through nginx the bodyhandler doesnt seem to get called at all...even if it just a simple log statement in the body handler nothing is printed.

How can I delay the response until reading the request is finished?

On Thursday, June 23, 2016 at 12:36:16 PM UTC+1, Paulo Lopes wrote:

[Patrick Conway]

Eureka....

I moved the response inside the body handler and a 2k file was read 

[Paulo Lopes]

Alternatively use the BodyHandler from vertx-web: http://vertx.io/docs/vertx-web/java/#_request_body_handling

This will only call the next handler once the body has been parsed.

[Jez P]

Maybe nginx adds just a little delay which means the end() call occurs while still being read. Bodyhandler won't get called until entire body has been read. If the end() call occurs before this it won't. 

On Thursday, June 23, 2016 at 12:55:55 PM UTC+1, Patrick Conway wrote:

[Jez P]

+1 - vertx-web FTW, unless you really need to avoid the abstraction. Personally I'd start with vertx-web as a default. It makes life much easier. 

[Patrick Conway]

Great, thanks guys, your help is much appreciated!

[Patrick Conway]

I didnt want to use  BodyHandler as I dont want the files on disk....

As they come in I'm streaming them out into S3.