endHandler() never called when receiving a http post payment notification from WorldPay gateway

[Bruno Salmon]

hi everyone,

I'm developping a booking system based on Vert.x 2.1M5 and have to implement different payment gateways.

While it's working fine with other gateways, for any reason the endHandler is not called when receiving a payment notification from WorldPay system (a simple post with form attributes).

For example, the following simple post handler:

            public void handle(final HttpServerRequest req) {

                System.out.println("Receiving WorldPay notification");

                req.expectMultiPart(true);

                req.endHandler(new VoidHandler() {

                    @Override

                    protected void handle() {

                        System.out.println("endHandler called");

                    }

                });

            });

will trace the 'Receiving WorldPay notification' message but not 'endHandler called'. dataHandler and bodyHandler are not called neither. 

Here are the WorldPay post http headers:

Content-Type: application/x-www-form-urlencoded; charset=UTF-8

Content-Length: 896

Host=mybookingwebsite

User-Agent: WJHRO/1.0 (WorldPay Java HTTP Request Object)

WorldPay is the main payment system used by my company. I'm stuck...

Any idea why vert.x doesn't call the endHandler?

[Norman Maurer]

Maybe the transmitted content is not 896 bytes long ?

-- 
Norman Maurer

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bruno Salmon]

hi Norman,

That's a possibility indeed. If it is the case, is there workaround to get the received content in a buffer even before the 896 bytes are reached? I will be happy with any way to get the received information even if it's not very clean...

--

Bruno Salmon

PS: I already used your niosmtp library which is great, thank you so much for this :-) 

[Tim Fox]

Can you try with the latest master?

[Bruno Salmon]

I upgraded to vert.x 2.1RC1 but this doesn't fix the problem, I receive the post but the endHandler is still not called.

Actually I already had to face this problem with vert.x 1.3 and the only workaround I found was this:

public void handle(final HttpServerRequest req) {

                System.out.println("Receiving WorldPay notification");

                req.endHandler(new SimpleHandler() { // never called, don't know why...

                    protected void handle() {

                        // my business code (doesn't matter here)

                        req.headers().put("worldpay-done", "true"); // just to flag the job is done

                    }

                });

                vertx.setTimer(2000, new Handler<Long>(){  // checking 2s after WorldPay post

                    public void handle(Long timerID) {

                        if (!"true".equals(req.headers().get("worldpay-done"))) { // actually always happen                                        try {

                                System.out.println("Trying workaround");

                                // Ugly code to get the post content received by vert.x

                                java.lang.reflect.Field field = DefaultHttpServerRequest.class.getDeclaredField("request");

                                field.setAccessible(true);

                                DefaultHttpRequest request = (DefaultHttpRequest) field.get(req);

                                field = DefaultHttpMessage.class.getDeclaredField("content");

                                field.setAccessible(true);

                                ChannelBuffer content = (ChannelBuffer) field.get(request);

                                // Now getting the WorldPay form parameters

                                Buffer buff = new Buffer(content);

                                Map<String, String> parameters = new HashMap<>();

                                for (Map.Entry<String, List<String>> entry : new QueryStringDecoder(buff.toString(), false).getParameters().entrySet()) {

                                   parameters.put(entry.getKey(), entry.getValue().get(0));

                                   System.out.println(entry.getKey() + " = " + entry.getValue().get(0));

                                }

                                // my business code (doesn't matter here)

                                req.headers().put("worldpay-done", "true");

                            } catch (Exception e) {

                                CoreSystem.log(e);

                            }

                        }

                   });

               }

This code is ugly but at least it works with vert.x 1.3.

I couldn't find a similar workaround with vert.x 2.1

Not sure it's useful but here is the 'Callback Failure Alert From WorldPay' email I receive from WorldPay on each payment notification attempt:

Our systems have detected that your callback has failed.

This callback failure means we were unable to pass information

to your server about the following transaction:

   Transaction ID: xxx

   Cart ID: xxx

   Installation ID: xxx

   Error reported: Callback to: http://mybookingwebsite/payment-notification/: failed CAUSED BY Read timed out

   Server Reference: ukdc2-pz-cen07:callbackFailureEmail-12067:MerchReq-373-46

Also, if you usually return a response page for us to display to the Shopper

within the time allowed (1 minute), this will not have been displayed.

WorldPay will have displayed to the Shopper the response page file

(resultY.html or resultC.html) held for your installation on the WorldPay

server. This will be your own custom version, if you have supplied one, or,

if not, the WorldPay default version.

We hope this information will be of assistance. Please refer to the WorldPay

Knowledgebase - http://www.worldpay.com/support -

or contact your local Technical Support team if you want help or advice

about using the callback facility or about this particular callback failure.

If you would like to switch off the callback failure alerts or would like to

change the address to which these emails are sent, you can do so by

following the steps below on the Merchant Interface (WorldPay

Administration Server) - http://www.worldpay.com/admin.

- Log on to the Merchant Interface.

- Click on the "Configuration Options" button for the relevant installation.

- Edit the "Callback Failure Alert email address" field. If you wish to switch

  off the callback failure alerts, set the "Callback Failure Alert email

  address" field as blank.

- Make sure you click on "Save Changes" before leaving the page.

Thank you,

WorldPay Technical Support

Technical Support (Europe, Middle East & Africa)

mailto:sup...@worldpay.com

Technical Support (The Americas)

mailto:sup...@worldpay.com

Technical Support (Asia Pacific)

mailto:sup...@worldpay.com

[Tim Fox]

Maybe try and find out what "callback failure" means?

I'd also use curl or wireshark to inspect the response and make sure it really is the right length.

[Bruno Salmon]

When people come to the payment step, I redirect them to WorldPay website and once they have entered their cc details and made the payment, WorldPay calls me back (this is that callback they are talking about) to notify if the payment is successful or not (a quite essential information). WorldPay do it through a post request and expect me to reply by an acknowledgement within the next minute. Since I can't handle the post request, I can't get the payment state and reply with the acknowledgement. After that 1 minute timed out, WorldPay send me this message. This is what "callback failure" means.

I would be surprised if their post request doesn't respect the http standards since WorldPay is quite widely used - at least in the UK - but I will investigate and check the post request length. Thank you for suggesting the tools, I will write back when I will have this info.

mailto:...@worldpay.com

Technical Support (The Americas)

mailto:...@worldpay.com

Technical Support (Asia Pacific)

mailto:...@worldpay.com

[Tim Fox]

On 10/03/14 19:49, Bruno Salmon wrote:

hi everyone,

I'm developping a booking system based on Vert.x 2.1M5 and have to implement different payment gateways.

While it's working fine with other gateways, for any reason the endHandler is not called when receiving a payment notification from WorldPay system (a simple post with form attributes).

For example, the following simple post handler:

            public void handle(final HttpServerRequest req) {

                System.out.println("Receiving WorldPay notification");

                req.expectMultiPart(true);

Are you sure it's a multipart request?

                req.endHandler(new VoidHandler() {

                    @Override

                    protected void handle() {

                        System.out.println("endHandler called");

                    }

                });

            });

will trace the 'Receiving WorldPay notification' message but not 'endHandler called'. dataHandler and bodyHandler are not called neither. 

Here are the WorldPay post http headers:

Content-Type: application/x-www-form-urlencoded; charset=UTF-8

Content-Length: 896

Host=mybookingwebsite

User-Agent: WJHRO/1.0 (WorldPay Java HTTP Request Object)

WorldPay is the main payment system used by my company. I'm stuck...

Any idea why vert.x doesn't call the endHandler?

[Bruno Salmon]

I think so. Anyway I tried with req.expectMultiPart(false); but that doesn't make any difference

[Tim Fox]

What do you get in the dataHandler? Is it the correct length?

Have you looked at wireshark to see what is received on the wire?

[Bruno Salmon]

Only the requestHandler() of the httpServer is called when receiving this post request, I couldn't get any other handler (I tried them all) to be called after that initial one... So when I add a dataHandler to the received HttpServerRequest, it is never called.

I'm too busy to look at wireshark today, but I will try to do it in the next days and will report you the results.

[Norman Maurer]

Ok… Please provide us with a wireshark dump that captures such a request. 

-- 
Norman Maurer

[Bruno Salmon]

WorldPay is set to call our production server which currently runs on vert.x 1.3.1 with my ugly workaround I explained in a previous email (I use a timer when receiving the post call to then get the form attributes in a dirty way).

So I ran tcpdump on our production server until WorldPay made a call back regarding a customer payment and I extracted that post paquet as a pcap file with wireshark. This file contains personal information about the customer so I will send it to you as a separated private email.

Could you please have a look at it and tell me if the problem comes from WorldPay, Vert.x or if I did something wrong?

[Bruno Salmon]

Thank you Norman for your help which allowed me to replay the WorldPay http post on my development pc, so much easier now to look deeper...

I still have the problem even with the latest Vertx release (2.1RC3) but it's perhaps a misuse of the Vert.x API, I don't know. I was able to isolate and extract a simplified piece of code of our server code in order to reproduce the problem:

import org.vertx.java.core.Handler;

import org.vertx.java.core.VoidHandler;

import org.vertx.java.core.eventbus.Message;

import org.vertx.java.core.http.HttpServer;

import org.vertx.java.core.http.HttpServerRequest;

import org.vertx.java.core.http.RouteMatcher;

import org.vertx.java.platform.Verticle;

public class TestServer extends Verticle {

    @Override

    public void start() {

        Handler<HttpServerRequest> paymentNotificationHandler = new Handler<HttpServerRequest>() {

            @Override

            public void handle(HttpServerRequest req) {

                container.logger().info("Payment notification received, will the endHandler be called?");

                req.expectMultiPart(true);

                req.endHandler(new VoidHandler() {

                    @Override

                    protected void handle() {

                        container.logger().info("Yes!!!");

                    }

                });

            }

        };

        final RouteMatcher routeMatcher = new RouteMatcher();

        routeMatcher.all(".*/payment-notification/.*", paymentNotificationHandler);

        // other routes doesn't matter here...

        Handler<HttpServerRequest> webSecurityHandler = new Handler<HttpServerRequest>() {

            @Override

            public void handle(final HttpServerRequest req) {

                // Note that a simple call to routeMatcher.handle(req) would work here

                // But let's continue with the real scenario:

                // Extracting some security info from the request (session cookie, IP, etc...)

                Object securityInfo = "This logic doesn't matter here"; // let's simplify

                // Delegating the security check to the busSecurityHandler

                vertx.eventBus().send("security", securityInfo, new Handler<Message>() {

                    @Override

                    public void handle(Message message) {

                        routeMatcher.handle(req);

                    }

                });

            }

        };

        Handler<Message> busSecurityHandler = new Handler<Message>() {

            @Override

            public void handle(Message message) {

                message.reply(true); // always accept here (let's simplify...)

            }

        };

        vertx.eventBus().registerHandler("security", busSecurityHandler);

        HttpServer server = vertx.createHttpServer();

        server.requestHandler(webSecurityHandler);

        server.listen(8080);

    }

}

All http requests are handled by a webSecurityHandler that delegates the security check to a busSecurityHandler - a central handler for both web frontend and java backend that manages sessions, login, etc... once the busSecurityHandler accepted the request, the webSecurityHandler takes over with a call to the routeMatcher.

This works well for all traffic except the WorldPay notification callback (actually the only POST request in our traffic so far) where the endHandler() is never called.

I noticed in the above code that if the routeMatcher is called immediately in the webSecurityHandler, the endHandler() of the paymentNotificationHandler is correctly called. So the problem occurs when the call to the routeMatcher is delayed in the reply of the busSecurityHandler and in that case the endHandler() is never called.

I'm not sure to understand all subtleties of Vert.x and if the problem is a bug or a miseuse of the Vert.x API...

Any idea?

[Nick Scavelli]

Try pausing the request in your web security handler, then resuming in your payment one.

[Bruno Salmon]

Brilliant, adding what you suggested fixed the problem:

                req.pause();

                vertx.eventBus().send("security", securityInfo, new Handler<Message>() {

                    @Override

                    public void handle(Message message) {

                        req.resume();

                        routeMatcher.handle(req);

                    }

                });

I'm not sure to understand why it is needed but it's working and and this is what matters...

Thank you very much Nick and Norman!

[Nick Scavelli]

On Thursday, April 24, 2014 10:49:44 AM UTC-4, Bruno Salmon wrote:

Brilliant, adding what you suggested fixed the problem:

                req.pause();

                vertx.eventBus().send("security", securityInfo, new Handler<Message>() {

                    @Override

                    public void handle(Message message) {

                        req.resume();

                        routeMatcher.handle(req);

                    }

                });

I'm not sure to understand why it is needed but it's working and and this is what matters...

The http request body is being read before you set your data handler (since you're doing that asynchronously).

[Bruno Salmon]

I think I understood the point: the endHandler needs to be set immediately (I mean within the http server request handler thread) otherwise it's too late to be called (unless you paused the request as you suggested).

In my initial code (without pausing), the payment handler set the endHandler later in another thead (due to the call to the security bus handler) and it's too late, the request is already 'terminated' so the endHandler is never called.

Is that right?

Can I suggest to throw an exception in such a case (when trying to set an endHandler on a 'terminated' request)?

It would have saved me a lot of time if I had got such an exception

[Tim Fox]

On 24/04/14 11:59, Bruno Salmon wrote:

I think I understood the point: the endHandler needs to be set immediately (I mean within the http server request handler thread) otherwise it's too late to be called (unless you paused the request as you suggested).

In my initial code (without pausing), the payment handler set the endHandler later in another thead (due to the call to the security bus handler) and it's too late, the request is already 'terminated' so the endHandler is never called.

Is that right?

Kind of. But everything is called by the exact same thread, just later on. In the time between receiving the request and you setting the endHandler, 10000 other requests could have been handled.

[Norman Maurer]

Nice that we finally figured itt out. Spent quite some time on this over the week.

[Tim Fox]

This is actually a classic, and has come up several times in the past on this group ;)

[Bruno Salmon]

I meant within the http request handler call (and not thread, because you are right all calls are executed in the same thread...)

Is my suggestion to throw an exception in such case possible? (when calling req.endHandler())

I migrated our production server from Vertx 1.3 to 2.1 with Nick's solution and it's working :-)

Thank you again Norman, I know you spent some time on this.

                System.out.println("Receiving WorldPay notification");</fo

...

[Tim Fox]

On 24/04/14 12:38, Bruno Salmon wrote:

I meant within the http request handler call (and not thread, because you are right all calls are executed in the same thread...)

Is my suggestion to throw an exception in such case possible? (when calling req.endHandler())

That seems reasonable. Can you add a BZ?

--

[Bruno Salmon]

Ok, here is the new BZ: https://bugs.eclipse.org/bugs/show_bug.cgi?id=433459

...