Adyen Notifications and Stored Payments

[kristia...@gmail.com]

Hi,

Pierre was able to help me get the HPP working, but now i'm having issues with notifications and storing payment details.

Prior to generating the HPP link, i create an empty payment method:

curl -v \
-u admin:password \
-H "X-Killbill-ApiKey: bob" \
-H "X-Killbill-ApiSecret: lazar" \
-H "Content-Type: application/json" \
-H "X-Killbill-CreatedBy: admin" \
-X POST \
--data-binary '{
"pluginName": "killbill-adyen",
"pluginInfo": {
"properties": []
}
}' \
"http://127.0.0.1:8080/1.0/kb/accounts/5655bc3f-e347-4cd1-8408-5663cc91be37/paymentMethods?isDefault=true"

I generate the HPP link using the following parameters:

country, recurringType (RECURRING), email, serverURL, resultURL, amount, currency

I'm able to successfully get to the HPP, make a payment, and have it authorized and immediately settled with Adyen.

Both the authorization webhook, and the capture hook, are sent to KillBill through our notification proxy. However, ONLY the Authorization event seems to be logged. The capture event is not recorded for some reason, and so the invoice is never marked as paid.

Here are screenshots:

A few payment attempts, all show 0. --- https://db.tt/RWkszpGQ
Payment details, same for all 3 --- https://db.tt/VYtkmTN1
Charge in Adyen --- https://db.tt/8MDovRaa

By the way, the actual invoice total is 7.42, so i'm not sure why it says 742.

Here are the related server log events:

2016-07-09T22:06:35,574+0000 lvl='INFO', log='AdyenNotificationService', th='catalina-exec-9', xff='', rId='0d5d2ae3-79f9-42f0-9e4a-5942a4eeeec5', aRId='', tRId='1', Handling notification: eventCode='AUTHORISATION', pspReference='8514681019943711', originalReference='null', success='true', reason='2324:1111:08/2018', merchantReference='b390c834-9903-4728-b3b6-9bef9213a697'
2016-07-09T22:06:35,598+0000 lvl='INFO', log='DefaultPaymentApi', th='catalina-exec-9', xff='', rId='0d5d2ae3-79f9-42f0-9e4a-5942a4eeeec5', aRId='1', tRId='1', ENTERING PaymentApi: transactionType='AUTHORIZE', accountId='154d2f1f-14d6-4857-92e2-9905c0e473f5', paymentMethodId='8aedb0c3-4a06-42a2-871d-a3c0f7962669', paymentExternalKey='b390c834-9903-4728-b3b6-9bef9213a697', paymentTransactionExternalKey='b390c834-9903-4728-b3b6-9bef9213a697', amount='742', currency='EUR'
2016-07-09T22:06:35,794+0000 lvl='INFO', log='DefaultPaymentApi', th='catalina-exec-9', xff='', rId='0d5d2ae3-79f9-42f0-9e4a-5942a4eeeec5', aRId='1', tRId='1', EXITING PaymentApi: transactionType='AUTHORIZE', accountId='154d2f1f-14d6-4857-92e2-9905c0e473f5', paymentMethodId='8aedb0c3-4a06-42a2-871d-a3c0f7962669', paymentExternalKey='b390c834-9903-4728-b3b6-9bef9213a697', paymentTransactionExternalKey='b390c834-9903-4728-b3b6-9bef9213a697', paymentId='db66f2f4-6b9c-4689-9f53-8f0bbc8a3ee3', transactionId='965edacb-6fc5-4b88-9379-735f8b2b877c', amount='742.00', currency='EUR', transactionStatus='SUCCESS'

2016-07-09T22:10:57,839+0000 lvl='INFO', log='AdyenNotificationService', th='catalina-exec-9', xff='', rId='3ac4629f-7697-4ae8-ba31-25d84ffc0af4', aRId='', tRId='1', Handling notification: eventCode='CAPTURE', pspReference='8814681022186404', originalReference='8514681019943711', success='true', reason='', merchantReference='b390c834-9903-4728-b3b6-9bef9213a697'
2016-07-09T22:10:57,870+0000 lvl='INFO', log='LoggingFilter', th='catalina-exec-9', xff='', rId='3ac4629f-7697-4ae8-ba31-25d84ffc0af4', aRId='1', tRId='1', 391 * Server out-bound response
391 < 200
391 < Content-Type: application/json
391 < Vary: Accept-Encoding
391 < Content-Encoding: gzip
391 <

And finally, and i'm not sure if the Adyen Plugin is supposed to do this, but it's not storing any data locally. It should store some values to make future payments possible. Is this something that our app needs to do using the API?

I really appreciate any help or pointers you can give me.

Thanks,
Kris

[Pierre-Alexandre Meyer]

Hi again Kris,

It seems like you are you using Adyen Hosted Payment Pages for credit card payments? If that's the case, this could explain the source of your various issues, as I don't believe this type of integration has been tested in the past. We've only used their Certified API for PCI compliant merchants to trigger credit card payments and HPP for all other payment methods (e.g. PayPal). Because Adyen doesn't support auth/capture on HPP (for non credit card payment methods), the plugin assumes an HPP payment will be recording in Kill Bill as a PURCHASE (i.e. auto-capture) transaction.

That being said, the code is probably not far off, so let's try to make it work.

First, regarding the payment method: Adyen requires indeed an initial payment with a recurringType to tokenize the card (you cannot tokenize separately). Once it's tokenized (it usually takes a few seconds in Adyen after the payment), you can sync the payment method in Kill Bill with the token in Adyen by calling the resetPaymentMethods plugin API (http://docs.killbill.io/0.16/payment_plugin.html#_paymentpluginapi_overview), exported as POST /1.0/kb/accounts/<accountId>/paymentMethods/refresh.

In more details, this would:

• Query the plugin API getPaymentMethods with refreshFromGateway=true: the plugin should call the recurring endpoint in Adyen
• Call the plugin API resetPaymentMethods to let the plugin know about the created payment method ids (the plugin will update its mappings token <-> payment method id)

You can take a look at this test to understand how it works from the plugin perspective: https://github.com/killbill/killbill-adyen-plugin/blob/90a648623e9a7937a78a5c4f17c649d2a47a077a/src/test/java/org/killbill/billing/plugin/adyen/api/TestAdyenPaymentPluginApi.java#L117-L161

Note that for this to work, the easiest is to have the Kill Bill account external key mapping to the shopper reference in Adyen (if that's not the case and/or you cannot make it work in your environment, the alternative is to pass in all JAX-RS calls the shopper reference as the plugin property customerId).

Regardless of the capture being recorded on the Kill Bill side, I believe this should work today already. Could you try triggering the refresh and verifying the recurring details id (i.e. token) is stored in the plugin payment methods table? Future payments against that payment method should work (auth, capture, etc.).

Second, regarding the HPP with credit cards flow, I've pushed a patch that should fix it: https://github.com/killbill/killbill-adyen-plugin/commit/4a118259861be92cdf6d1845f0f356353629f418

To build the Adyen plugin, simply run: mvn clean install -DskipTests=true. The plugin can then be found under target/adyen-plugin-0.3.3-SNAPSHOT.jar. Replace the current one with it (under /var/tmp/bundles/plugins/java/adyen-plugin/...) and restart Kill Bill.

Could you retry with the latest version? Bear with me, we might have to go back and forth on that one due to the complicated nature of the plugin (it supports lots of flows and various configurations, and the asynchronous nature of the Adyen API doesn't help) and because I didn't have an environment setup to try it. If you still have issues, Kill Bill logs definitively help, but it would also be useful to get the entries in the adyen_* tables of the plugin, as well as the payment_* tables in Kill Bill.

Finally, regarding the payment amount: Adyen amounts are in cents and we were storing them as cents in the table adyen_notifications (used mainly for debugging). I've changed this so that the notification correctly sets the right amount in Kill Bill: https://github.com/killbill/killbill-adyen-plugin/commit/ffc2400388826c9e24d299afaf3d28ae10065969

Good luck!

--
You received this message because you are subscribed to the Google Groups "Kill Bill users mailing-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to killbilling-us...@googlegroups.com.
To post to this group, send email to killbill...@googlegroups.com.
Visit this group at https://groups.google.com/group/killbilling-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/1b24aa15-9491-4bf1-9230-eff49aa9d44b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pierre

[kristia...@gmail.com]

Thanks for all the work! I think we're close.

Here is a zip file with screenshots and log data: https://db.tt/A3Y8y4RV

The payment method refresh appears to work now, and killbill is able to initiate payments from previous HPP recurring payments.

The initial payment through the HPP is not quite working yet. It's recording the capture payment, but not handling that correctly.

Here are the issues i'm seeing:

1) The capture responses show as 'pending' initially, and then moves to payment failure. In the zip file i included, you'll see the first database output for notification #3 shows the capture received, and additional data is null. I guess at some point later some processing happens, and the next screenshot shows additional data with the output from adyen. At that point, you'll see the payment has failed in Kaui.

2) The payment list still shows a value of 0 for the amount. On that screenshot, you'll see that the last payment (#3) has the correct value. This payment was done through Killbill directly, not from the HPP responses.

I should also note that Adyen does not send Capture notifications for payments that are automatically captured. So initially I had our adyen merchant settings set to capture immediately, but that led to the issue that KillBill would only receive the initial authorized response. I had thought that's why the invoice was still not being marked as paid, but now i see that's the case for Capture as well.

Do I need to build something outside of killbill that will go through and initiate a capture request on all HPP payments so Killbill will get the notification? Or can KillBill be configured to see that initial authorization as a successful payment and allow me to use the auto-capture feature within Adyen?

I really appreciate your help!

-Kris

On Saturday, July 9, 2016 at 6:18:42 PM UTC-7, Pierre-Alexandre Meyer wrote:
> Hi again Kris,
>
>
> It seems like you are you using Adyen Hosted Payment Pages for credit card payments? If that's the case, this could explain the source of your various issues, as I don't believe this type of integration has been tested in the past. We've only used their Certified API for PCI compliant merchants to trigger credit card payments and HPP for all other payment methods (e.g. PayPal). Because Adyen doesn't support auth/capture on HPP (for non credit card payment methods), the plugin assumes an HPP payment will be recording in Kill Bill as a PURCHASE (i.e. auto-capture) transaction.
>
>
> That being said, the code is probably not far off, so let's try to make it work.
>
>
> First, regarding the payment method: Adyen requires indeed an initial payment with a recurringType to tokenize the card (you cannot tokenize separately). Once it's tokenized (it usually takes a few seconds in Adyen after the payment), you can sync the payment method in Kill Bill with the token in Adyen by calling the resetPaymentMethods plugin API (http://docs.killbill.io/0.16/payment_plugin.html#_paymentpluginapi_overview), exported as POST /1.0/kb/accounts/<accountId>/paymentMethods/refresh.
>
>
>
> In more details, this would:

> Query the plugin API getPaymentMethods with refreshFromGateway=true: the plugin should call the recurring endpoint in AdyenCall the plugin API resetPaymentMethods to let the plugin know about the created payment method ids (the plugin will update its mappings token <-> payment method id)

[Pierre-Alexandre Meyer]

On Sun, Jul 10, 2016 at 3:37 PM, <kristia...@gmail.com> wrote:

1) The capture responses show as 'pending' initially, and then moves to payment failure. In the zip file i included, you'll see the first database output for notification #3 shows the capture received, and additional data is null. I guess at some point later some processing happens, and the next screenshot shows additional data with the output from adyen. At that point, you'll see the payment has failed in Kaui.

Could you give https://github.com/killbill/killbill-adyen-plugin/commit/0de6d3bc83bf670037771834aa5cc9816c814eeb a try?

There was an issue with my patch causing the capture notification to trigger another (different) capture in Adyen, instead of simply recording the notification and transitioning the status accordingly. Hopefully, that should fix it. I tried to reproduce the requests and notifications XML from your logs - hope I got it right!

Do I need to build something outside of killbill that will go through and initiate a capture request on all HPP payments so Killbill will get the notification? Or can KillBill be configured to see that initial authorization as a successful payment and allow me to use the auto-capture feature within Adyen?

Kill Bill supports PURCHASE transactions, which represent payments that are auto-captured: http://docs.killbill.io/0.16/userguide_payment.html#_payment_states. Recurring payments triggered by the invoicing module will always be recorded as PURCHASE for example.

If Adyen is configured to auto-capture (and won't send an additional CAPTURE notification), the payment should be recorded as a PURCHASE instead of separate AUTH/CAPTURE in Kill Bill. To do so, pass the plugin property authMode=false (i.e. ?pluginProperty=authMode=false in the query parameters) when calling buildFormDescriptor.

(The plugin already understood the property, but in a different scenario. I've updated it so that it works also in your setup, see https://github.com/killbill/killbill-adyen-plugin/commit/761213689597c38c6baf9fd13e80e117c1b76ade)

Once we get it working in your setup, could you help me enhance our documentation (a PR to update https://github.com/killbill/killbill-adyen-plugin/blob/master/README.md would be awesome)? Since I don't have a fully working environment similar to yours, I want to make sure we capture all the cURLs, configuration steps, etc.

Thanks!

--

Pierre

[Pierre-Alexandre Meyer]

On Mon, Jul 11, 2016 at 5:49 PM, Kris Watson <kristia...@gmail.com> wrote:

I see the adyen plugin was updated to use the new adyen key type, sha256, however the merchantSig calculation does not seem to be correct. I looked through the commits for the java adyen plugin, and the only thing that jumped out at me is that it appears the plugin is only hashing specific fields. I do know that the new method requires all submitted fields to be included in the signature. 

Sorry about that - when I merged the PR, I hadn't realized that this would change the default behavior. sha1 is still supported (you can configure it by setting the property org.killbill.billing.plugin.adyen.hmac.algorithm=HmacSHA1).

Regarding the errors, I've added Andre, who submitted the PR. Could you provide an example of fields you submit? We could add this as a regression test.

For my project, it's not an issue as I just included the Adyen ruby gem and generate the signature on the fly, but i'm sure others will have issues with it. 

    def adyen_signature(data)

      data.delete("merchantSig")

      data.merge!('sharedSecret' => my_hmac_key

      Adyen::Signature.sign(data)

    end

FWIW, while this works of course, it's kind of a Kill Bill anti-pattern: one of the goals of the platform is to abstract the gateways from the API user (as much as possible). It's especially important when using advanced functionalities like payment routing (you don't always know against which gateway the payment will be submitted).

1) HPP + Server notifications are 90% there! They are processed correctly as purchases (using the authMode=false), however the payment amount isn't actually being applied to anything. Whether it's to the outstanding invoices, or as a credit on the account.

I think I know why but could you describe in more details your scenario, the flow (how does your user tokenize the cart, at what point does he subscribe, what kind of invoice is generated, etc.) and the Kill Bill APIs you are using for this?

2) On a test using a Maestro card with 3d secure, the manual payment (after the token was synced) failed with an error of "configuration 905 Payment details are not supported". This generates the following killbill error: Error while communicating with the Kill Bill server: Error 500: java.sql.SQLDataException: Data too long for column 'gateway_error_code' at row 1

Interesting - are you running MySQL in strict mode? We probably never saw it because the column was silently truncated in our tests. Should be fixed by https://github.com/killbill/killbill-adyen-plugin/commit/fa88297067b601fca41207958e8a1e73bcc23733.

Regarding the error itself, are you completing the 3D-S flow in one go (buildFormDescriptor -> Adyen HPP -> 3D-S redirect -> Kill Bill/website redirect)? When using the PCI APIs, there is an extra step (extra API call) required to "complete" the 3D-S payment, I'm not familiar how it works with HPP though. Are you sure Adyen sees the payment method as validated?

Also, you mention a token - does Adyen let you create a recurring contract with a Maestro card? I don't think you can trigger recurring payments with Maestro unless you are part of MARP.

3) I created a new HPP request using a standard visa card, updated the token in the database, and then tried again. I was able to create a manual payment once, but then all subsequent charges failed....

Same error as above in killbill, but in the adyen_responses table, I can see this: "{"exceptionMessage":"validation 803 PaymentDetail not found","exceptionClass":"org.apache.cxf.binding.soap.SoapFault","adyenCallErrorStatus":"RESPONSE_ABOUT_INVALID_REQUEST"}"

So, you can re-use the token once, then it fails? That's odd indeed. Could you share the full Kill Bill logs, including the XML requests to/from Adyen (you can mask the headers, and any sensitive information)?

--

Pierre

[Kris Watson]

Hi Pierre,

Here is a new zip file containing logs / screenshots, etc: https://db.tt/GPIh4Dop

I extracted portions of our app that builds the HPP pages and handles payments to show you what we're doing.

On Tue, Jul 12, 2016 at 9:06 AM, Pierre-Alexandre Meyer <pie...@kill-bill.org> wrote:

On Mon, Jul 11, 2016 at 5:49 PM, Kris Watson <kristia...@gmail.com> wrote:

I see the adyen plugin was updated to use the new adyen key type, sha256, however the merchantSig calculation does not seem to be correct. I looked through the commits for the java adyen plugin, and the only thing that jumped out at me is that it appears the plugin is only hashing specific fields. I do know that the new method requires all submitted fields to be included in the signature. 

Sorry about that - when I merged the PR, I hadn't realized that this would change the default behavior. sha1 is still supported (you can configure it by setting the property org.killbill.billing.plugin.adyen.hmac.algorithm=HmacSHA1).

Regarding the errors, I've added Andre, who submitted the PR. Could you provide an example of fields you submit? We could add this as a regression test.

No problem. Here are all the fields that i'm including to get a correct signature: currencyCode, shopperReference, countryCode, skinCode, merchantReference, shopperEmail, shopper.firstName, merchantAccount, shipBeforeDate, paymentAmount, shopperLocale, sessionValidity, resURL, recurringContract, shopper.lastName

 

For my project, it's not an issue as I just included the Adyen ruby gem and generate the signature on the fly, but i'm sure others will have issues with it. 

    def adyen_signature(data)

      data.delete("merchantSig")

      data.merge!('sharedSecret' => my_hmac_key

      Adyen::Signature.sign(data)

    end

FWIW, while this works of course, it's kind of a Kill Bill anti-pattern: one of the goals of the platform is to abstract the gateways from the API user (as much as possible). It's especially important when using advanced functionalities like payment routing (you don't always know against which gateway the payment will be submitted).

I completely agree. This is just a work around so we could get this working.

 

1) HPP + Server notifications are 90% there! They are processed correctly as purchases (using the authMode=false), however the payment amount isn't actually being applied to anything. Whether it's to the outstanding invoices, or as a credit on the account.

I think I know why but could you describe in more details your scenario, the flow (how does your user tokenize the cart, at what point does he subscribe, what kind of invoice is generated, etc.) and the Kill Bill APIs you are using for this?

All invoices are generated by a subscription. So a user signs up for a subscription, KillBill generates the invoice, the user views that invoice, and then makes their initial payment through the HPP. Adyen will then send the notification back to KillBill (which works). What isn't clear, is what KillBill does with that payment. It appears that it does not know what invoice should get that payment, but then shouldn't a credit be generated? Right now the payment goes nowhere and can't be used for anything...

Perhaps i'm missing something in my HPP request that should tell KillBill what invoice to apply the payment to? If that's not possible, i'm fine with it just generating credits. I'm not sure what you see as the best way for KillBill to handle unknown payments. 

2) On a test using a Maestro card with 3d secure, the manual payment (after the token was synced) failed with an error of "configuration 905 Payment details are not supported". This generates the following killbill error: Error while communicating with the Kill Bill server: Error 500: java.sql.SQLDataException: Data too long for column 'gateway_error_code' at row 1

Interesting - are you running MySQL in strict mode? We probably never saw it because the column was silently truncated in our tests. Should be fixed by https://github.com/killbill/killbill-adyen-plugin/commit/fa88297067b601fca41207958e8a1e73bcc23733.

Regarding the error itself, are you completing the 3D-S flow in one go (buildFormDescriptor -> Adyen HPP -> 3D-S redirect -> Kill Bill/website redirect)? When using the PCI APIs, there is an extra step (extra API call) required to "complete" the 3D-S payment, I'm not familiar how it works with HPP though. Are you sure Adyen sees the payment method as validated?

Also, you mention a token - does Adyen let you create a recurring contract with a Maestro card? I don't think you can trigger recurring payments with Maestro unless you are part of MARP.

In Adyen, it showed the recurring contract & payment...so I think that worked. By the way, we're moving from our own home-grown billing / adyen integration to KillBill, and we would use the following API call to make recurring payments...and it seemed to work fine. Although, offhand, i'm not sure if there are any 3ds customers...

     Adyen::API.authorise_recurring_payment(

        invoice_id,

        { :currency => account_currency, :value => amount },

        { :reference => account_id, :email => account_email, :ip => current_user_ip, :statement => "Invoice ##{invoice_id}" }

      )

If KillBill can't handle those particular payments, that's fine, but it should show as a regular declined payment.  

3) I created a new HPP request using a standard visa card, updated the token in the database, and then tried again. I was able to create a manual payment once, but then all subsequent charges failed....

Same error as above in killbill, but in the adyen_responses table, I can see this: "{"exceptionMessage":"validation 803 PaymentDetail not found","exceptionClass":"org.apache.cxf.binding.soap.SoapFault","adyenCallErrorStatus":"RESPONSE_ABOUT_INVALID_REQUEST"}"

So, you can re-use the token once, then it fails? That's odd indeed. Could you share the full Kill Bill logs, including the XML requests to/from Adyen (you can mask the headers, and any sensitive information)?

Sure, those are included in the zip file linked above. 

--

Pierre

[Pierre-Alexandre Meyer]

Hi Kris,

I'm still parsing the logs to figure out why subsequent requests fail, but here are some partial answers in the meantime.

On Tue, Jul 12, 2016 at 1:00 PM, Kris Watson <kristia...@gmail.com> wrote:

No problem. Here are all the fields that i'm including to get a correct signature: currencyCode, shopperReference, countryCode, skinCode, merchantReference, shopperEmail, shopper.firstName, merchantAccount, shipBeforeDate, paymentAmount, shopperLocale, sessionValidity, resURL, recurringContract, shopper.lastName

Great - I've created https://github.com/killbill/killbill-adyen-plugin/issues/51 for tracking.

All invoices are generated by a subscription. So a user signs up for a subscription, KillBill generates the invoice, the user views that invoice, and then makes their initial payment through the HPP. Adyen will then send the notification back to KillBill (which works). What isn't clear, is what KillBill does with that payment. It appears that it does not know what invoice should get that payment, but then shouldn't a credit be generated? Right now the payment goes nowhere and can't be used for anything...

Perhaps i'm missing something in my HPP request that should tell KillBill what invoice to apply the payment to? If that's not possible, i'm fine with it just generating credits. I'm not sure what you see as the best way for KillBill to handle unknown payments.

The typical Kill Bill flow is:

1. Create an account
2. Add a default payment method (e.g. tokenize the card in the gateway)
3. Add a subscription

Step 3 triggers an invoice and a payment on that default payment method (if the invoice has a positive balance, e.g. if there is no trial period).

Because Adyen doesn't let you tokenize outside of a payment call, you could emulate step 2 as an AUTH + VOID operation:

1. Create an account
2. Add the default payment method:
1. Trigger the HPP call to create a recurring contract (AUTH mode - typically for $1)
2. Sync the payment method
3. VOID the payment
3. Add a subscription

Would that work in your environment?

If not, I could add a parameter in the Adyen plugin to link the invoice to that payment. But I'm curious about how you would configure Kill Bill in that scenario: are the accounts in AUTO_PAY_OFF? Otherwise, after the invoice is generated, the first payment will fail (until the user makes the initial payment). This has some potential side effects on entitlement (the user has access to the service for free) and overdue.

About credits: they are really driven by the invoice module (e.g. generated when the user downgrades to a cheaper plan and the change is configured as IMMEDIATE in the catalog). You could indeed add manually some credit to reflect the payment, but that seems error prone (the link invoice-payment should really be handled internally by Kill Bill).

In Adyen, it showed the recurring contract & payment...so I think that worked. By the way, we're moving from our own home-grown billing / adyen integration to KillBill, and we would use the following API call to make recurring payments...and it seemed to work fine. Although, offhand, i'm not sure if there are any 3ds customers...

Kill Bill does support one-off payments with 3D-S. In a recurring environment, if the payment method always requires 3D-S, this is more difficult though (since you have to make the user follow the redirect each time). This could be solved by sending the user the invoice (and putting the account into MANUAL_PAY) - we can talk in more details about it when you have that use-case.

--

Pierre

[Kris Watson]

On Wed, Jul 13, 2016 at 11:32 AM, Pierre-Alexandre Meyer <pie...@kill-bill.org> wrote:

Hi Kris,

I'm still parsing the logs to figure out why subsequent requests fail, but here are some partial answers in the meantime.

Awesome, I appreciate it! 

On Tue, Jul 12, 2016 at 1:00 PM, Kris Watson <kristia...@gmail.com> wrote:

No problem. Here are all the fields that i'm including to get a correct signature: currencyCode, shopperReference, countryCode, skinCode, merchantReference, shopperEmail, shopper.firstName, merchantAccount, shipBeforeDate, paymentAmount, shopperLocale, sessionValidity, resURL, recurringContract, shopper.lastName

Great - I've created https://github.com/killbill/killbill-adyen-plugin/issues/51 for tracking.

All invoices are generated by a subscription. So a user signs up for a subscription, KillBill generates the invoice, the user views that invoice, and then makes their initial payment through the HPP. Adyen will then send the notification back to KillBill (which works). What isn't clear, is what KillBill does with that payment. It appears that it does not know what invoice should get that payment, but then shouldn't a credit be generated? Right now the payment goes nowhere and can't be used for anything...

Perhaps i'm missing something in my HPP request that should tell KillBill what invoice to apply the payment to? If that's not possible, i'm fine with it just generating credits. I'm not sure what you see as the best way for KillBill to handle unknown payments.

The typical Kill Bill flow is:

1. Create an account
2. Add a default payment method (e.g. tokenize the card in the gateway)
3. Add a subscription

Step 3 triggers an invoice and a payment on that default payment method (if the invoice has a positive balance, e.g. if there is no trial period).

Because Adyen doesn't let you tokenize outside of a payment call, you could emulate step 2 as an AUTH + VOID operation:

1. Create an account
2. Add the default payment method:
1. Trigger the HPP call to create a recurring contract (AUTH mode - typically for $1)
2. Sync the payment method
3. VOID the payment
3. Add a subscription

Would that work in your environment?

Yeah, this flow seems totally doable. The thing I'll need to figure out is how to automate the void process. So a user goes through our order flow, creates their payment through the HPP for $1. Now our app will need to i guess poll KB looking for a $1 payment, and void that? Since the notifications are going to KB, our app won't know if that payment from the Adyen should be voided or not.

If not, I could add a parameter in the Adyen plugin to link the invoice to that payment. But I'm curious about how you would configure Kill Bill in that scenario: are the accounts in AUTO_PAY_OFF? Otherwise, after the invoice is generated, the first payment will fail (until the user makes the initial payment). This has some potential side effects on entitlement (the user has access to the service for free) and overdue.

Yeah, so it was a bit wonky, but we were using set_auto_pay_off() and remove_auto_invoicing_off() in our app, and storing the state of the payment method (i.e. is it ready to use or not). The above scenario seems much better. ;) 

About credits: they are really driven by the invoice module (e.g. generated when the user downgrades to a cheaper plan and the change is configured as IMMEDIATE in the catalog). You could indeed add manually some credit to reflect the payment, but that seems error prone (the link invoice-payment should really be handled internally by Kill Bill).

With your recommended flow, this is now a non-issue.  

In Adyen, it showed the recurring contract & payment...so I think that worked. By the way, we're moving from our own home-grown billing / adyen integration to KillBill, and we would use the following API call to make recurring payments...and it seemed to work fine. Although, offhand, i'm not sure if there are any 3ds customers...

Kill Bill does support one-off payments with 3D-S. In a recurring environment, if the payment method always requires 3D-S, this is more difficult though (since you have to make the user follow the redirect each time). This could be solved by sending the user the invoice (and putting the account into MANUAL_PAY) - we can talk in more details about it when you have that use-case.

Ok, yeah I'm not sure right now. I'll see if we even have any of these types of situations.  

--

Pierre

[Pierre-Alexandre Meyer]

On Wed, Jul 13, 2016 at 11:57 AM, Kris Watson <kristia...@gmail.com> wrote:

Yeah, this flow seems totally doable. The thing I'll need to figure out is how to automate the void process. So a user goes through our order flow, creates their payment through the HPP for $1. Now our app will need to i guess poll KB looking for a $1 payment, and void that? Since the notifications are going to KB, our app won't know if that payment from the Adyen should be voided or not.

Yes, that's the downside of the asynchronous notifications-based model :(

You have several options though, in order of complexity:

• Have your app poll Kill Bill. Maybe in the redirect, after the HPP, you could show a spinner to the user ("Validating payment method") while you poll, void and refresh the payment method (should take a couple of seconds at most)
• Have your app listen to Kill Bill webhooks (you can register your endpoint via POST /1.0/kb/tenants/registerNotificationCallback?cb=http://myapp.com/notify). Upon receiving a PAYMENT_SUCCESS event for a payment method validation call, trigger the void from your app
• Create a notifications plugin (in Ruby or Java) to listen to these events, and trigger the void from the plugin

Yeah, so it was a bit wonky, but we were using set_auto_pay_off() and remove_auto_invoicing_off() in our app, and storing the state of the payment method (i.e. is it ready to use or not). The above scenario seems much better. ;) 

Yup :-)

For future reference, AUTO_PAY_OFF should mainly be used in case of production issues (e.g. a customer calls complaining about charges, set the flag for a few days while you figure out if there is an issue, then remove the flag). AUTO_INVOICING_OFF is similar for subscriptions (e.g. in case the account doesn't have the right subscriptions).

--

Pierre

[Kris Watson]

On Wed, Jul 13, 2016 at 12:32 PM, Pierre-Alexandre Meyer <pie...@kill-bill.org> wrote:

On Wed, Jul 13, 2016 at 11:57 AM, Kris Watson <kristia...@gmail.com> wrote:

Yeah, this flow seems totally doable. The thing I'll need to figure out is how to automate the void process. So a user goes through our order flow, creates their payment through the HPP for $1. Now our app will need to i guess poll KB looking for a $1 payment, and void that? Since the notifications are going to KB, our app won't know if that payment from the Adyen should be voided or not.

Yes, that's the downside of the asynchronous notifications-based model :(

You have several options though, in order of complexity:

• Have your app poll Kill Bill. Maybe in the redirect, after the HPP, you could show a spinner to the user ("Validating payment method") while you poll, void and refresh the payment method (should take a couple of seconds at most)
• Have your app listen to Kill Bill webhooks (you can register your endpoint via POST /1.0/kb/tenants/registerNotificationCallback?cb=http://myapp.com/notify). Upon receiving a PAYMENT_SUCCESS event for a payment method validation call, trigger the void from your app
• Create a notifications plugin (in Ruby or Java) to listen to these events, and trigger the void from the plugin

Great ideas! Thanks! I like option #2. Our app has a pretty good framework for web hooks already since we integrate with a lot of external services.

[Pierre-Alexandre Meyer]

On Wed, Jul 13, 2016 at 11:57 AM, Kris Watson <kristia...@gmail.com> wrote:

I'm still parsing the logs to figure out why subsequent requests fail, but here are some partial answers in the meantime.

Awesome, I appreciate it! 

Not much luck understanding what's going on I'm afraid :( Kill Bill logs are fine[0] and nothing obvious from the plugin (I even diff'ed the XML to/from Adyen).

Could you reach out to sup...@adyen.com directly and ask them why the payment with reference 5c566a1c-3375-46b0-8c23-812b74b3edbe is failing whereas d51da40d-51f2-46d0-a445-7a1eda922953 isn't?

[0] Minor detail: you are passing controlPluginName=killbill-adyen to the buildFormDescriptor call. This isn't necessary because the plugin name will be looked-up via the payment method id (also Adyen isn't a payment control plugin but simply a payment plugin).

--

Pierre

[Kris Watson]

Will do, thanks for looking at this.

We're about to test on a fresh killbill installation after making changes to our order flow, so maybe a fresh dataset will resolve it...

-Kris

[Kris Watson]

Hi Pierre,

I tried again with a fresh install and had the same issue, so i sent data on 2 transactions to Adyen to show them how the 1st would succeed, but the second would fail.

Here is their response:

##

Hi Kris,

It seems that you have hardcoded the selectedRecurringDetailReference from the initial transaction. However the selectedRecurringDetailReference can change:

shopperreference | shopperemail | recurringdetailreference | creationdate | validtilldate 

------------------+--------------+--------------------------+----------------------------+-------------------------------

1m9191HmctUKV6 | us...@example.com | 8314683504149467 | 2016-07-12 21:06:54.917+02 | 2016-07-12 21:11:09.026478+02

1m9191HmctUKV6 | us...@example.com | 8314683506691367 | 2016-07-12 21:11:09.027+02 | 9999-12-31 00:00:00+01

1m9191HmctUKV6 | us...@example.com | 8414685480379231 | 2016-07-15 04:00:37.816+02 | 2016-07-15 04:59:12.802507+02

1m9191HmctUKV6 | us...@example.com | 8314685515523572 | 2016-07-15 04:59:12.804+02 | 9999-12-31 00:00:00+01

So at the moment of transaction 8514685549750694 you tried it with 8414685480379231 which was already closed.

You can solve this issue by:

1. Always perform a listRecurringDetails call first, fetch the recurringdetailreference and then perform the recurring payment.

2. Use LATEST as selectedRecurringDetailReference then we will make sure that we always use an active recurring detail reference.

I hope this helps you! Please let me know if there are remaining questions.

##

So in Adyen, a recurring contract can have multiple stored payment options, but i noticed in KB only a single payment token is stored per payment method (adyen_payment_methods). So conceivably, it could be as simple as just using 'LATEST' when passing selectedRecurringDetailReference in your plugin instead of the stored reference?

-Kris

On Wed, Jul 13, 2016 at 3:30 PM, Pierre-Alexandre Meyer <pie...@kill-bill.org> wrote:

[Pierre-Alexandre Meyer]

Thanks for following-up with them - but I'm really confused now :-)

On Fri, Jul 15, 2016 at 1:13 AM, Kris Watson <kristia...@gmail.com> wrote:

It seems that you have hardcoded the selectedRecurringDetailReference from the initial transaction. However the selectedRecurringDetailReference can change:

[...]

So at the moment of transaction 8514685549750694 you tried it with 8414685480379231 which was already closed.

AFAIK, the recurring details can indeed change when it's disabled and then re-created (i.e. when calling https://docs.adyen.com/developers/recurring-manual#disablingarecurringcontract). But it's not the case in your scenario. Could they investigate on their end why it changed?

Also, it looks like it changed an hour after the previous one was created - didn't you try the transactions back to back?

By the way, I'm not experiencing this behavior on our sandbox:

• https://github.com/killbill/killbill-adyen-plugin/commit/1047f0d9f79ff92d7fb9de9da225dee9f2caf361
  
• https://github.com/killbill/killbill-adyen-plugin/commit/e4fcf8e50f0c0a7c2ad8a5b212ad4a40ed58a806
  

(I've verified the XML sent is identical to the one in your logs)

So in Adyen, a recurring contract can have multiple stored payment options, but i noticed in KB only a single payment token is stored per payment method (adyen_payment_methods). So conceivably, it could be as simple as just using 'LATEST' when passing selectedRecurringDetailReference in your plugin instead of the stored reference?

The idea is to map one payment method in Kill Bill with one recurringDetailReference, so the user can have different payment options on file (several credit cards or bank accounts - each would be linked to a different payment method, for the same account). Using LATEST wouldn't always work: imagine your customer having a business credit card and a personal one, he wouldn't want to pay business-related invoices with his personal one and vice versa.

Happy to add work-arounds or change the behavior in the plugin, but I'd like to get to the bottom of this with Adyen first, so we're sure we understand the full picture.

--

Pierre

[Pierre-Alexandre Meyer]

Hi Kris,

Were you able to figure out what the issue was?

FYI I've created a sample Sinatra app, this may be helpful for future discussions: https://github.com/killbill/killbill-adyen-demo

(note: it's only compatible with Kill Bill 0.17.x and the latest Adyen plugin, which fixes the SHA256 issues you reported)

--

Pierre

[Kris Watson]

Hi Pierre --- thanks for checking in.

I had to table this for a few days in order to get our beta out. We built a quick work around using the web hooks feature to handle adyen payment captures in our app, and then post them as generic external payments in KB.

Obviously this is not a long term solution and we want everything to be in KB. I'm hoping to re-visit this within the next few days and try to track down why we're having different results.

Thanks for the sinatra app -- i'll take a look at that!

-Kris

[Pierre-Alexandre Meyer]

No worries -- no rush.

Good luck with the beta launch!

--

Pierre

[Ольга Павлушко]

Hi Pierre, 

could you tell me if issue with SHA256 encryption is solved? I found a ticket opened here https://github.com/killbill/killbill-adyen-plugin/issues/51

but above you say that in adyen-plugin version 0.5.1 this issue is fixed.

I'm asking because I face a problem with merchant signature, and it is calculated different in plugin then in Adyen (HMAC calculation page).

Best regards, 

Olga

пятница, 22 июля 2016 г., 0:46:52 UTC+3 пользователь Pierre-Alexandre Meyer написал:

[Pierre-Alexandre Meyer]

On Fri, Feb 3, 2017 at 5:38 AM, Ольга Павлушко <lisaal...@gmail.com> wrote:

could you tell me if issue with SHA256 encryption is solved? I found a ticket opened here https://github.com/killbill/killbill-adyen-plugin/issues/51

but above you say that in adyen-plugin version 0.5.1 this issue is fixed.

I'm asking because I face a problem with merchant signature, and it is calculated different in plugin then in Adyen (HMAC calculation page).

Yes, SHA256 is supported with 0.5.x. I'm not aware of any issue.

Try maybe modifying the values in the test to see what is different?

--

Pierre