NAB Transact down – Stripe / Braintree out, thanks to Pin for help

[Hugh Stephens]

Hi guys

Warning: long post. tl;dr, thanks to the guys at Pin for helping us out, would recommend them, haven't had a great experience with Stripe.

Just wanted to throw some kudos a few ways, and vent on some others. Some of you have probably (like us) been affected by NAB Transact's disastrous problems, culminating in all non-AUD transactions to stop going through over the last day or so (after rolling outages). This has affected Stripe, Braintree and Pin (at least).

Stripe's support has been pretty useless throughout. No error messages on their status page, no email to customers, nothing. Their AUD payments have been going through (as have Pin's – think Braintree maybe not?), so I get that many customers might not be affected, but we transact ~85% in non-AUD (USD and EUR predominantly, some GBP). They finally put up a notice 16 hours down the track in their UI, and back-filled their status page to "partial degradation" for the last few days (was all green-green until now).

Finally managed to get onto Susan from Stripe (AU country manager) this morning after a nightmare of customers last night complaining (rightfully) that their payments were busted after calling their bank etc (NB: we also had no registrations in non-AUD during this time...). This was only after sending a whole series of angry as hell tweets, and knowing that she was the country manager - the @stripe account generally doesn't do a lot of customer service it seems. She was very apologetic for them dropping the ball on the issue re keeping customers updated, and have been helpful in trying to get us up and running through a USD-native account that we can use as escrow while they fix the problem (rerouting all non-AUD through another bank). (update: apparently they have put a workaround in place, haven't tested it yet but some are going through).

I've generally had pretty poor support experiences with Stripe (which is email-only, if that, little to no prioritisation) – slow or nonexistent response times to issues like customers being charged when Stripe says it failed (when the customer is *definitely* charged – I've seen statements etc). Has happened to us a couple of times now, resulting in a couple of lost customers (and having to just ask them to chargeback us because of Stripe as we can't refund a "failed" transaction) and a whole lot of hand-wringing.

BUT in good news (this is the kudos), the guys at Pin came to our rescue. 

They've apparently been doing the same things as Stripe (rerouting etc), and have managed to move us to their new infrastructure immediately so we could just change our default gateway in our billing platform and charge in non-AUD. Apparently they're now able to support (or soon will be) some of the reasons that we churned to Stripe for in the first place (like settling into a US bank account for USD), and the support has been excellent (disclaimer: I do know one of their team personally). 

So a big A+ to me for another Australian business, and I would strongly encourage anyone considering what gateway they should use to look at Pin – at least from my perspective, it seems they can compete like-for-like with Stripe (albeit without subscriptions included free which is a bummer, but I use Chargebee for that anyway). As we have used Pin for a few customers previously, I'm already subscribed to their outage notices which are very timely (bordering on too timely, which is OK with me).

From the point of view of this list, I come from a comms background and our platform does have outages, so I wanted to share my principles for crisis comms. Outages and issues happen, and the value is often more in how you respond to it than just fixing the issue as quickly as you can.

1. ADMIT: Always admit fault, and acknowledge the problem *in public*. Don't just tell people to DM you etc without actually acknowledging you've dropped the ball. Looks bad to anyone external.

2. EXPLAIN: Tell the customer what the cause is (if you can) in plain English, rather than relying on the "black box" approach and assuming it's too hard/complex/boring for them (I will even put 'headings' saying "Explanation below, tl;dr <x>" in an email)

3. COMMUNICATE: Your communications should: acknowledge the issue, explain what it is, give a timeline for resolution, tell the customer what you are *doing* to solve their problem, and tell them when you will get in touch next

4. FOLLOW UP: Follow up regularly – for example, we will send an update each hour if something is extended to customers, even if saying "unfortunately this is still persisting, we will get back to you with an update when available"

From my experience, customers respond really well to this, particularly if they haven't prompted the response initially – we can obviously identify with a DB lookup customers affected by outages, and will chuck notices in our UI even if only for 30 minutes. I've been consistently surprised how many customers tell us that it's refreshing for a service to approach it like this. 

Note that above doesn't include "give them a discount" – we do that when someone has clearly lost some major jujubeans, but that actually doesn't solve the problem for many customers - they just want to know it will be fixed and won't happen again, rather than death by a thousand small discounts as 'payment' for frequent outages.

Thanks to the guys at Pin for helping us out again.

Hugh

[Sam James]

I had the same issues today with Braintree - only found out after numerous customer emails saying that payments weren't going through.

Furious that Braintree don't even notify their customers there's an issue? 

I have paypal as a backup, but not sure how many sales I lost that give up after the credit card option not working.

Would never use braintree again

[ry...@marketplacer.com]

We're having the same issues at marketplacer.com. We've had about 500 transactions fail in the past 24 hours due to this bug. There's been a lot of panicking here, but not much we have been able to do about it.

Thank you muchly for your post, Hugh.

[Andrew Beeston]

Hey Hugh good story in the end hopefully! The guys at Pin seem like they've done a good thing again.

I've have had a bunch of conversations with people at EnvoyRecharge (disclaimer: I'm a founder) about how switching really quickly from something like NAB Transact/SecurePay would have saved them thousands of dollars, but there was no option immediately obvious. Luckily the guys from Pin could help lots of others like you.

We're only 8 days away from launching, and this is one of the core things our system helps people do (plus adding functionality like recurring billing) - so it was frustrating to see we weren't already helping people out there.

I really hope the guys at Pin have benefited from this - they've a right to, and they're a great business too.

[blindman2k]

We are direct customers of NAB and had no idea anything was wrong until we had a customer complain. I suspect others just went to our competitors.

We sent in a (still unresponded) support ticket days ago and have never received any of the correspondences mentioned on their support page. Well done finding a provider that can step around this stupidity.

    A.

[Hugh Stephens]

Just following up on my post too – a note for those of you who serve international customers, flipping to another payment gateway (we use a similar kind of third party billing platform so we have the capability – we use it because it has better controls than Stripe's recurring platform) doesn't necessarily solve all your problems. We get probably 1 in 3 customers' banks declining charges, and changing gateway (often, not always) makes them have to go through the same process again – so this has also given us some grumpy customers who (luckily for us) care more about our platform than they do about the inconvenience of calling the bank again to sort it all out.

Stripe has also totally stuffed up customer support here – I have a collection of customers who were charged even though NAB listed them as failed, and they never got refunded. I can't refund it, angry (as hell) customer who subsequently cancels, Stripe saying it's refunded, find out ~1.5 weeks later (28 May > 11 June) that no, it is NOT refunded and they are looking into it because their system didn't void the authorisation. So now we're having to go through and prompt customers to check, because I can no longer rely on any of their statuses (and isn't that a great look?). I get that NAB Transact totally stuffed them around here, but as a customer, getting 5 different answers from support in as many days (as I've been following the news, and often seem to have been a step ahead of them in all this) is rather frustrating. Luckily we seem to have finally gotten to the stage where Stripe has accepted that our customer was right from day 1, but it's taken rather a while and I've lost a few subs as a result.

Pin has been totally fine though – my only complaint is that they don't do foreign currency transactions for Amex cards (notably for us USD on Amex) but otherwise haven't seen any issues since we flipped the switch (beyond some customers having to re-auth with their bank).

Thanks

Hugh

[Chris Dahl]

Thanks Hugh. You're right regarding Amex support, and we're working on adding the equivalent of our multi currency features (https://pin.net.au/docs/currency-support) with Amex. We'll post an update once we're ready to start releasing this to our customers.

We posted a blog post last week regarding the recent issues with NAB Transact that might be interesting for those following this thread - https://pin.net.au/blog/nab-outage.