Flush rendering immediately?

[Phil Edelbrock]

Have a few cases where it is critically important to flush the output of ‘render’ immediately. For example:


def place_order
if validate_and_record_order?
render :text => “OK"
else
render :text => “ERROR"
end
# DESPERATELY WANT TO FLUSH OUTPUT HERE!
# Do lots of talking with banks, send out mailers, etc.
end


Another wrinkle here is we can’t store the customer’s banking information to run as a deferred task. Ideally we just want to say ‘flush the output’ after the ‘if’ block. Even if there’s an exception after the ‘if’ we still don’t want them to get a 500, we want them to get the OK/ERROR.

The problem we have is the people connecting to our app are sometimes timing out (they have a short timeout) and think the order failed when in fact it was recorded, customer was charged, etc. I’m a little hesitant about using Thread.new, but perhaps that’s the only reasonable way to flush the output early?

btw- This is Rails 4.0.x, Apache+Passenger.

Thanks!


Phil

[Dave Aronson]

On Thursday, August 18, 2016, Phil Edelbrock <ede...@gmail.com> wrote:

Have a few cases where it is critically important to flush the output of ‘render’ immediately.

Sounds to me like you probably want to just enqueue a background task after the render, and have that do the heavy lifting later.

--
Sent from Gmail Mobile; please excuse top posting, typos, etc. :-(

[Phil Edelbrock]

On Aug 18, 2016, at 5:10 PM, Dave Aronson <googlegro...@codosaur.us> wrote:

On Thursday, August 18, 2016, Phil Edelbrock <ede...@gmail.com> wrote:

Have a few cases where it is critically important to flush the output of ‘render’ immediately.

Sounds to me like you probably want to just enqueue a background task after the render, and have that do the heavy lifting later.

Yeah, that’s probably the ‘proper’ way to do it.  Trouble is that it is banking information and we’re supposed to not store that stuff in order to be PCI compliant.  Ironically, the slow task that’s causing us the most delay/problems is the connection to the bank’s API to store the banking information in a PCI compliant way, ha.  *shakes fist at authorize.net*

Phil

[Dave Aronson]

On Thu, Aug 18, 2016 at 8:18 PM, Phil Edelbrock <ede...@gmail.com> wrote:

> it is
> banking information and we’re supposed to not store that stuff in order to
> be PCI compliant.

How does PCI define "store"? If it's OK to have it in *RAM*, just not
on *disk*, maybe you can "store" it long enough to launch the
background task, in something like Redis -- just make sure it's never
flushed to disk.

Alternately, perhaps you can fire off an asynchronous request to
another system (whether on the same box or no) that could handle the
tasks. IIRC, PCI will require that it be protected in transit so
there's all that overhead, but even so, there may be some
possibilities along those lines.

--
Dave Aronson, consulting software developer of Codosaur.us,
PullRequestRoulette.com, Blog.Codosaur.us, and Dare2XL.com.

[Jim]

I have used the spawnling gem to fork a new long-running process. It handles closing/re-opening database connections, etc.

https://github.com/tra/spawnling

Another alternative could be Server-Sent Events (SSE), which works with Passenger > 4.0.5. It will require changes in how you set up the front end as well, but is good if you want to pass status back to the user during/after this long-running task.

Jim Crate

[Brendon]

If you are trying to be PCI compliant, then this approach just doesn't cut it. PCI DSS compliance rules apply to anybody who is Processing, Transmitting or Storing card info. Since your server is handling (processing and transmitting) the "banking information" it needs to be inside the PCI 'zone' with all the logging, software requirements physical access restrictions, etc. I'd strongly urge you to NOT do this.

If somebody gains access to your server, they can easily log the request parameters and *bang* you are liable for the data breach, even if you don't store the data. That is basically what happened at Target, where the system sent the card info out of the company until the breach was found.

The easy way to dodge PCI responsibility is to NEVER touch the sensitive data.

It is much easier than it used to be:

1. You render your card info entry screen in the browser, but have the submit button go STRAIGHT to Stripe, Braintree, Authorize.net, etc.
2. When the user submits, the browser sends the PCI data to the payment processor directly.
3. They grab the payment info and do whatever needs doing... then
4. The Payment processor then REDIRECTS the client back to your server with some context so you can match up the client, payment, etc.  
   
5. You get the request from the client and do whatever you need to tell them if it worked or not.

Take a look at: https://stripe.com/docs/security for an overview. And then look at https://stripe.com/docs/stripe.js as an easy way of handling the details.

All the other providers provide similar capabilities -- and if they don't, change provider! I looked by Authorize has moved all the docs around so I grabbed the above from Stripe instead.

We are a Stripe customer and I've been a Braintree and Authorize.net customer in the past.

Good luck,

Brendon.

[Phil Edelbrock]

Thanks!  We’re at a merchant tier where we’re required to have 3rd party security audits every quarter.  We’re compliant.  Our whole system from the datacenter to equipment to code is with PCI compliance in mind.  We could store the banking info ourselves, but I’d only do it if we had a server on a private network that was storing that information.  It is cheaper, easier and safer to use a service like Authorize.net’s CIM, though.  Our service is odd in that we’re very fluid in needing to charge and refund on the fly, security deposits, complex tax and other government agency compliances, trust accounts, talking directly to banks via ACH, etc.  If we were just selling widgets or something, it would be a lot easier to use something like you describe (like Authorize.net’s SIM integration method).  And you are right, that’s the safer/better route in many cases.

Thanks for the information, though.  I’m sure others here could find it interesting and valuable, too.

Phil

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/52341f3a-7cf6-40a1-8914-1cb162210599%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brendon]

The real challenge with the "flush now" idea is that you cannot generally (websockets and http2 change the game) continue to feed information to the browser later. so the flush now would break the connection to the browser. If you walk through the rack/rails stack, you will see that the final sending really ends up happening after you return from the call, in the same thread.

I'd suggest doing an asynchronous call to the slow systems in a new thread or use rabbitMQ or similar to take the request and run it in a different ruby thread, possibly in a different process (or container if you are using Docker and doing the microservices thing.) Redis will also let you use messaging like this, but risks storing the data on disk. (Actually, you run that risk anyway, given that the OS may swap out pages with the data to disk and then it lives on disk until the swap space is recycled....) Anyway, using a messaging system to offload the slow process lets the original request end quickly. You can then either have the client poll for the final result (a little uck, but very reliable without any added technology -- especially if you just do the slow request off of a thread) or use websockets to feed the results back when you know them.

Having been down the road you are suggesting of having a "flush", I can tell you that fighting the framework like that brings a lot of pain, and ongoing pain as time goes on and the underlying code changes.

Happy to advise if you PM me... I've BTDT.

Brendon