calling PHP script (payment gateway) from Tornado + using tornado.web.asynchronous

[Ethan Collins]

I have a tornado app for ecommerce and need to connect the payment gateway code. My provider has provided me only PHP supported library for this payment gateway. It has 2 APIs, 1 to connect with the payment gateway server to get the user redirection URL (this consumes ~400ms due to the network roundtrip) and the second one is a callback API. Think of this as synonymous to the Facebook auth code in Tornado, at a high level.

I currently intend to call the PHP code from my python handlers. There are 2 ways as I have found.
1) Use subprocess module, like as mentioned here: https://mail.python.org/pipermail/tutor/2009-October/071973.html

2) Run PHP-FPM on the backend and push requests to it via unix sockets. I find this method for convenient, more so because it will have less invocation cost and I belive several calls to the unix sockets can be done simultaneously (hoping that socket calls will be asynchronous, need to find out) if need arises.

My Queries:
1) Has anyone tried to call PHP code from their python code? Are there any challenges that I need to take care of? (I don't know PHP, first timer)

2) If unix sockets cannot be called asynchronously, then I have to choose the option #1 but then I also need to use the decorator @tornado.web.asynchronous. Can this decorator be used over subprocess module? If possible, anything specific that I need to take care of? (till now I didn't have the need to use this decorator in my code, so really don't know)

Will be great if I can get your comments/view here.

Cheers,
Ethan

[A. Jesse Jiryu Davis]

Either approach sounds reasonable to me. I don't know anything about PHP either so I may be naïve thinking that PHP can run equally well as a CGI or a command-line script. If you go with #2, you can connect an IOStream to a unix socket.

For example, look in UnixSocketTest in the Tornado source. There's a server in that test that binds to a Unix socket, but that's not the part you want to look at. Instead, examine the client code around here:

stream = IOStream(socket.socket(socket.AF_UNIX))

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ethan Collins]

Thanks for the lead, Davis.

In the meantime I am looking at tornado's asynchronous code and gen code. I notice another thread where you posted some notes related to it.

[aliane abdelouahab]

hi

i found this

http://stackoverflow.com/a/11601572/861487

and what about Thrift http://thrift.apache.org/

[Ethan Collins]

Hi Aliane,

I had seen that. It's synchronous, and if I have to do like that I can call the PHP script from tornado in a Subprocess call itself. However, I am trying to make it asynchronous.

[Ethan Collins]

Hi Davis,

Just saw the UnixSocketTest class in httpserver_test.py. Are you saying that using it I can get asynchronous behaviour using something like this:

@gen.coroutine
def get_data_from_php(php_script):
    sockfile = os.path.join(self.tmpdir, "test.sock")
    stream = IOStream(socket.socket(socket.AF_UNIX)
    stream.connect(sockfile, self.stop)
    stream.write(....)
    ...
    stream.read_bytes(len)
    ...
    return data

and calling this as:
@gen.coroutine
def get(self):
   result = yield get_data_from_php(php_script)
   <handle returned result>
   self.redirect(url)

On Saturday, December 14, 2013 12:58:23 AM UTC+5:30, A. Jesse Jiryu Davis wrote:

Either approach sounds reasonable to me. I don't know anything about PHP either so I may be naïve thinking that PHP can run equally well as a CGI or a command-line script. If you go with #2, you can connect an IOStream to a unix socket.

For example, look in UnixSocketTest in the Tornado source. There's a server in that test that binds to a Unix socket, but that's not the part you want to look at. Instead, examine the client code around here:

stream = IOStream(socket.socket(socket.AF_UNIX))

On Fri, Dec 13, 2013 at 1:51 PM, Ethan Collins <collins...@gmail.com> wrote:

I have a tornado app for ecommerce and need to connect the payment gateway code. My provider has provided me only PHP supported library for this payment gateway. It has 2 APIs, 1 to connect with the payment gateway server to get the user redirection URL (this consumes ~400ms due to the network roundtrip) and the second one is a callback API. Think of this as synonymous to the Facebook auth code in Tornado, at a high level.

I currently intend to call the PHP code from my python handlers. There are 2 ways as I have found.
1) Use subprocess module, like as mentioned here: https://mail.python.org/pipermail/tutor/2009-October/071973.html

2) Run PHP-FPM on the backend and push requests to it via unix sockets. I find this method for convenient, more so because it will have less invocation cost and I belive several calls to the unix sockets can be done simultaneously (hoping that socket calls will be asynchronous, need to find out) if need arises.

My Queries:
1) Has anyone tried to call PHP code from their python code? Are there any challenges that I need to take care of? (I don't know PHP, first timer)

2) If unix sockets cannot be called asynchronously, then I have to choose the option #1 but then I also need to use the decorator @tornado.web.asynchronous. Can this decorator be used over subprocess module? If possible, anything specific that I need to take care of? (till now I didn't have the need to use this decorator in my code, so really don't know)

Will be great if I can get your comments/viw here.

Cheers,
Ethan

[aliane abdelouahab]

ah! sorry, since the first line is a disk read, so it will block untill the python code finishes.

sorry.

[A. Jesse Jiryu Davis]

Close, Ethan. self.stop() is a test method. For a coroutine:

@gen.coroutine
def get_data_from_php(php_script):
    sockfile = os.path.join(self.tmpdir, "test.sock")
    stream = IOStream(socket.socket(socket.AF_UNIX)

    yield gen.Task(stream.connect, sockfile)
    yield gen.Task(stream.write, output_data)
    ...
    data = yield gen.Task(stream.read_bytes, len)
    ...
    return data

That's a start. Unfortunately, IOStream isn't designed to be used with coroutine so you need to hack a little, as you can see. You should register a callback with stream.set_close_callback, but it's a bit tricky to say for certain what that callback should do.

Hey, guys—we were talking a while ago about writing an IOStream wrapper that worked well with coroutine. It's sort of obvious what that wrapper should do, but has anyone actually written it yet?

[Ethan Collins]

Thanks Davis. The hacking makes me feel jittery, as I am going to connect the payment gateway php code to this, any issue and I will be facing the wrath of customers and negativity on my website. I will look into this.

Are there other ways which are more standard? (I will be setting up the server initially on a single core droplet with DO)

Thanks for your help!
Ethan

[A. Jesse Jiryu Davis]

Rereading your initial post, I have a new thought. So, does PHP-FPM speak HTTP over the Unix socket, or does it speak some other protocol? If it's just an HTTP server, you should reconfigure PHP-FPM to listen on a TCP/IP socket and use Tornado's AsyncHTTPClient to talk with it. If it's speaking a protocol other than HTTP then that won't work, of course....

[Ethan Collins]

Davis, this is exactly what I was wondering while going through the Tornado code. I haven't yet tried the PHP-FPM thing, just know that it can be enabled over socket, and I think I can make it speak HTTP. Tell me this, doing this way is better than the IOStream method you had mentioned?

The current thing that I am trying to understand is the fundamentals: tornado's async support. On a first read of the code, things look very cryptic to me still. So, probably I should read up generators and coroutines first.

I need to understand your other sample code also (the one you gave with modifications). 1 query there: do I really need the usage of 'yield Task(..)' for stream.connect and stream.write? They should be rapid anyway, right?

1 very fundamental question: Making this GET routine (which takes ~400ms) async: does this mean that I can service multiple such GET request simultaneously by a single tornado process? (That is my effort)

Ethan

[A. Jesse Jiryu Davis]

I recommend the "normal http option" (that is, using AsyncHTTPClient to talk to PHP-FPM using HTTP and TCP/IP) more than the "unix socket option" (that is, connecting an IOStream to a Unix socket) because AsyncHTTPClient wraps up a lot of very finicky details for you. Using an IOStream directly is much harder to do right. You're better off relying on Ben's work than your own. =)

I understand that coroutine code looks cryptic. I have some slides on it and I keep meaning to write a blog post explaining the implementation....

About my sample code: you need to yield after calling stream.connect so your coroutine isn't resumed until the connect succeeds. It doesn't matter how fast the connection is, your code can't continue until it's completed. yield after write() provides a bit of flow-control, so you won't overwhelm a slow reader on the other side of the connection.

The answer to your fundamental question is, yes, as long as your code is async all the way down, Tornado can process multiple slow requests concurrently. You can tell it's async all the way down because, every time you do I/O (connect, write, read) you "yield" until the I/O completes.

[Ethan Collins]

Yes, I will concentrate on AsyncHTTPClient for the moment. And auth.py's FacebookGraphMixin is a good example to get started with. Yes, even though Ben's work is awesome without doubt, I think I feel more comfortable when I read and undestand the code, may be for now I should change that.

Thanks for patiently explaining me the streams and async methods, thank you again!

It will be of great help if you upload the slides quickly. May be it can get started and be a WIP.

Wanted to ask you this: with gen.coroutine, web.asynchronous becomes unnecessary. I notice that in auth.py for FacebookGraphMixin, the example however uses both. Is that necessary or just something Ben missed out to remove?

Cheers,
Ethan

[A. Jesse Jiryu Davis]

Yeah, the docs say, "This decorator is unnecessary if the method is also decorated with @gen.coroutine (it is legal but unnecessary to use the two decorators together, in which case @asynchronous must be first)." I suppose Ben hasn't updated all the examples yet.

[Ethan Collins]

I could successfully 'talk' to php running on the server.

php-fpm cannot be talked with directly. I installed php-fpm and configured it to use unix socket. From my server's Nginx, I created a new subdomain 'php.example.com', listening on port 8090 and connected with php-fpm's socket. I had to ensure that external world cannot access this subdomain by using nginx's 'deny' directive.

My tornado handler looks like this:

class PhpHandler(BaseHandler):
    @coroutine
    @cookies_pre_process
    def get(self):
        self._test_php('/status?full')
        self._test_php('/ping')
        return

    @coroutine
    def _test_php(self, url):
        t1 = time.time()
        http_client = AsyncHTTPClient()
        response = yield http_client.fetch('http://php.example.com:8090'+url)
        if response.error:
            log.debug('php fetch error: %s', str(response))
            return

        time_delta = t_ms(time.time() - t1)
        log.debug('%s', response.body)
        log.debug('%.3f', time_delta)

The timing is quite high: ~7ms.

Currently, I will connect the payment gateway code here. Post staging, I will be converting the php code to python. Tell me this: since payment gateways work over SSL, can I use AsyncHTTPClient() for that?

Cheers,
Ethan

[A. Jesse Jiryu Davis]

This sounds like a solid, ordinary architecture to me.

Make sure you yield self._test_php(...) before doing the next operation. When you call a coroutine, it starts running and returns a Future, and the caller keeps running, too. Only by yielding the Future do you pause and wait for the called coroutine to complete, and get its result / throw its exception. As it is, your get() method starts two coroutines and immediately returns. If you want to call a coroutine, wait for it to complete, then call the next, you need to yield the Future each coroutine returns.

[A. Jesse Jiryu Davis]

Oh, and yes: You can use AsyncHTTPClient with SSL.

[Ethan Collins]

Thanks for pointing out the missing yield. Yes, it was running without waiting for the return.

I now tend to feel that this probably is an overkill. Yes, my intend was to get started with the php code as then I get support from the payment gateway company regarding error conditions/issues with their server. I did the php-fpm so that it can easily be made to handle lots of requests as need arises with time. But given that I will porting their code later on, quite sooner actually, to python, probably it makes sense to use 'subprocess' module now -- things then remain much simpler with configurations, no need to install php-fpm or change nginx (it's quite distributed now unnecessarily). I can run the php scripts as subprocess and think of using coroutines there.

May be I will try that.

Thanks for your help.
Ethan

[Ben Darnell]

On Sat, Dec 14, 2013 at 1:28 PM, A. Jesse Jiryu Davis <je...@emptysquare.net> wrote:

Rereading your initial post, I have a new thought. So, does PHP-FPM speak HTTP over the Unix socket, or does it speak some other protocol? If it's just an HTTP server, you should reconfigure PHP-FPM to listen on a TCP/IP socket and use Tornado's AsyncHTTPClient to talk with it. If it's speaking a protocol other than HTTP then that won't work, of course....

I think you can do HTTP over unix sockets with SimpleAsyncHTTPClient if you pass in a custom Resolver that returns an AF_UNIX address for some fake hostname.  However, I'd only recommend this if you're running into problems running it over TCP (e.g. port allocation headaches or a need for better security against other users on the same machine).

Overall, my recommendation for foreign code like this would go in this order:

1) Port it to python.  It's easier to interface with a well-designed API directly than to build an ad-hoc cross-language interface.

2) Run the foreign code in a subprocess (but watch out for shell injection and other potential security issues). PHP has a usable command-line mode, although running it in a subprocess is likely to be slower than a server-based option due to interpreter startup.

3) Run a web server and use HTTP and TCP (with appropriate access controls).

4) Use more exotic networking setups like unix sockets only if you have specific needs that are difficult to meet with the previous options.

-Ben

[Ethan Collins]

Hi Ben,

Thanks for your suggestions. Below comments.

On Monday, December 16, 2013 10:50:41 PM UTC+5:30, Ben Darnell wrote:

On Sat, Dec 14, 2013 at 1:28 PM, A. Jesse Jiryu Davis <je...@emptysquare.net> wrote:

Rereading your initial post, I have a new thought. So, does PHP-FPM speak HTTP over the Unix socket, or does it speak some other protocol? If it's just an HTTP server, you should reconfigure PHP-FPM to listen on a TCP/IP socket and use Tornado's AsyncHTTPClient to talk with it. If it's speaking a protocol other than HTTP then that won't work, of course....

I think you can do HTTP over unix sockets with SimpleAsyncHTTPClient if you pass in a custom Resolver that returns an AF_UNIX address for some fake hostname.  However, I'd only recommend this if you're running into problems running it over TCP (e.g. port allocation headaches or a need for better security against other users on the same machine).

Currently, it's on the TCP (tornado -> nginx, handling a fake domain + port -> phpfpm on unix socket), done as I mentioned a few posts back. Maybe I can try out the one you mentioned, sometime.

Overall, my recommendation for foreign code like this would go in this order:

1) Port it to python.  It's easier to interface with a well-designed API directly than to build an ad-hoc cross-language interface.

That's the next step. (This payment gateway code is big and poorly written, and I get integration support only when I run their code)

2) Run the foreign code in a subprocess (but watch out for shell injection and other potential security issues). PHP has a usable command-line mode, although running it in a subprocess is likely to be slower than a server-based option due to interpreter startup.

Tried this yesterday. The interpreter startup cost is heavy. Takes ~15ms for running a bare echo service. However, the current setup that I have is called only with self-generated arguments and not user args. Therefore, hoping there should be no security issues, unless you point me if any possible.

Best,
Ethan