Received: by 10.180.73.242 with SMTP id o18mr340716wiv.0.1349775336889; Tue, 09 Oct 2012 02:35:36 -0700 (PDT) X-BeenThere: python-tornado@googlegroups.com Received: by 10.180.19.169 with SMTP id g9ls9349957wie.3.canary; Tue, 09 Oct 2012 02:35:21 -0700 (PDT) Received: by 10.181.11.234 with SMTP id el10mr339991wid.2.1349775321928; Tue, 09 Oct 2012 02:35:21 -0700 (PDT) Received: by 10.181.11.234 with SMTP id el10mr339989wid.2.1349775321910; Tue, 09 Oct 2012 02:35:21 -0700 (PDT) Return-Path: Received: from mail-we0-f170.google.com (mail-we0-f170.google.com [74.125.82.170]) by gmr-mx.google.com with ESMTPS id cq5si1240640wib.1.2012.10.09.02.35.21 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 09 Oct 2012 02:35:21 -0700 (PDT) Received-SPF: pass (google.com: domain of serge.ko...@gmail.com designates 74.125.82.170 as permitted sender) client-ip=74.125.82.170; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of serge.ko...@gmail.com designates 74.125.82.170 as permitted sender) smtp.mail=serge.ko...@gmail.com; dkim=pass header...@gmail.com Received: by mail-we0-f170.google.com with SMTP id x10so3050370wey.15 for ; Tue, 09 Oct 2012 02:35:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XDVjfB8JKD1Hn0O4XL8wCS56GXOEJ2x9Hw3q+Br+be0=; b=iOFDYhvU1FK4n0S5B6zOtr0DZejq1cEtG9Rg5IO17pdYEVKEdQ54UbGsOU5XU1F5GM IIko8KUlBVCixkUhwKsZjz4e8w02EPB8Ie+l32CAIIz74KNZ5HdKhZBtL83oFVUgdTnN Qp+49krua2wiXyUrx1DxCQiTfX2o7WkTbLJg/EegXRfcIexNN4wudaJAjhX9CkcVgp1a wLsT7tKMLHZTkfByQjpW7cbYel+GGe/Z2u1CtwNcdhZuPsnxfDlcqRD+bbDmvvfxMz/+ 78gsHTHaYYJ7kcqQ0uQGLNoSLLEsL+LWdFeuMWFCvcp+m6Z+CqkblYwO/c26sz6aF+EY jPVw== MIME-Version: 1.0 Received: by 10.180.90.78 with SMTP id bu14mr3161989wib.21.1349775271886; Tue, 09 Oct 2012 02:34:31 -0700 (PDT) Received: by 10.216.14.145 with HTTP; Tue, 9 Oct 2012 02:34:31 -0700 (PDT) In-Reply-To: <5073DBD0.3050...@ei-grad.ru> References: <421a0a63-332c-4560-b271-0c825699fe2d@googlegroups.com> <58e52be9-74c1-4ad1-8075-4a9bae12cb52@googlegroups.com> <5073DBD0.3050...@ei-grad.ru> Date: Tue, 9 Oct 2012 12:34:31 +0300 Message-ID: Subject: Re: [tornado] Making database connections in SockJS-Tornado From: "Serge S. Koval" To: Andrew Grigorev Cc: python-tornado@googlegroups.com Content-Type: multipart/alternative; boundary=f46d043c09389b7df204cb9d0b86 --f46d043c09389b7df204cb9d0b86 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable Redis protocol is synchronous, come on. Request in -> request out. You can't make requests to redis and receive replies out of order. Redis will block while executing request. But yes, asynchronous redis driver can work with redis, as internally it: 1. Sends request to redis 2. Waits for response from redis 3. Dispatches response to oldest listener So, if you do something like (pseudo code): def query(): redis.send('GET abc', callback=3Dcallback1) redis.send('GET def', callback=3Dcallback2) driver will: 1. Send 'GET abc', 'GET def' immediately 2. Wait for data from socket 3. Receive first reply: 'abc' and pass it to callback1 - because it was added to listeners queue first 4. Receive second reply: 'def' and pass it to callback2, because no one else left. How you can accomplish this with pipelining or without listeners queue? Serge. On Tue, Oct 9, 2012 at 11:09 AM, Andrew Grigorev wrote: > 09.10.2012 09:53, Serge S. Koval =D0=C9=DB=C5=D4: > >> On Tue, Oct 9, 2012 at 3:37 AM, Chris > bhp...@gmail.com>> wrote: >> > I assume that by 'async', you meant the db itself, not the driver for >> it, because there are async drivers for redis? >> DB and driver. >> > > OMG :-) > > Redis uses one-threaded asynchronous design. Like the most of tornado > applications. It is asynchronous. > There are many asynchronous clients for redis to use it with tornado. Som= e > of them are better, some of them - worse, but almost all of them are > asynchronous. :-)Redis usesa very simple protocol that allows you to easi= ly > write asynchronous clients. > > > For example, if you pick redis - you can issue command over active >> connection and you wait for response for that command over same connecti= on. >> You can not issue two commands over same connection and expect results t= o >> be received in proper order. >> > > You actually can. Read this for sure: http://redis.io/topics/**pipelining= > > >> However, if driver is smart enough, it can queue requests if there's one >> going already. In this case, you can have one redis connection per >> application and rely on the driver queue. Only problem with this approac= h - >> throughput, you won't be able to fully load redis with this approach. >> >> Serge. >> >> On Tue, Oct 9, 2012 at 3:37 AM, Chris > bhp...@gmail.com>> wrote: >> >> >> Hi Serge, >> >> Thank you for the tips. >> >> I assume that by 'async', you meant the db itself, not the driver >> for it, because there are async drivers for redis? >> >> On Monday, October 8, 2012 2:38:32 PM UTC-4, Serge S. Koval wrote: >> >> Hi, >> >> Best approach: if you use async capable DB (mongo, for >> example) - you can create one connection for your application. >> If you use non-async DB (redis, as it can't send replies >> asynchronously), you need to open new connection for each client= . >> 1. __init__ is called for every new client, but it is advised >> to do all initialization code in on_open, because created >> class does not mean that logic SockJS connection was established= . >> 2. No, don't do it. Having one connection per connected client >> is quite expensive by itself. Opening and closing connections >> for every message is even more expensive. >> >> Serge. >> >> >> On Mon, Oct 8, 2012 at 3:17 PM, Chris wrote: >> >> >> Hello, >> >> I am developing with sockjs-tornado library, and wonder >> what is the right way to make connections to database >> (e.g. mongo or redis) for each connected sockjs client. >> >> So far, I can think of two ways of doing it: >> >> 1. Establish the connection inside '__init__' constructor >> of my SockJSConnection subclass. The question is, will >> this connection stay open as long as the client is >> connected to sockjs-tornado? >> >> 2. Establish the connection inside 'on_message' method >> whenever the connected sockjs client needs to do db work, >> and close it when the work is done. Not sure if this is good= . >> >> Which of the above would be the right way to go with >> sockjs-tornado? And If there is better alternative, please >> do share! Thank you. >> >> >> >> > > -- > Andrew > > --f46d043c09389b7df204cb9d0b86 Content-Type: text/html; charset=KOI8-R Content-Transfer-Encoding: quoted-printable
Redis protocol is synchronous, come on. Request in -> request out.= =9AYou can't make requests to redis and receive replies out of order. R= edis will block while executing request.

But yes, = asynchronous redis driver can work with redis, as internally it:
1. Sends request to redis
2. Waits for response from redis
3. Dispatches response to oldest listener

So, if you do something like (pseudo code):
def query():
=9A redis.send('GET abc', callback=3Dcallback1)
=9A redis= .send('GET def', callback=3Dcallback2)

dri= ver will:
1. Send 'GET abc', 'GET def' immediatel= y
2. Wait for data from socket
3. Receive first reply: 'ab= c' and pass it to callback1 - because it was added to listeners queue f= irst
4. Receive second reply: 'def' and pass it to callba= ck2, because no one else left.

How you can accomplish this with pipelining or without = listeners queue?

Serge.

On Tue, Oct 9, 2012 at 11:09 AM, Andrew Grigorev = <and...@ei-grad.r= u> wrote:
09.10.2012 09:53, Serge S. Koval =D0=C9=DB= =C5=D4:
On Tue, Oct 9, 2012 at 3:37 AM, Chris <bhp...@gmail.com <mailto:bhp...@gmail.com>> wrote:
> I assume that by 'async', you meant the db itself, not the dri= ver for it, because there are async drivers for redis?
DB and driver.

OMG :-)

Redis uses one-threaded asynchronous design. Like the most of tornado appli= cations. It is asynchronous.
There are many asynchronous clients for redis to use it with tornado. Some = of them are better, some of them - worse, but almost all of them are asynch= ronous. :-)Redis usesa very simple protocol that allows you to easily write= asynchronous clients.


For example, if you pick redis - you can issue command over active connecti= on and you wait for response for that command over same connection. You can= not issue two commands over same connection and expect results to be recei= ved in proper order.

You actually can. Read this for sure: http://redis.io/topics/pipelining


However, if driver is smart enough, it can queue requests if there's on= e going already. In this case, you can have one redis connection per applic= ation and rely on the driver queue. Only problem with this approach - throu= ghput, you won't be able to fully load redis with this approach.

Serge.

On Tue, Oct 9, 2012 at 3:37 AM, Chris <bhp...@gmail.com <mailto:bhp...@gmail.com>> wrote:


=9A =9A Hi Serge,

=9A =9A Thank you for the tips.

=9A =9A I assume that by 'async', you meant the db itself, not the = driver
=9A =9A for it, because there are async drivers for redis?

=9A =9A On Monday, October 8, 2012 2:38:32 PM UTC-4, Serge S. Koval wrote:<= br>
=9A =9A =9A =9A Hi,

=9A =9A =9A =9A Best approach: if you use async capable DB (mongo, for
=9A =9A =9A =9A example) - you can create one connection for your applicati= on.
=9A =9A =9A =9A If you use non-async DB (redis, as it can't send replie= s
=9A =9A =9A =9A asynchronously), you need to open new connection for each c= lient.
=9A =9A =9A =9A 1. __init__ is called for every new client, but it is advis= ed
=9A =9A =9A =9A to do all initialization code in on_open, because created =9A =9A =9A =9A class does not mean that logic SockJS connection was establ= ished.
=9A =9A =9A =9A 2. No, don't do it. Having one connection per connected= client
=9A =9A =9A =9A is quite expensive by itself. Opening and closing connectio= ns
=9A =9A =9A =9A for every message is even more expensive.

=9A =9A =9A =9A Serge.


=9A =9A =9A =9A On Mon, Oct 8, 2012 at 3:17 PM, Chris <bhp...@gmail.com> wrote:


=9A =9A =9A =9A =9A =9A Hello,

=9A =9A =9A =9A =9A =9A I am developing with sockjs-tornado library, and wo= nder
=9A =9A =9A =9A =9A =9A what is the right way to make connections to databa= se
=9A =9A =9A =9A =9A =9A (e.g. mongo or redis) for each connected sockjs cli= ent.

=9A =9A =9A =9A =9A =9A So far, I can think of two ways of doing it:

=9A =9A =9A =9A =9A =9A 1. Establish the connection inside '__init__= 9; constructor
=9A =9A =9A =9A =9A =9A of my SockJSConnection subclass. The question is, w= ill
=9A =9A =9A =9A =9A =9A this connection stay open as long as the client is<= br> =9A =9A =9A =9A =9A =9A connected to sockjs-tornado?

=9A =9A =9A =9A =9A =9A 2. Establish the connection inside 'on_message&= #39; method
=9A =9A =9A =9A =9A =9A whenever the connected sockjs client needs to do db= work,
=9A =9A =9A =9A =9A =9A and close it when the work is done. Not sure if thi= s is good.

=9A =9A =9A =9A =9A =9A Which of the above would be the right way to go wit= h
=9A =9A =9A =9A =9A =9A sockjs-tornado? And If there is better alternative,= please
=9A =9A =9A =9A =9A =9A do share! Thank you.





--
Andrew


--f46d043c09389b7df204cb9d0b86--