Building a lua-resty-aerospike driver

[Vincent MADY]

Hi,

agentzh, do you plan to add an Aerospike driver in the official roadmap ? We really believe that Openresty + Aerospike is the perfect match for high frequency / low latency services just like onlime gaming or real time advertising.

So far it is possible to wrap the Aerospike C Client API so it can be called from Lua but as this C Client API is synchronous, it impacts badly on the overall performances (more information here : http://www.aerospike.com/community/labs/aerospike_lua_client.html)

Is there anybody interrested in joining us to build a nonblocking Lua Aerospike driver library for ngx_lua ?

Regards,

Vincent

[Yichun Zhang (agentzh)]

Hello!

On Mon, Mar 16, 2015 at 3:34 AM, Vincent MADY wrote:
> agentzh, do you plan to add an Aerospike driver in the official roadmap ?

Not really. I'd like to rely on the community to work on such things :)

> So far it is possible to wrap the Aerospike C Client API so it can be called
> from Lua but as this C Client API is synchronous, it impacts badly on the
> overall performances (more information here :
> http://www.aerospike.com/community/labs/aerospike_lua_client.html)
>

Take a look at existing lua-resty-* libraries based on the ngx_lua
cosocket API, like lua-resty-mysql and lua-resty-redis. Usually it
should be quite straightforward.

Regards,
-agentzh

[Alexander Altshuler]

понедельник, 16 марта 2015 г., 21:56:21 UTC+3 пользователь agentzh написал:

Aerospike client is active. It performs regular network activities on its own.

Aerospike C Client API spawn some threads for this purpose.

There is also Aerospike libevent C API which is able to run single thread, but main loop should be libevent.

So we cannot just use cosocket API. Am I right?

One possible approach:

Spawn OS thread for every nginx worker and run main loop by libevent. This thread will do all internal jobs to communicate with cluster and 

also will be intermediary (using unix socket for example) with nginx worker process.

In this way we may use cosocket API and make our client unblocking with price of additional thread per worker.

May be it is possible to write nginx C module to run Aerospike client completely in context of nginx worker?

Say like single-threaded version of Aerospike libevent C AP, but main loop will be nginx.

What do you think about?

 

Regards,

Alexander Altshuler

[Yichun Zhang (agentzh)]

Hello!

On Tue, Mar 17, 2015 at 3:55 AM, Alexander Altshuler wrote:
> Aerospike client is active. It performs regular network activities on its
> own.
> Aerospike C Client API spawn some threads for this purpose.
>
> There is also Aerospike libevent C API which is able to run single thread,
> but main loop should be libevent.
>
> So we cannot just use cosocket API. Am I right?
>

I didn't suggest you reuse the existing C library. Rewriting the
driver in Lua atop cosockets is almost always easier.

> One possible approach:
> Spawn OS thread for every nginx worker and run main loop by libevent. This
> thread will do all internal jobs to communicate with cluster and
> also will be intermediary (using unix socket for example) with nginx worker
> process.

This is sub-optimal (extra OS threads) and complicated (OS thread
synchronization).

> May be it is possible to write nginx C module to run Aerospike client
> completely in context of nginx worker?

Even for nginx C module development, reusing existing C library is
harder than coding the client from scratch (though still much harder
than the cosocket Lua approach). Just see the ngx_drizzle module for
MySQL (using libdrizzle) and ngx_postgres module for PostgreSQL (using
libpq) for evidences.

Regards,
-agentzh

[Alexander Altshuler]

Hi

Thanks for fast answer.

17 March 2015., 22:15:00 UTC+3 user agentzh wrote:

Hello!

On Tue, Mar 17, 2015 at 3:55 AM, Alexander Altshuler wrote:
> Aerospike client is active. It performs regular network activities on its
> own.
> Aerospike C Client API spawn some threads for this purpose.
>
> There is also Aerospike libevent C API which is able to run single thread,
> but main loop should be libevent.
>
> So we cannot just use cosocket API. Am I right?
>

I didn't suggest you reuse the existing C library. Rewriting the
driver in Lua atop cosockets is almost always easier.

lua-resty-redis and lua-resty-mysql use simple one-TCP-connection-per-database object model.

Aerospike client connect to set of servers, but it does it internally, transparent for user.

Below is quote from http://www.aerospike.com/docs/architecture/clients.html:

===============8<===========================

The Aerospike Client API has several capabilities essential for high performance and easier development:

Tracks cluster state so that you, the developer, do not have to. At any given instant, the client uses the info protocol to communicate periodically with the cluster and maintain a list of nodes that form the cluster. It also uses the Aerospike Smart Partitions™ algorithm to determine which node stores a particular partition of data. Any changes to the cluster size are tracked automatically by the Client; such changes are entirely transparent to the Application. In practice, this means that transactions will not fail during the transition, and the Application does not need to be restarted during node arrival and departure.

===============8<===========================

Do you really think HttpLuaModule with cosocket API has enough power to write such complex client?

Regards,

Alexander Altshuler

[Lord Nynex]

If the resty cassandra driver has this capability, I'm sure aerospike is in the realm of possibility. 

While the marketing blurb you've posted seems like a lot of voodoo and dark magic, it seems pretty attainable. The only questionable bit would be the info protocol (whatever that is) running outside the request context. This is possible with timers, though. Also you may be able to fake it with some architectural creativity. As an example, I've seen many large business run something tantamount to haproxy on localhost to maintain persistent connections to an upstream DB. This way the code only needs to talk to localhost on a specific port. Maybe aerospike has a similar project?

There are plenty of databases I'd like to talk to from openresty but am limited by my own free time for driver development. Perhaps your company should sponsor some driver development if it is that integral to your business. 

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Altshuler]

Hi

Thanks for reply

On March 18 2015., 22:07:27 UTC+3 user Lord Nynex wrote:

If the resty cassandra driver has this capability, I'm sure aerospike is in the realm of possibility. 

While the marketing blurb you've posted seems like a lot of voodoo and dark magic, it seems pretty attainable. The only questionable bit would be the info protocol (whatever that is) running outside the request context. This is possible with timers, though. Also you may be able to fake it with some architectural creativity. As an example, I've seen many large business run something tantamount to haproxy on localhost to maintain persistent connections to an upstream DB. This way the code only needs to talk to localhost on a specific port. Maybe aerospike has a similar project?

Yes, I dig a bit into aerospike client internals. They C client library is totally mess.

It is possible to implement all required logic for key-value store and UDF operation.

Tend thread's job (which use info protocol) may be implemented with timers or in some other transparent way.

 

Regards

Alexander Altshuler

[Yichun Zhang (agentzh)]

Hello!

On Tue, Mar 17, 2015 at 11:38 PM, Alexander Altshuler wrote:
> lua-resty-redis and lua-resty-mysql use simple
> one-TCP-connection-per-database object model.
>
> Aerospike client connect to set of servers, but it does it internally,
> transparent for user.

I don't see why it can be a hurdle. The lua-resty-redis and
lua-resty-mysql libraries use the one-TCP-connection-per-data model
just because they only need this simple model. I don't see why you
cannot just create multiple connections for a single abstract
high-level object. (In OO, an object can be an aggregation object
manipulating many low-level objects internally.) It's Lua anyway.

Regards,
-agentzh

[Alexander Altshuler]

Hi

Thanks, I have decided to go this way.
And using ngx.timer.at() from init_worker_by_lua to initiate cluster info polling.

Regards,
Alexander Altshuler

[Mark]

Check slide 17/40 from http://www.slideshare.net/AerospikeDB/aerospike-comparing-nosql-databases-for-rtb    for "ngx-aerospike"

Looks like someone has already written ngx-aerospike connection driver (btw - sorry, that's not us. So can't help with this)

Regards

Mark 

Dne petek, 20. marec 2015 09.07.22 UTC+1 je oseba Alexander Altshuler napisala:

[Alexander Altshuler]

On Tuesday, April 14, 2015 at 6:32:48 PM UTC+3, Mark wrote:

Check slide 17/40 from http://www.slideshare.net/AerospikeDB/aerospike-comparing-nosql-databases-for-rtb    for "ngx-aerospike"

Looks like someone has already written ngx-aerospike connection driver (btw - sorry, that's not us. So can't help with this)

Yes I know about this slides, but this implementation is closed and I believe just binding to C aerospike client - really no make sense for me in this context.

From these slides: how to shoot yourself in the foot - nginx + blocking IO

PS: Sorry for late reply.

[les2...@gmail.com]

Hi, is there any project implemented openresty --> aerospike so far ?

All the projects I can find using ngx_lua cosocket are using Lua to implement wire protocol:

lua-resty-memcached

lua-resty-redis library 

lua-resty-mysql library 

lua-resty-upload library 

lua-resty-dns library 

lua-resty-websocket 

But the aerospike client to server protocol is more complicated than sending simple client request and server response, it also use the TCP connection to load seed node, update the data distribution in the server cluster periodically, check heartbeat, etc with thread pool & lock.  This is making LUA implementation of wire protocol very difficult.

Is there an example of how to modify a C client with blocking I/O to make it work with openresty ?

Is the following the right way to do it ?

1. find all the socket related calls and comment out these calls:

connect

send

receive

close

settimeout

setoption

receiveuntil

setkeepalive  

2. save the data/buffer that need to be sent/receive somewhere

3. call lua_yield before the above socket call

4. let nginx to send/receive the data/buffer by using cosocket's 

sock:connect()

sock:send()

sock:receive()

5. call lua_resume to resume the execute of  statement right after the above socket call, but how ?

Thanks a lot,

Tim

[Thibault Charbonnier]

Hi,

On Dec 23, 2017, at 8:56 PM, les2...@gmail.com wrote:

Hi, is there any project implemented openresty --> aerospike so far ?

All the projects I can find using ngx_lua cosocket are using Lua to implement wire protocol:

lua-resty-memcached

lua-resty-redis library 

lua-resty-mysql library 

lua-resty-upload library 

lua-resty-dns library 

lua-resty-websocket 

But the aerospike client to server protocol is more complicated than sending simple client request and server response, it also use the TCP connection to load seed node, update the data distribution in the server cluster periodically, check heartbeat, etc with thread pool & lock.  This is making LUA implementation of wire protocol very difficult.

The lua-cassandra driver is another of those binary protocol implementations which has to deal with similar constraints as the one you just mentioned to maintain a certain state of the cluster, load balance between nodes, provide retry and reconnection policies, etc...

Thibault

--