Duplicate traffic to staging environment using cosocket based library

[Vladimir Varfolomeev]

Hello Yichun Zhang,

Can you please provide a way to duplicate every requests to two different upstreams  - production and staging environment. Clients requests goes to openresty and mirroring  to two different upstreams(prod and stg). I would like to ignore staging responses and forward to the client response from production upstream only. Also it should be coscocket solution since it would be nice to avoid any delay for production environment if staging environment  is unavailable.

So what i wrote is pretty common task about trafic duplication. You allready discussed similiar issue here: https://www.ruby-forum.com/topic/6179585   but i am unable to write lua script  which will use  coscoket library https://github.com/bakins/lua-resty-http-simple  for this purporse. 

As for know i have tried the following ways:

• lua cosocket based library lua-resty-http-simple  - probably best way to go but need help to set it up
• lua ngx.location.capture

This kind of solution does work:

 ngx.req.read_body()

local arguments = ngx.var.args

r1 = ngx.location.capture('/prod/', { args = arguments, share_all_vars = true })

ngx.print(r1.body)

ngx.eof()

r2 = ngx.location.capture('/dev/', { args = arguments, share_all_vars = true })

but i am not happy with this because of ngx.eof() and possible delay to production environment in case of staging is unavailable.

• tcpcopy 

Another option is go with tcpcopy project: https://github.com/session-replay-tools/tcpcopy   

It also does work for some time in production environment but i have copied about half of the production trafic(i monitor incoming/outgoing network interface activity to figure this out)

• openresty + zeroMQ  http://glipho.com/albert/http-traffic-duplication-with-nginx  Looking promising, but i did not find any use case in production systems
•  gor https://github.com/buger/gor/ looking like tcpcopy project but i did not test it yet.

It would be nice if you can make  some example of lua script for this kind of  task here : http://openresty.org/download/agentzh-nginx-tutorials-en.html.  Sure i will be not the only person who benefit from it. From my site i can provide extra details or perform troubleshooting steps.

Regards Vladimir

[Yichun Zhang (agentzh)]

Hello!

On Tue, Mar 31, 2015 at 8:11 AM, Vladimir Varfolomeev wrote:
> Another option is go with tcpcopy project:
> https://github.com/session-replay-tools/tcpcopy
> It also does work for some time in production environment but i have copied
> about half of the production trafic(i monitor incoming/outgoing network
> interface activity to figure this out)
>

IMHO, it's better done on the lower level than nginx. Even if tcpcopy
does not do the job, it should be something on a similar level.

If you really want to do this in nginx, then the ngx.location.capture
+ ngx.eof approach is the simplest approach to my knowledge. Other
than that, you can use ngx.timer.at + some version of lua-resty-http
to do the duplicated off-line requests asynchronously.

Regards,
-agentzh

[Lord Nynex]

agentzh,

Couldn't you do this with capture_multi and never read the response? Also, what happens if you spawn a thread and never wait() for it?

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Birdsong]

I've found success by using the tcpcopy approach, but I've found it's much easier to configure and reason about a system that uses some sort of offline queueing which drives a set of workers to simulate user traffic. This way I can investigate the server's response from the client's perspective and, if needed, I can mutate the request on the queue client.

I use log_by_lua to write to Heka for logging, then use Heka + NSQ for creating work. The latter part means I can leverage the work put into the streaming log system that's already in place.

You could swap in Heka with a number of stream systems and NSQ with any number of queue systems. Even log_by_lua's not strictly necessary, parsing nginx logs could work too.

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en+unsubscribe@googlegroups.com.

[Yichun Zhang (agentzh)]

Hello!

On Tue, Mar 31, 2015 at 12:54 PM, Lord Nynex wrote:
> Also,
> what happens if you spawn a thread and never wait() for it?
>

The request handler creating the light threads will automatically wait
for those light threads to complete unless the light threads are
detached from the request handler by means of ngx.timer.at().

Regards,
-agentzh