How to use NXWEB as proxy

[Eric Chang]

Hi Yaroslav,

First of all, NXWEB is very nice and efficient.  I like NXWEB.

I compiled and install the NXWEB successfully.  It works very well as a web server.  But what I plan to do is intercept http/https requests and return error page if that URL is not allowed. 

After running NXWEB, the follow messages are shown:

2013-01-21 18:02:39 [3259:0x7fd0622ab700]: nxweb binding :8055 for http
2013-01-21 18:02:39 [3259:0x7fd0622ab700]: nxweb binding :8056 for https
2013-01-21 18:02:39 [3259:0x7fd0622ab700]: proxy backend #0: localhost:8000
2013-01-21 18:02:39 [3259:0x7fd0622ab700]: proxy backend #1: localhost:8080

If I setup my firefox's proxy http setting to 8055 and SSL settings to 8056, NXWEB acts as web server and return me the "hello" page.

If I setup my firefox's proxy http setting to 8000, NXWEB rejects my connection.

I do read all posts in this group and studied the official wiki page but can not find proxy info.  Please teach me how to use NXWEB as proxy.

Thanks a lot!

[Yaroslav]

Hi Eric,

You do not need to setup proxy in Firefox. Just request http://localhost:8055/ or https://localhost:8056/ to access nxweb.

nxweb can proxy requests coming to it via http/https ports (8055/8056 by default) to external servers configured by default at ports 8000 and 8080. You have to change proxy url prefixes that get forwarded to those external servers and their port numbers in main.c according to your specific needs. Use default setup as a sample.

Please read more about proxy functionality here: http://en.wikipedia.org/wiki/Proxy_server

I feel you do not quite understand what it is.

Yaroslav

[Eric Chang]

Sorry for not describing the scenario clearly.

Since what I want to do is intercept http/https request and check its URL, my plan is set up an NAT gateway and try to run nxweb as a transparent proxy.

The NAT gateway has eth0 and eth1 (192.168.1.1).   eth0 is WAN.  eth1 is LAN.  The nxweb is running on this gateway,
----------------------------------------------------------------
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

/sbin/iptables -t nat -A PREROUTING -i eth1 -p tcp --dport  80 -j REDIRECT --to-port 8055
/sbin/iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8056
----------------------------------------------------------------

Therefore, all HTTP (port 80) request from eth1 are redirected to NXWEB's port 8055.   all HTTPS (port 443) requests from eth1 are redirected to NXWEB's port 8056.

The LAN PC is 192.168.1.21 with Firefox running.  

In this transparent proxy case, if I browse a normal website, NXWeb does not forward HTTP request to the real destination web servers and return me the "hello" page.

So I set up the proxy server as "192.168.1.1:8055" in Firefox manually, but NXWEB still return me the "hello" page.

I do not know why NXWEB is not forwarding my HTTP requests to external web servers.

I installed and run Squid cache proxy server before.  Squid can act as a transparent proxy but is very fat -
http://www.rahulpahade.com/content/squid-transparent-proxy-over-ssl-https

So I am thinking NXWEB is more suitable to me.  Sorry for bothering you.



Yaroslav於 2013年1月21日星期一UTC+8下午6時24分18秒寫道：

[Yaroslav]

Eric,

There are two types of proxy servers: FORWARD and REVERSE.

Here is the explanation:

http://stackoverflow.com/questions/224664/difference-between-proxy-server-and-reverse-proxy-server

Here it is with pictures:

http://www.jscape.com/blog/bid/87783/Forward-Proxy-vs-Reverse-Proxy

If I understand you right you need FORWARD proxy while nxweb can only act as REVERSE one.

If you are looking for something lighter than squid, take a look at:

http://en.wikipedia.org/wiki/Polipo

http://en.wikipedia.org/wiki/Tinyproxy

Yaroslav

[Eric Chang]

Hi Yaroslav,

Many thanks for your lessons.  I understand now.
Yes!  What I want is a forward proxy.

I have studied Polipo and Tinyproxy.  
Tinyproxy supports TLS a little bit but use a fork-process approach.
Polipo does not support TLS and use poll() only now.
I hope an epoll() based solution.

Many thanks for your time.  Nxweb inspires me a lot. I like the design of Nxweb very much.

[Andries]

Hi Yaroslav,

Is it possible to use (or to manipulate) the proxy handler so I could make http get requests to the backend (so I could get a response from the bakcend)?

[Yaroslav]

Hi Andries,

If you want to process result of backend request you have to write a filter and use it over proxy handler. See templates filter for example.

You can also make subrequests from your handler, then process their results in callback. See nxweb_http_server_subrequest_start() function.

Yet another approach is to open synchronous tcp connection to backend from a worker thread. This is probably the simplest as you don't need to dig into nxweb code to accomplish that.

Yaroslav

On Wed, Aug 21, 2013 at 10:53 PM, Andries <andrie...@gmail.com> wrote:

Hi Yaroslav,

Is it possible to use (or to manipulate) the proxy handler so I could make http get requests to the backend (so I could get a response from the bakcend)?

--
You received this message because you are subscribed to the Google Groups "nxweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nxweb+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Andries]

Hi Yaroslav,

I'm trying the subrequest method, but I keep getting this error:

backend_request_ready: Assertion `beb' failed.

Aborted

So, this is what I have so far:

char *host = "mybackend:80";

char *url = "/test";

nxweb_http_server_connection* subconn = nxweb_http_server_subrequest_start(conn, backend_request_ready, host, url);

static void backend_request_ready(nxe_data data)

{

        nxweb_http_server_connection* subconn = (nxweb_http_server_connection*)data.ptr;

nxweb_http_request* subreq = &subconn->hsp.req;

nxweb_http_server_connection* conn = subconn->parent;

nxweb_http_response* resp = subconn->hsp.resp;

be_buffer* beb = nxweb_get_request_data(subreq, BEB_KEY).ptr;

assert(beb);

...

It's always crashing at this assertion.

I also added these lines in the program:

static const char beb_key;

#define BEB_KEY ((nxe_data)&beb_key)

Do you have any idea?

[Yaroslav]

Subrequest goes to the same nxweb server. You can not make subrequest to backend directly. You can setup proxy handler for some uri "/my-backend-uri", then make subrequest to that uri, which will invoke proxy_handler with corresponding parameters.

Please note also that backend_request_ready() will be called immediately after proxy handler _starts_ receiving response from backend, not after the whole response will be received. You have to plug something like nxd_ibuffer into subrequest's content_out stream in order to buffer the data.

You have to set request data with nxweb_set_request_data() with the same key before calling nxweb_get_request_data().

--

[Andries]

Hi Yaroslav,

I am digging into the code of nxweb to buffer my backend request via subrequest and proxy handler, but I can't find it out.

I think I will make a spinoff of the nxweb_composite_stream_start() function that you use in the subrequest program.

I was looking into the code to find out where I can buffer the response in a variable in stead of streaming it out to the response, but I can't figure it out.

I think the route the function follows is this:

=> nxd_streamer_start > nxd_streamer_node_start > nxe_ostream_set_ready > nxe_link

Can you help me with this? Where exactly can I buffer the response?

Thanks,

Andries

[Yaroslav]

Hi Andries,

It should be something like the following:

on_buffer_done(content, good) {

  // do something with subrequest response

}

// from nxd_buffer.c clone the source of ibuffer

// (2 functions: ibuffer_data_in_do_read and nxd_ibuffer_init)

// - rename it mybuffer

// replace line: nxe_publish(&ib->data_complete, (nxe_data)-1/0);

// with this line: on_buffer_done(ib->data_ptr, 0/1);

static void mybuffer_data_in_do_read(nxe_ostream* os, nxe_istream* is) {

  ...

  on_buffer_done(ib->data_ptr, 0);

  ...

  on_buffer_done(ib->data_ptr, 1);

  ...

}

static const nxe_ostream_class mybuffer_data_in_class={.do_read=mybuffer_data_in_do_read};

void mybuffer_init(nxd_ibuffer* ib, nxb_buffer* nxb, int max_data_size) {

  ...

  ib->data_in.super.cls.os_cls=&mybuffer_data_in_class;

  ...

}

on_response_ready(subconn) {

  // connect mybuffer to response content_out to collect the response body

  int status=subconn->hsp.resp->status_code;

  if (!subconn->subrequest_failed && (!status || status==200) && subconn->hsp.resp->content_length) {

    mybuffer_init(&ib, subconn->hsp.nxb, subconn->hsp.resp->content_length>0? subconn->hsp.resp->content_length+1 : NXWEB_MAX_REQUEST_BODY_SIZE);

    nxe_connect_streams(subconn->tdata->loop, subconn->hsp.resp->content_out, &ib.data_in);

  }

}

// initiate subrequest

subconn=nxweb_http_server_subrequest_start(cs->conn, on_response_ready, host, url);

--

[Andries]

Hi Yaroslav,

How exactly do I need to initiaze ib for mybuffer_init?

 if (!subconn->subrequest_failed && (!status || status==200) && subconn->hsp.resp->content_length) {

    mybuffer_init(&ib, subconn->hsp.nxb, subconn->hsp.resp->content_length>0? subconn->hsp.resp->content_length+1 : NXWEB_MAX_REQUEST_BODY_SIZE);

If I do nxd_ibuffer ib; ==> endless loop

If I do mybuffer_init(&conn->ib, ...) ==> endless loop

If I do mybuffer_init(&subconn->ib, ...) ==> segmentation fault

[Yaroslav]

This is your module's variable you have to allocate it somewhere. You can allocate it from nxb and store reference in request_data, for example.