haproxy on osv

[Asias He]

Hello guys,

I am trying to run haproxy on osv but it is not successful so far.

This is what I did:

host$ git clone http://git.1wt.eu/git/haproxy.git/
host$ cd haproxy
host$ git co -b osv v1.5-dev21
host$ make TARGET=generic CFLAGS="-fPIC -shared" LDFLAGS="-fPIC -shared" -j4
host$ cp -v haproxy /home/asias/src/cloudius-systems/osv/build/release/haproxy.so
host$ cp -v haproxy.cfg /home/asias/src/cloudius-systems/osv/build/release/haproxy.cfg
host$ cat haproxy.cfg
global
maxconn 4000
user root

defaults
mode http
timeout client 1m
timeout connect 10s
timeout server 1m

frontend main *:6000
default_backend app

backend app
balance roundrobin
server app1 ip...@192.168.122.1:6001 check

host$ cd $osv
host$ git diff origin/master
diff --git a/bootfs.manifest.skel b/bootfs.manifest.skel
index 9816e92..0a51b44 100644
--- a/bootfs.manifest.skel
+++ b/bootfs.manifest.skel
@@ -3,5 +3,7 @@
/zpool.so: zpool.so
/libzfs.so: libzfs.so
/zfs.so: zfs.so
+/haproxy.so: haproxy.so
+/haproxy.cfg: haproxy.cfg
/tools/mkfs.so: tools/mkfs/mkfs.so
/tools/cpiod.so: tools/cpiod/cpiod.so


host$ sudo ./scripts/run.py -nv -e "haproxy.so -f /haproxy.cfg -p haproxy.pid -d -V"
...

[I/31 dhcp]: Waiting for IP...
[I/322 dhcp]: Server acknowledged IP for interface eth0
[I/322 dhcp]: Configuring eth0: ip 192.168.122.89 subnet mask 255.255.255.0 gateway 192.168.122.1
Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 2 (1 usable), will use poll.
Using poll() as the polling mechanism.
[WARNING] 363/054355 (0) : Server app/app1 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 363/054355 (0) : backend 'app' has no server available!
...


host$ python -m SimpleHTTPServer 6001
----------------------------------------
Exception happened during processing of request from ('192.168.122.89', 48428)
Traceback (most recent call last):
File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
self.handle()
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 310, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 104] Connection reset by peer
----------------------------------------

Wireshark captured packets are attached. It seems the haproxy in osv
sends RST immediately and can not access the backend http server.

Your thoughts are highly appreciated!

--
Asias

[Benoît Canet]

Le Monday 30 Dec 2013 � 16:41:51 (+0800), Asias He a �crit :

Hi Asias,

I remember that the first time I tried this I had to explicitely select the
polling method on the haproxy command line by using the following switchs.
-de disables epoll() usage even when available
-ds disables speculative epoll() usage even when available
-dp disables poll() usage even when available
-dS disables splice usage (broken on old kernels)

Best regards

Beno�t

[Benoît Canet]

Le Monday 30 Dec 2013 � 16:41:51 (+0800), Asias He a �crit :

> Hello guys,
>
> I am trying to run haproxy on osv but it is not successful so far.

More precisely I used haproxy.so -f /etc/haproxy.conf -ds -de -dp with the
attached configuration.

Best regards

Beno�t

> --
> You received this message because you are subscribed to the Google Groups "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Pekka Enberg]

On Mon, Dec 30, 2013 at 10:53 AM, Benoît Canet <benoit...@irqsave.net> wrote:
> I remember that the first time I tried this I had to explicitely select the
> polling method on the haproxy command line by using the following switchs.
> -de disables epoll() usage even when available
> -ds disables speculative epoll() usage even when available
> -dp disables poll() usage even when available
> -dS disables splice usage (broken on old kernels)

That's an OSv bug, right? We should be able to make haproxy just work
out of the box?

Pekka

[Nadav Har'El]

On Mon, Dec 30, 2013 at 10:41 AM, Asias He <as...@cloudius-systems.com> wrote:

host$ make TARGET=generic CFLAGS="-fPIC -shared" LDFLAGS="-fPIC -shared" -j4

Eventually, you should probably switch to the Linux target, but for now I am guessing it won't work because I believe it uses splice(), which we need to implement. It also uses epoll(), which we have working, but as a thin compatibility layer over poll() - for great performance we'll need a bona-fide epoll() implementation.

 

backend app
    balance     roundrobin
    server app1 ip...@192.168.122.1:6001 check

host$ sudo ./scripts/run.py -nv  -e "haproxy.so -f /haproxy.cfg -p haproxy.pid -d -V"

...

[I/31 dhcp]: Waiting for IP...
[I/322 dhcp]: Server acknowledged IP for interface eth0
[I/322 dhcp]: Configuring eth0: ip 192.168.122.89 subnet mask 255.255.255.0 gateway 192.168.122.1
Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED

This doesn't explain the problem you're seeing, but it's not good :( Why would their select() test fail? It shouldn't... We probably have a bug in select() somehow, and should fix it.

Can you please try to see what they are testing? Unfortunately we cannot just copy their test into our tests/ directory because haproxy is GPL, but we can still see what they are testing.

--
Asias

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Nadav Har'El

n...@cloudius-systems.com

[Asias He]

Hello Beno�t,

On Mon, Dec 30, 2013 at 09:53:32AM +0100, Beno�t Canet wrote:
> Le Monday 30 Dec 2013 � 16:41:51 (+0800), Asias He a �crit :
>
> Hi Asias,
>
> I remember that the first time I tried this I had to explicitely select the
> polling method on the haproxy command line by using the following switchs.
> -de disables epoll() usage even when available
> -ds disables speculative epoll() usage even when available
> -dp disables poll() usage even when available
> -dS disables splice usage (broken on old kernels)
>
> Best regards

Thanks. So, you managed to run haproxy in osv with the config and options?
How did you compile haproxy. I used the generic TARGET.

--
Asias

[Asias He]

Now I got:

Available polling systems :
poll : pref=200, test result OK

select : pref=150, test result OK
Total: 2 (2 usable), will use poll.

Using poll() as the polling mechanism.

And I can not reproduce the failure of select anymore.

--
Asias

[Benoît Canet]

Le Monday 30 Dec 2013 � 17:09:49 (+0800), Asias He a �crit :

> Hello Beno�t,
>
> On Mon, Dec 30, 2013 at 09:53:32AM +0100, Beno�t Canet wrote:
> > Le Monday 30 Dec 2013 � 16:41:51 (+0800), Asias He a �crit :
> >
> > Hi Asias,
> >
> > I remember that the first time I tried this I had to explicitely select the
> > polling method on the haproxy command line by using the following switchs.
> > -de disables epoll() usage even when available
> > -ds disables speculative epoll() usage even when available
> > -dp disables poll() usage even when available
> > -dS disables splice usage (broken on old kernels)
> >
> > Best regards
>
> Thanks. So, you managed to run haproxy in osv with the config and options?
> How did you compile haproxy. I used the generic TARGET.
>

I hacked the linux target in the makefile to make it compile with -PIC -shared

> make TARGET=generic CFLAGS="-fPIC -shared" LDFLAGS="-fPIC -shared" -j4

Best regards

Beno�t

[Asias He]

On Mon, Dec 30, 2013 at 04:41:51PM +0800, Asias He wrote:
> Hello guys,
>
> I am trying to run haproxy on osv but it is not successful so far.

Updates:

It turns out that 'TARGET=linux26' makes haproxy work on osv while
'TARGET=generic' does not.

From the Makefile of haproxy, we can see that, 'TARGET=linux26' does not make
haproxy use splice which osv does not support.

ifeq ($(TARGET),linux26)
# This is for standard Linux 2.6 with netfilter and standard epoll()

ifeq ($(TARGET),linux2628)
# This is for standard Linux >= 2.6.28 with netfilter, epoll, tproxy and splice

I've sent out patches to add haproxy to our apps, so that make

'make image=haproxy'

will work.

--
Asias