Azure Load Balancer probe not able to receive tcp ack from fluentd port 24224

[Arun John Varughese]

Hi,

I guess there is an issue where the Azure load balancer probe is not able to get a sucessfull tcp ack, to tell the LB that the fluent  instance is available.

This might be a problem with Azure Load balancer, but just wanted to know if this was an issue on fluentd side ?

I have posted this also in MSDN forums : https://social.msdn.microsoft.com/Forums/azure/en-US/fca163f9-0d41-4a42-8278-0caaed91b3ff/azure-load-balancer-rule-not-working

[Arun John Varughese]

I am using the secure_forward plugin to send data from the forwarder to the aggregator with the LB in between. 

Fleuntd_Forwarder > [Azure Load Balancer : TCP port 24224] > Fluentd_aggregator

Thanks

AJV

[Arun John Varughese]

I compared this with another application that makes use of port 443 and used the forward plugin rather than the secure plugin.

And from the tcpdump you can see that for the application that works on 443 was able to receive acknowledgement from the load balancer, which could have come after the application would have sent out a TCP ack.

[root@rm-sftynet-qa01-devicefiltering-apivm01 ~]# tcpdump -i eth0 src 168.63.129.16 and port 443

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

11:53:52.978372 IP 168.63.129.16.54754 > 10.0.0.4.https: Flags [F.], seq 2030601259, ack 3743586656, win 513, length 0

11:53:52.978690 IP 168.63.129.16.54835 > 10.0.0.4.https: Flags [SEW], seq 2814343234, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:53:52.978928 IP 168.63.129.16.54754 > 10.0.0.4.https: Flags [.], ack 2, win 513, length 0

11:53:52.978931 IP 168.63.129.16.54835 > 10.0.0.4.https: Flags [.], ack 3633393567, win 513, length 0

11:53:57.978406 IP 168.63.129.16.54835 > 10.0.0.4.https: Flags [F.], seq 0, ack 1, win 513, length 0

11:53:57.979295 IP 168.63.129.16.54847 > 10.0.0.4.https: Flags [SEW], seq 3867076172, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:53:57.979958 IP 168.63.129.16.54835 > 10.0.0.4.https: Flags [.], ack 2, win 513, length 0

This is how it looks on the fluentd side where it did not receive any ack message from the LB.

11:58:29.904604 IP 168.63.129.16.50327 > 10.4.1.11.24224: Flags [SEW], seq 777293687, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:32.904366 IP 168.63.129.16.50327 > 10.4.1.11.24224: Flags [SEW], seq 777293687, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:34.904905 IP 168.63.129.16.50362 > 10.4.1.11.24224: Flags [SEW], seq 2497065274, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:37.904536 IP 168.63.129.16.50362 > 10.4.1.11.24224: Flags [SEW], seq 2497065274, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:39.905295 IP 168.63.129.16.50409 > 10.4.1.11.24224: Flags [SEW], seq 3072265725, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:42.944834 IP 168.63.129.16.50409 > 10.4.1.11.24224: Flags [SEW], seq 3072265725, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

11:58:44.905535 IP 168.63.129.16.50463 > 10.4.1.11.24224: Flags [SEW], seq 1070785877, win 8192, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0

 

So the question is ..does the fluentd service send back a TCP ack back ?

[Satoshi Tagomori]

Of course, fluentd (in_secure_forward) return TCP ack, because TCP requires it.

2016年4月5日火曜日 21時14分22秒 UTC+9 Arun John Varughese:

[Arun John Varughese]

Hi Satoshi,

Thanks for confirming.

The reason why I asked this was because in the documentation mentioned in https://github.com/tagomoris/fluent-plugin-secure-forward

Under 'Data transferring' > Current version has no ACKs > only supports burst transferring (same as ForwardInput/Output)

Can you please comment on the above statement mentioned in the documentation, just to make sure I understand how this works.

Thanks and Regards,

AJV

[Arun John Varughese]

 I had followed some pre-installation steps which seems to have caused the issue for the Azure load balancer under 'Optimize Network Kernel Parameters'http://docs.fluentd.org/articles/before-install The attribute net.ipv4.tcp_tw_recycle is the one that had caused issues to the LB. A discussion on that can be found here in this link.

On removing this config I was able to see acknowledgments from the Azure LB IP using the command on the VM's :

tcpdump -i eth0 src 168.63.129.16 and port 24224

Regards,

AJV

On Tuesday, April 5, 2016 at 7:31:02 PM UTC+5:30, Satoshi Tagomori wrote:

[Mr. Fiber]

> Optimize Network Kernel Parameters

This is for high load environment when normal setting had a problem.

So if you don't have a problem, no need this parameter changes.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.