Consul Gossip Firewall Rules
198 views
Skip to first unread message
Marek Slabicki
Feb 26, 2018, 8:59:13 PM2/26/18
to Consul
Hello,
Solution:
A potential solution I can think of is to identify patterns within the packets being transmitted, and allow traffic between network segments using ONLY the Serf Gossip protocol employed by Consul.
This would be done by making "Application Signatures" in a Palo Alto firewall.
How I need help:
Problem: A segmented network does not allow traffic between VLANs. In order to get around it I want to restrict traffic to Serf Gossip Protocol only. As opposed to opening up a universal tunnel on 8301 TCP/UDP.
Details:
It is my understanding that every consul agent must be able to reach any other consul agent on port 8301 using TCP or UDP. Because of a segmented network this is not an option for me.
It is my understanding that every consul agent must be able to reach any other consul agent on port 8301 using TCP or UDP. Because of a segmented network this is not an option for me.
Solution:
A potential solution I can think of is to identify patterns within the packets being transmitted, and allow traffic between network segments using ONLY the Serf Gossip protocol employed by Consul.
This would be done by making "Application Signatures" in a Palo Alto firewall.
How I need help:
When using tcpdump to analyse traffic, I cannot identify human readable patterns of the packet contents that could be used to generate an application signature.
Does anyone have any wisdom they can impart on identifying patterns in the packets, both TCP and UDP?
Does anyone have any wisdom they can impart on identifying patterns in the packets, both TCP and UDP?
Example tcpdump with -A flag:
root@agent1:~# service consul restart; tcpdump -s0 -vvv port 8301 -Atcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes01:22:41.745014 IP (tos 0x0, ttl 64, id 21175, offset 0, flags [DF], proto UDP (17), length 894)agent1.8301 > server2.8301: [bad udp cksum 0x2b78 -> 0x32cd!] UDP, length 866E..~R.@.@...... e m m.j+x.o....F.U8......|.._....... ........<.w\........tA|&.................K.p.Vv..8...1.5.....+;.!...u.WL.T..$.s.D.eN....P...~MgB..7_4^vn..X.@%..d&..c4...h......O.n.o..........o=..\....?.Ns.9.........v...Q..Bir.*...{&G..7<......<#..Eu..~........kr...p.....|.P54@..<..R~...jF.X.KR$vf...7.J;P...&..L8c.,BuDp.x........-.....{.o.... ...M..H.>..r.......*..&;ut....O....6.Q...wb.....iZ....5;5_..I|#...)~J..J.`....z..46..<3B.Q.+....z}.A..o.2......7~d..$.O...._;.d.......KG...X...Ge...B../,...M.M... .ge..6...M..uv...H.......QQ..........7vL&.......$..t.0f."...V...$n.......w..}...QS#p.].c}Z..TX....k..B.`.aB..x.....J.A..y..@Q)*Xa......].....JA.h6B.....|...X...=.'...mc?..D.X$X.]z..1df.....;@wr..p.0....,..La..8~..}Cb.....R..rD..at.{@.......v...6.b..eB..'....Bh....~{.....c'..l[.u.-ql..9..5B.r..I.[t..!..0..i^M.m....A..._<]E.....30..T.b......AO...-{&.....lZ.'......b.{..G6....01:22:41.745143 IP (tos 0x0, ttl 64, id 29039, offset 0, flags [DF], proto UDP (17), length 894)agent1.8301 > server3.8301: [bad udp cksum 0x2b79 -> 0xa106!] UDP, length 866E..~qo@.@...... f m m.j+y..i.4.w..=w.X.L..9......`p..n.PPL.z9.)#hE.q.....wG.,...'@^`.;..c3..u=...$<..n..LXpi{..3 i.*..c.$.B..;.. ......[*~..aXjJ..=..............h.\.`v.....U..:.,N.....(.=....B.^.......%...!..4..e..qgg....x.!..4.J....5<..WdG....?x.*.qF#.....M.h(T*..^....Q0.f..`.....c._M...........K.=.,...!....lw....X.2....g..F...J.g.PMhb+.m....KO..;m..SA@.W.N*..kI...2...V ....S.t."..{...&....;J.......k..)..s...'.c.(.c.30MMB.2z..m'K..S(s........u...../X.J......f.k......f........s...w......nY.......$..R-.z...e. ....K....j.w.>..i..Zj........,...:.0....d_....c..G-.v.J'..A.. ..;>..@tB...O...O.....nu......O..r...A.=..W..&...i1...[..9.......r...H..oK..........a.l...7...X........,.C....-*.....uN..Rd.C......J..v...;~.....'s..z..W_:.%.n....x*.L.Z.8f....C'.(p.|.....1.EB.....l.2..n=.4.G0...v:...E....U`. .X..h9...\...fmuA....a..&S....'.]..hP...YH......V{c....n....#.....s.H..~v=.].........
Example tcpdump with -X flag:
root@agent1:~# service consul restart; tcpdump -s0 -vvv port 8301 -Xtcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes01:53:47.787299 IP (tos 0x0, ttl 64, id 35298, offset 0, flags [DF], proto UDP (17), length 132)agent3.8301 > agent1.8301: [udp sum ok] UDP, length 1040x0000: 4500 0084 89e2 4000 4011 87cc 0a01 0a23 E.....@.@......#0x0010: 0a01 0a96 206d 206d 0070 ba74 01e4 d3a2 .....m.m.p.t....0x0020: 38dc 8f5b 8ea1 1961 694b 1492 22bc 5a8e 8..[...aiK..".Z.0x0030: 9958 2998 964d b64d 7c15 de71 89ad efac .X)..M.M|..q....0x0040: 4a05 b7bd dc22 397d 86a7 edfb 0a36 09d0 J...."9}.....6..0x0050: c513 02dd 1278 b38a cf53 6c41 fb82 f3f1 .....x...SlA....0x0060: b50a cf9c 8761 54fe 48aa bdc5 0ef3 e46a .....aT.H......j0x0070: a065 3313 513e 1081 b896 0d6f 3d46 4c24 .e3.Q>.....o=FL$0x0080: bb72 c2b2 .r..01:53:47.788026 IP (tos 0x0, ttl 64, id 64182, offset 0, flags [DF], proto UDP (17), length 986)agent1.8301 > agent3.8301: [bad udp cksum 0x2c92 -> 0x957f!] UDP, length 9580x0000: 4500 03da fab6 4000 4011 13a2 0a01 0a96 E.....@.@.......0x0010: 0a01 0a23 206d 206d 03c6 2c92 0164 315f ...#.m.m..,..d1_0x0020: daf0 dcca d2cb 183f 9386 af11 01be a2ad .......?........0x0030: 9c00 23ae 1980 eee6 40e1 b666 df3e 09e1 ..#.....@..f.>..0x0040: cbc0 10da b7a4 7f16 8cf8 cfbb 3384 4bf8 ............3.K.0x0050: be09 6da1 b9b4 7551 7761 30c6 9bb6 481a ..m...uQwa0...H.0x0060: 4e76 8520 1724 ed7a dccd c8a0 9a90 ee6b Nv...$.z.......k0x0070: ac01 a0dd df04 8f3d 2fb2 51c2 5361 8fa0 .......=/.Q.Sa..0x0080: 151d 81fd 9141 e57f 5a70 5619 6230 7859 .....A..ZpV.b0xY0x0090: d7e7 84a4 6c82 bdbd 2839 6b20 135a 8a4e ....l...(9k..Z.N0x00a0: c19a 16b3 b6dd 7269 f28f 6c58 5dc6 c9bd ......ri..lX]...0x00b0: 76c6 f223 1588 d133 b9f2 cf95 c611 d70e v..#...3........0x00c0: 2302 9b9c a084 d27c 7cb9 f72b cb35 c419 #......||..+.5..0x00d0: 61b0 f42d 7936 aa0e a7c2 6868 d8d9 d166 a..-y6....hh...f0x00e0: 50d3 a1a7 e7ef 1232 ad56 036d 83cf adc5 P......2.V.m....0x00f0: 4449 3b79 ca73 4212 f3df 7ef5 6583 6484 DI;y.sB...~.e.d.0x0100: e99e d8f0 1531 6481 0a81 c35f d94d c4df .....1d...._.M..0x0110: c655 d312 1c9f 4bcf 9152 5822 9f39 2af4 .U....K..RX".9*.0x0120: 2d7c b9e0 a4a1 19ed b521 778b 7549 3ba0 -|.......!w.uI;.0x0130: fbe4 caab 83df 7d2b 6036 fc49 45c4 8fe0 ......}+`6.IE...0x0140: fffb 680a 5676 701a 8365 4e16 22ee b5f6 ..h.Vvp..eN."...0x0150: 1e89 4225 040b 1748 5a92 7375 a5f3 52de ..B%...HZ.su..R.0x0160: 4827 6d3b 5410 0ac1 6df1 9019 0591 4314 H'm;T...m.....C.0x0170: 62f5 30ee f933 1d7f ff11 81ab 6158 0a36 b.0..3......aX.60x0180: c0b2 e8bc 0b2f baba da67 a3b7 174c c2bd ...../...g...L..0x0190: c6ec 9da0 57be f7a9 1717 48e9 f153 dd01 ....W.....H..S..0x01a0: 50ae cf86 22c4 fc82 43f3 f75e f6d5 cd5f P..."...C..^..._0x01b0: 48bb 9516 c2d3 e316 3adf b057 d115 eb6c H.......:..W...l0x01c0: 3ae1 8f4e 762b f3f2 4799 734e 9596 0471 :..Nv+..G.sN...q0x01d0: 05ef eb38 9ff7 f1ae ce78 0289 d0ad b707 ...8.....x......0x01e0: ba11 6f92 d5bf 865b 124d faef 9110 c6b8 ..o....[.M......0x01f0: c13a ffce c56b 500c 4394 e4dd 7dd5 476f .:...kP.C...}.Go0x0200: 5f09 1e10 0b0e b218 3afd 590b 924d bd9a _.......:.Y..M..0x0210: eeb8 598c c014 a501 46ba d5d8 b4b7 b4b9 ..Y.....F.......0x0220: 9efb 8d42 102a 63f5 d4d7 cb64 d91c bc02 ...B.*c....d....0x0230: f6ea f959 603e 83d6 491c c698 8491 7374 ...Y`>..I.....st0x0240: 75a3 e43a 70b2 2675 ba70 5c4c facf 22c6 u..:p.&u.p\L..".0x0250: 539a 0fd0 7d5a 84a4 a343 76b9 789c 5b7e S...}Z...Cv.x.[~0x0260: 4db4 7aa3 4c70 8dd1 ba2b ebf9 61b0 c878 M.z.Lp...+..a..x0x0270: 20c7 3bf8 0398 d05c 8396 05ed 7f27 9372 ..;....\.....'.r0x0280: c999 c872 763e c98a b73e 328e 9631 fdfe ...rv>...>2..1..0x0290: 57fb a5c6 90c6 9804 c1b6 f738 7bff 2168 W..........8{.!h0x02a0: dac8 1c1d 1858 8415 effe b810 85b0 28f5 .....X........(.0x02b0: 6372 90f7 f37e 2713 8d9f 9ff2 eba4 5ed8 cr...~'.......^.0x02c0: 62ec 25cf 481f 147f 52d2 3c17 c3b8 9669 b.%.H...R.<....i0x02d0: d731 d685 79c5 17d7 256e 8cc9 8e03 a9b0 .1..y...%n......0x02e0: f9ab dda7 022e c5ca 18c6 6667 c573 9808 ..........fg.s..0x02f0: 7600 4c82 9f79 9f2e d37e 9cc4 c151 140f v.L..y...~...Q..0x0300: 279a 9a51 d176 6601 97bc 45fd 0756 5558 '..Q.vf...E..VUX0x0310: 9e76 92a6 9a9f 81e2 3fc7 662a ddc8 d9e4 .v......?.f*....0x0320: 1acd 2ce8 6559 1fba 19f3 c0c8 23fa 43e2 ..,.eY......#.C.0x0330: b21f 5a94 d7a5 b204 6019 a326 632b 7438 ..Z.....`..&c+t80x0340: 227e 1d4e 47f1 3076 8484 ae49 864b d3b7 "~.NG.0v...I.K..0x0350: 6aee 7fa2 f0fc 1c92 6422 d551 afbb a0b3 j.......d".Q....0x0360: 1264 39fc 1452 ddc6 f075 29f0 b356 a885 .d9..R...u)..V..0x0370: 3c33 d148 d60b 6430 fda9 d0a7 81bc 0be3 <3.H..d0........0x0380: 6473 cf05 6362 3554 1fb2 b16b 97b7 a633 ds..cb5T...k...30x0390: 2fdf 9dfb d92c acea 0d8d daf9 403d 169f /....,......@=..0x03a0: 3c64 baf9 ac06 759c ed84 536b 78c7 cc1b <d....u...Skx...0x03b0: 159b 3631 89ef a715 dc6f df62 5172 36e5 ..61.....o.bQr6.0x03c0: 327b 5a7c da6c 2a9e d92d 369e 05d3 f2f3 2{Z|.l*..-6.....0x03d0: 0bec 168e 5fac ddee 3b83 ...._...;.
Special Consideration:
I am using an encryption key for all traffic between agents. Even after disabling the encryption key on the client agent, the packet contents are still nonsensical.
I am using an encryption key for all traffic between agents. Even after disabling the encryption key on the client agent, the packet contents are still nonsensical.
Thanks for reading this entire post,
Thanks, Marek
David Karlsen
Feb 27, 2018, 12:13:29 PM2/27/18
to Consul
I am wondering about the same - I think the docs should be clearer in this area: https://github.com/hashicorp/consul/issues/3921
Kyle Havlovitz
Feb 27, 2018, 1:09:46 PM2/27/18
to consu...@googlegroups.com
Have you looked into Consul's network segments at all? It's an enterprise feature to do pretty much exactly what you're describing here: https://www.consul.io/docs/enterprise/network-segments/index.html. I don't think there's any unique header in the memberlist traffic (the library serf uses for gossip) that would work here, and the packets are typically compressed where possible which would make the rest of the message unusable for identifying patterns that way.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/6bc4bbfa-464d-4f62-9841-f141e3e44834%40googlegroups.com.--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
David Karlsen
Feb 27, 2018, 2:44:19 PM2/27/18
to consu...@googlegroups.com
That explains a bit more - but I still think the docs are not crystal clear in this area.
Also the segmenting seems to be a commercial feature - I had hoped to get away with the open-source version.
Also the segmenting seems to be a commercial feature - I had hoped to get away with the open-source version.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CA%2Bo85FaEGMiF%2B3V8Kv%2B-2hi4kyBS921PvcSGOwKDx22ceY4c6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Marek Slabicki
Feb 27, 2018, 5:56:15 PM2/27/18
to Consul
Consul requiring a full mesh doesn't bother me, so long as I can restrict the traffic to Gossip only. My primary concern is to make sense of the packets.
Thanks Kyle for the insight on how Consul is using Serf. I presume, then, that the packets are essentially useless in order to try and identify a signature.
When I have more time I'll start digging into source code to see exactly how much of a difference compression is giving in terms of performance.
Thanks Kyle for the insight on how Consul is using Serf. I presume, then, that the packets are essentially useless in order to try and identify a signature.
When I have more time I'll start digging into source code to see exactly how much of a difference compression is giving in terms of performance.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/6bc4bbfa-464d-4f62-9841-f141e3e44834%40googlegroups.com.
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CA%2Bo85FaEGMiF%2B3V8Kv%2B-2hi4kyBS921PvcSGOwKDx22ceY4c6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages