Strange carp issues
Steven S
external interface (carp1, carp12, carp14, carp15, carp16, carp17, carp18,
carp19, carp20). FW1 has all carp interfaces set with advbase 1 advskew 0
and FW2 has all carp interfaces with advbase 1 advskew 180. Frequently FW2
thinks it is the master for some of the carp interfaces. Here is a tcpdump
(-ni fxp0 proto carp) from FW2. As you can see, even though FW2 sees the
advertisement for carp16, carp17, carp18, carp19 and carp20 from FW1 it
sometimes takes over as master for those interfaces and advertises. To find
these events look for advskew=180 in the tcpdump below.
The event at 19:19:05.023848 seemed to be from lost packets. The event at
19:19:10.013844 is very odd since FW2 saw the carp20 advertisement from FW1
at 19:19:09.074444. This should be enough time for a failover, should it?
Any pointers would be appreciated (relevant pf rules below.)
-Steve S.
19:19:02.290779 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290807 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290828 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290849 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290869 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290887 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290914 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290936 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.290957 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890823 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890849 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890871 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890892 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890912 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890933 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890962 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.890986 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:02.891010 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880791 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880818 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880839 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880860 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880881 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880901 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880932 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880955 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:03.880979 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.023848 CARPv2-advertise 36: vhid=17 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:05.024936 CARPv2-advertise 36: vhid=18 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:05.026003 CARPv2-advertise 36: vhid=19 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:05.027069 CARPv2-advertise 36: vhid=20 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:05.341023 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341047 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341068 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341088 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341109 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341129 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341154 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341176 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:05.341199 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.295736 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.295760 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.295782 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.295802 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.295822 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.297299 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.297318 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.297335 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.297352 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.900831 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.900876 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.900925 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.900973 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.901021 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.901069 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.901112 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.901158 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:06.901205 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:07.990804 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.000806 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010812 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010832 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010853 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010874 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010894 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010913 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:08.010940 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.060841 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.070814 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074312 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074332 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074353 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074373 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074394 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074414 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:09.074444 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.013844 CARPv2-advertise 36: vhid=20 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:10.103204 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103228 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103249 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103268 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103286 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103305 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103331 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103353 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.103374 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:10.613850 CARPv2-advertise 36: vhid=19 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:11.000985 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001009 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001030 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001050 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001069 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001087 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001116 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001138 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.001160 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:11.733838 CARPv2-advertise 36: vhid=20 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:12.040852 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040877 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040898 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040920 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040940 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040960 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.040987 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.041011 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.041035 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:12.333855 CARPv2-advertise 36: vhid=19 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:12.793851 CARPv2-advertise 36: vhid=17 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:12.794962 CARPv2-advertise 36: vhid=18 advbase=1 advskew=180 (DF) [tos
0x10]
19:19:13.110832 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110857 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110878 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110900 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110920 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110941 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110967 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.110991 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.111014 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos
0x10]
19:19:13.453851 CARPv2-advertise 36: vhid=20 advbase=1 advskew=180 (DF) [tos
0x10]
OpenBSD 3.8-stable (GENERIC.MP) #0: Thu Jan 5 03:55:53 EST 2006
root@fw2# pfctl -sr |grep -i carp
pass quick on fxp0 proto carp all keep state
pass quick on fxp1 proto carp all keep state
pass quick on fxp2 proto carp all keep state
pass quick on fxp3 proto carp all keep state
pass quick on tl0 proto carp all keep state
pass quick on tl1 proto carp all keep state
Bryan Irvine
Bryan Irvine
There's some threads in the archives, but they probably won't help
since there is apparently no solution.
--Bryan
On 3/15/06, Steven S <ssur...@engineered-net.com> wrote:
> Bryan Irvine wrote:
> > I don't suppose you are using a quad card of some kind are you?
> >
> >
> ...
> Three dual cards, dmesg (extracted from /var/log/messages) below:
>
> OpenBSD 3.8-stable (GENERIC.MP) #0: Thu Jan 5 03:55:53 EST 2006
> ro...@fw1.glcomp.com:/usr/src/sys/arch/i386/compile/GENERIC.MP
> cpu0: Intel Pentium III ("GenuineIntel" 686-class) 798 MHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,
> FXSR,SSE
> real mem = 536436736 (523864K)
> avail mem = 482525184 (471216K)
> using 4278 buffers containing 26923008 bytes (26292K) of memory
> mainbus0 (root)
> bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000
> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
> pcibios0: PCI BIOS has 6 Interrupt Routing table entries
> pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks ROSB4 SouthBridge"
> rev 0x00)
> pcibios0: PCI bus #1 is the last bus
> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4000! 0xe8000/0x6000
> 0xee000/0x2000!
> mainbus0: Intel MP Specification (Version 1.4) (COMPAQ PROLIANT )
> cpu0 at mainbus0: apid 3 (boot processor)
> cpu0: apic clock running at 132 MHz
> cpu1 at mainbus0: apid 0 (application processor)
> cpu1: Intel Pentium III ("GenuineIntel" 686-class) 797 MHz
> cpu1:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,
> FXSR,SSE
> mainbus0: bus 0 is type PCI
> mainbus0: bus 3 is type PCI
> mainbus0: bus 9 is type ISA
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 35 pins
> ioapic0: misconfigured as apic 0, remapped to apic 8
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20LE Host" rev 0x05
> pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20LE Host" rev 0x05
> pci1 at pchb1 bus 3
> fxp0 at pci1 dev 4 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 10
> (irq 10), address 00:50:8b:e2:6e:fb
> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
> fxp1 at pci1 dev 5 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 11
> (irq 11), address 00:50:8b:e2:6e:fa
> inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
> ppb0 at pci1 dev 6 function 0 "DEC 21154 PCI-PCI" rev 0x05
> pci2 at ppb0 bus 4
> fxp2 at pci2 dev 4 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 11
> (irq 11), address 00:02:a5:60:58:50
> inphy2 at fxp2 phy 1: i82555 10/100 PHY, rev. 4
> fxp3 at pci2 dev 5 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 10
> (irq 10), address 00:02:a5:60:58:51
> inphy3 at fxp3 phy 1: i82555 10/100 PHY, rev. 4
> cac0 at pci0 dev 1 function 0 "Symbios Logic 53c1510" rev 0x02: apic 8 int 3
> (irq 3) Compaq Integrated Array
> scsibus0 at cac0: 1 targets
> sd0 at scsibus0 targ 0 lun 0: <Compaq, RAID1 volume #, > SCSI2 0/direct
> fixed
> sd0: 17359MB, 4357 cyl, 255 head, 32 sec, 512 bytes/sec, 35553120 sec total
> vga1 at pci0 dev 3 function 0 "ATI Mach64 GV" rev 0x7a
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> "Compaq Netelligent ASMC" rev 0x00 at pci0 dev 4 function 0 not configured
> ppb1 at pci0 dev 5 function 0 "IBM 82351 PCI-PCI" rev 0x01
> pci3 at ppb1 bus 1
> tl0 at pci3 dev 0 function 0 "Compaq DP Netelligent 10/100TX" rev 0x10: apic
> 8 int 5 (irq 5) address 00:08:c7:a4:84:6d
> nsphy0 at tl0 phy 1: DP83840 10/100 PHY, rev. 1
> ukphy0 at tl0 phy 31: Generic IEEE 802.3u media interface
> ukphy0: OUI 0x100014, model 0x0001, rev. 5
> tl1 at pci3 dev 1 function 0 "Compaq DP Netelligent 10/100TX" rev 0x10: apic
> 8 int 7 (irq 7) address 00:08:c7:a4:84:ed
> nsphy1 at tl1 phy 1: DP83840 10/100 PHY, rev. 1
> ukphy1 at tl1 phy 31: Generic IEEE 802.3u media interface
> ukphy1: OUI 0x100014, model 0x0001, rev. 5
> pcib0 at pci0 dev 15 function 0 "ServerWorks ROSB4 SouthBridge" rev 0x4f
> pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <COMPAQ, CD-ROM CRN-8241B, 2.23> SCSI0 5/cdrom
> removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> isa0 at pcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pmsi0 at pckbc0 (aux slot)
> pckbc0: using irq 12 for aux slot
> wsmouse0 at pmsi0 mux 0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> sysbeep0 at pcppi0
> npx0 at isa0 port 0xf0/16: using exception 16
> pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> biomask 0 netmask 0 ttymask 0
> pctr: 686-class user-level performance counters enabled
> mtrr: Pentium Pro MTRR support
> dkcsum: sd0 matches BIOS drive 0x80
> root on sd0a
> rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
Steven S
Steven S
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello.
>
> I have the same problem.
>
...
>
> Take a look in your date/time, maybe it's the reason of your strange
> carp issues.
...
I thought of that too. If time changed by a couple seconds on the backup
server then the backup might think it hadn't heard from FW1 in the carp
time-out, so I stopped ntpd on both servers. I still experienced the
problem. Oddly, it's not all carp interfaces on my fxp0. It only seems to
affect carp16 - carp20, but inconsistently so.
I'm going to try an experiment with a couple lab boxes, multiple carp
interfaces, and ifstate (for monitoring). The plan is to see if it is
related to the number of carp interfaces. Unless someone has tried this
already (hint, hint;-)
-Steve S.
Anderson Nadal
Hash: SHA1
Hello.
I have the same problem.
I have 2 Fw, the Master is a Dell 2850 (2 processors) and the slave is
a Dell 2850, with 1 processor, both with 3 dual cards.
The NTPD daemon doesn't work in the Master. I mean, the date/time is
always wrong (is it a bug in OpenBSD 3.7 with SMP ??).
When the difference between Master and Slave's date/time becomes too
large, the carp goes down!!
The Master becomes Slave, and the Slave becomes Master.
Using NTPDATE in cron (30 minutes), I was able to handle this weird
behavior.
Take a look in your date/time, maybe it's the reason of your strange
carp issues.
[]'s
Nadal
Bryan Irvine wrote:
> Thought so. Had the same problem. Never got them working with
> CARP.
>
> There's some threads in the archives, but they probably won't help
> since there is apparently no solution.
>
> --Bryan
>
> On 3/15/06, Steven S <ssur...@engineered-net.com> wrote:
>
- --
+-------------------------------------------------------+
| Anderson Nadal <na...@ondacorp.com.br> - RHCE |
| Coordenador Tecnico |
| Fone: + 55 41 3331 8200 |
| FAX: + 55 41 3331 8256 |
| OndaRPC |
| www.ondarpc.com.br |
| Registered Linux User: 56841 |
| PGP KEY: www.keyserver.net KEY ID 6ABB668D |
| M.O.V.I |
+-------------------------------------------------------+
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFEGrriLQAusHT90XQRApfOAJ9C7HzU3Fjx7saI6dSx3BRkw7+gnACdHk5F
mtkqIwAhxRcNyFwc8M8UCFQ=
=78r6
-----END PGP SIGNATURE-----
Bryan Irvine
> Anderson Nadal wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hello.
> >
> > I have the same problem.
> >
> >
> > Take a look in your date/time, maybe it's the reason of your strange
> > carp issues.
>
> I thought of that too. If time changed by a couple seconds on the backup
> server then the backup might think it hadn't heard from FW1 in the carp
> time-out, so I stopped ntpd on both servers. I still experienced the
> problem. Oddly, it's not all carp interfaces on my fxp0. It only seems to
> affect carp16 - carp20, but inconsistently so.
>
> I'm going to try an experiment with a couple lab boxes, multiple carp
> interfaces, and ifstate (for monitoring). The plan is to see if it is
> related to the number of carp interfaces. Unless someone has tried this
> already (hint, hint;-)
I tried before with 2 quad cards to no avail. That was under 3.6
though IIRC. 1 or 2 if's would fail over within a couple of hours,
but if left to it's own devices, eventually they all would.
If you do figure something out lemme know, I'd love to go back to the
quad cards.
ifstated didn't work for me but give it a go. I also had a script on
each machine that would ping the other every 5 seconds for ever....
The interfaces seemed to last longer but eventually failed that way
too.
--Bryan
Steven S
> I tried before with 2 quad cards to no avail. That was under 3.6
> though IIRC. 1 or 2 if's would fail over within a couple of hours,
> but if left to it's own devices, eventually they all would.
>
> If you do figure something out lemme know, I'd love to go back to the
> quad cards.
>
> ifstated didn't work for me but give it a go. I also had a script on
> each machine that would ping the other every 5 seconds for ever....
> The interfaces seemed to last longer but eventually failed that way
> too.
>
> --Bryan
The interfaces that I'm having the most problem with are the built-in
interfaces on a Compaq DL360 (I misstated earlier that it was a dual
interface nic) although I have two other dual port nics in the machine. I
wonder if the built-in looks like a dual port nic?
I find it odd that the problem might be related to the multi-port nic. Did
you try the same configuration with single port nics and it worked? I'm
beginning to think it might be a component of the number of carp interfaces
(you would likely have more carp interfaces on a machine with multiport
nics.)
Ifstated was broken for me on 3.8-stable too. I notice some changes in the
3.9 version so I compiled the 3.9 source under 3.8 and ifstated works *much*
better. Perhaps it should be posted for 3.8 as a "reliability patch."
-Steve S.
Henning Brauer
> beginning to think it might be a component of the number of carp interfaces
unlikely.
<brahe@cr20> $ ifconfig | grep '^carp' | wc -l
15
and growing.
and yes, that is real-world production use.
--
BS Web Services, http://www.bsws.de/
OpenBSD-based Webhosting, Mail Services, Managed Servers, ...
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)
Bryan Irvine
> > I tried before with 2 quad cards to no avail. That was under 3.6
> > though IIRC. 1 or 2 if's would fail over within a couple of hours,
> > but if left to it's own devices, eventually they all would.
> >
> > If you do figure something out lemme know, I'd love to go back to the
> > quad cards.
> >
> > ifstated didn't work for me but give it a go. I also had a script on
> > each machine that would ping the other every 5 seconds for ever....
> > The interfaces seemed to last longer but eventually failed that way
> > too.
> >
> > --Bryan
>
> The interfaces that I'm having the most problem with are the built-in
> interfaces on a Compaq DL360 (I misstated earlier that it was a dual
> interface nic) although I have two other dual port nics in the machine. I
> wonder if the built-in looks like a dual port nic?
I never had a machine with built-in NICs *and* multi-port cards.
> I find it odd that the problem might be related to the multi-port nic. Did
> you try the same configuration with single port nics and it worked? I'm
> beginning to think it might be a component of the number of carp interfaces
> (you would likely have more carp interfaces on a machine with multiport
> nics.)
Yup! The quad cards were all intel-based, replaced the cards with
other intel cards, and everything worked. Same pf.conf, same
hostname.if, everything.
--Bryan
Steven S
>> interfaces
>
> <brahe@cr20> $ ifconfig | grep '^carp' | wc -l
> 15
> and growing.
> and yes, that is real-world production use.
How do you monitor if a carp interface changes state? And are these on any
multi-port NICs?
Thanks!
-Steve S.
Adam D. Morley
> * Steven S <ssur...@engineered-net.com> [2006-03-17 19:10]:
> > beginning to think it might be a component of the number of carp interfaces
>
> unlikely.
> <brahe@cr20> $ ifconfig | grep '^carp' | wc -l
> 15
> and growing.
> and yes, that is real-world production use.
I would agree that number of carp interfaces doesn't matter:
# ifconfig |grep ^carp |wc -l
23
This is real-world also, with many of the carp interfaces layered on top
of VLANs. Intel dual GE cards.
Have you checked:
- carp settings in sysctl?
- carp pass rules (and ordering) in pf.conf (if you have default deny)?
- that you have advskew set "right" on the backup firewall?
# grep carp /etc/sysctl.conf
net.inet.carp.allow=1 # allow incoming CARP packets
net.inet.carp.preempt=1 # failover all CARP interfaces if one fails
# grep carp /etc/pf.conf
pass quick on $ext_ints proto carp keep state
pass on $int_phys proto carp keep state
pass on $int_vlan proto carp keep state
# cat /etc/hostname.carp1
vhid 1 advskew 100 pass XXXX
inet XXX 0xffffff00
--
adam
Steven S
...
> Have you checked:
>
> - carp settings in sysctl?
> - carp pass rules (and ordering) in pf.conf (if you have default
> deny)?
> - that you have advskew set "right" on the backup firewall?
>
> # grep carp /etc/sysctl.conf
> net.inet.carp.allow=1 # allow incoming CARP packets
> net.inet.carp.preempt=1 # failover all CARP
> interfaces if one fails
>
> # grep carp /etc/pf.conf
> pass quick on $ext_ints proto carp keep state
> pass on $int_phys proto carp keep state
> pass on $int_vlan proto carp keep state
>
> # cat /etc/hostname.carp1
> vhid 1 advskew 100 pass XXXX
> inet XXX 0xffffff00
Thanks, this is helpful. The settings on the FW's are as above. An
incorrect setting (above) would seem to make it not work -- as opposed to
what I'm seeing. Sometimes FW2 takes over as MASTER for some interfaces,
but FW1 never moves to BACKUP. I do have net.inet.carp.preempt=1 set on
FW1, but not FW2.
As another experiment I moved advbase on FW2 to '2' for all carps, but the
mysterious BACKUP-->MASTER transition still occurred on FW2 (in thinking
back, I did this with a reboot of FW2, which re-started ntpd.) Perhaps I'll
try again and not starting ntpd.
-Steve S.
Henning Brauer
> Henning Brauer wrote:
> > * Steven S <ssur...@engineered-net.com> [2006-03-17 19:10]:
> >> beginning to think it might be a component of the number of carp
> >> interfaces
> >
> > unlikely.
> > <brahe@cr20> $ ifconfig | grep '^carp' | wc -l
> > 15
> > and growing.
> > and yes, that is real-world production use.
I don't. carp works.
We do monitor the hosts in the cluster individually.
> And are these on any multi-port NICs?
in this case, no, but I have carps on multiport. which is entirely
irrelevant.
Steven S
> On Fri, Mar 17, 2006 at 02:35:55PM -0500, Steven S wrote:
>> Adam D. Morley wrote:
...
>> incorrect setting (above) would seem to make it not work -- as
>> opposed to
>
>
>> what I'm seeing. Sometimes FW2 takes over as MASTER for some
>> interfaces, but FW1 never moves to BACKUP. I do have
>> net.inet.carp.preempt=1 set on FW1, but not FW2.
>
With both firewalls set to preempt=1 I had a common DMZ switch get shut-off.
Both FW's went to a carp skew of 240. They had a MASTER fight. By setting
one with preempt=1 and the other with preempt=0, I avoid this.
>> As another experiment I moved advbase on FW2 to '2' for all carps,
>> but the
>
> base is how often. skew is priority.
Sort of... 'man ifconfig' Says,
"Taken together the advbase and advskew indicate how frequently, in seconds,
the host will advertise the fact that it considers itself master of the
virtual host. The formula is advbase + (advskew / 256). If the master does
not advertise within three times this interval, this host will begin
advertising as master."
So if I set FW1 with 1/0 and FW2 at 2/180, FW1 advertises every one second.
If FW2 hasn't heard a carp advertisement in 2.7*3=8.1 seconds it will take
over. When FW1 returns, it will start advertising once/sec. As noted in my
OP, this doesn't seem to happen on my FW pair.
-Steve S.
Jon Simola
> > As another experiment I moved advbase on FW2 to '2' for all carps, but the
>
> base is how often. skew is priority.
No, advbase is integer seconds between advertisements, advskew is
fractional seconds. Taken together, advbase and advskew are an 8.8 bit
fixed point number allowing you to specify advertisment intervals
between 4ms and 255.996s (in theory anyways, setting advskew to 240 or
above is used with preempting as a magic number).
Around line 610 of ip_carp.c:
ch_tv.tv_sec = ch->carp_advbase;
ch_tv.tv_usec = ch->carp_advskew * 1000000 / 256;
--
Jon Simola
Systems Administrator
ABC Communications
Adam D. Morley
> Adam D. Morley wrote:
> ...
> >
> > - carp settings in sysctl?
> > - carp pass rules (and ordering) in pf.conf (if you have default
> > deny)?
> > - that you have advskew set "right" on the backup firewall?
> >
> > # grep carp /etc/sysctl.conf
> > net.inet.carp.allow=1 # allow incoming CARP packets
> > net.inet.carp.preempt=1 # failover all CARP
> > interfaces if one fails
> >
> > # grep carp /etc/pf.conf
> > pass quick on $ext_ints proto carp keep state
> > pass on $int_phys proto carp keep state
> > pass on $int_vlan proto carp keep state
> >
> > # cat /etc/hostname.carp1
> > vhid 1 advskew 100 pass XXXX
> > inet XXX 0xffffff00
>
> incorrect setting (above) would seem to make it not work -- as opposed to
Ok. But mine works and yours doesn't?
> what I'm seeing. Sometimes FW2 takes over as MASTER for some interfaces,
> but FW1 never moves to BACKUP. I do have net.inet.carp.preempt=1 set on
> FW1, but not FW2.
You're supposed to set preempt on both, iirc.
>
> As another experiment I moved advbase on FW2 to '2' for all carps, but the
base is how often. skew is priority.
--
adam
Joachim Schipper
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello.
>
> I have the same problem.
>
> a Dell 2850, with 1 processor, both with 3 dual cards.
>
> The NTPD daemon doesn't work in the Master. I mean, the date/time is
> always wrong (is it a bug in OpenBSD 3.7 with SMP ??).
>
> When the difference between Master and Slave's date/time becomes too
> large, the carp goes down!!
> The Master becomes Slave, and the Slave becomes Master.
>
> Using NTPDATE in cron (30 minutes), I was able to handle this weird
> behavior.
>
> carp issues.
As to problems with adjtime(2) and SMP machines, there is a small diff
from tedu@ on tech@ at
http://marc.theaimsgroup.com/?l=openbsd-tech&m=113592306900483&w=2,
which stemmed from the discussion on misc@ around the same time,
involving another SMP machine with severely screwed timekeeping - in
fact, it was so bad that NTPd couldn't keep up. Ted's diff allows NTP to
keep up with time slew even on very imprecise hosts.
It's a workaround, but might work for you.
Joachim
Adam D. Morley
> On 3/17/06, Adam D. Morley <adam-ope...@gmi.com> wrote:
>
> >
> > base is how often. skew is priority.
>
> fractional seconds. Taken together, advbase and advskew are an 8.8 bit
> fixed point number allowing you to specify advertisment intervals
> between 4ms and 255.996s (in theory anyways, setting advskew to 240 or
> above is used with preempting as a magic number).
Of course. However, for many, the combination of both advbase and
advskew is confusing, and a simplication is "easier" for them to grasp.
Especially when trying to explain pre-empting. Ie: the average user won't
need advbase, and so explaining advskew as "priority" (when in fact is
is not) makes it "easier" to understand. For example, when a user is
following:
http://www.countersiege.com/doc/pfsync-carp/
explaining advskew as a "priority" often makes people grasp the concept
faster, and then once it "works," they can go, "hmm...oh yes, the man
page!"
--
adam
Adam D. Morley
> Adam D. Morley wrote:
> > On Fri, Mar 17, 2006 at 02:35:55PM -0500, Steven S wrote:
> >> Adam D. Morley wrote:
> ...
> >> incorrect setting (above) would seem to make it not work -- as
> >> opposed to
> >
> > Ok. But mine works and yours doesn't?
> >
> >> what I'm seeing. Sometimes FW2 takes over as MASTER for some
> >> interfaces, but FW1 never moves to BACKUP. I do have
> >> net.inet.carp.preempt=1 set on FW1, but not FW2.
> >
> > You're supposed to set preempt on both, iirc.
>
> Both FW's went to a carp skew of 240. They had a MASTER fight. By setting
> one with preempt=1 and the other with preempt=0, I avoid this.
This is likely because one of the firewalls does not have a pass rule for
carp packets. I have seen this happen before, especially when adding a
new carp interface and not updating ext_ints.
>
> >> As another experiment I moved advbase on FW2 to '2' for all carps,
> >> but the
> >
> > base is how often. skew is priority.
>
> Sort of... 'man ifconfig' Says,
Sort of, yes, but for the purposes of pre-empted pf-walls, it's a very
convinent simplification.
>
> "Taken together the advbase and advskew indicate how frequently, in seconds,
> the host will advertise the fact that it considers itself master of the
> virtual host. The formula is advbase + (advskew / 256). If the master does
> not advertise within three times this interval, this host will begin
> advertising as master."
Yes, I have read man ifconfig. ;-)
>
> So if I set FW1 with 1/0 and FW2 at 2/180, FW1 advertises every one second.
> If FW2 hasn't heard a carp advertisement in 2.7*3=8.1 seconds it will take
> over. When FW1 returns, it will start advertising once/sec. As noted in my
> OP, this doesn't seem to happen on my FW pair.
Well, FW1 expects FW2 to advertise as master within 3 seconds, and it's
advertising every 2.7 seconds. This is pretty close, though it should be
more than enough time. It would be "irrelevant" if one had default master
status (preempt on on both), but that's not the case in your setup.
Out of curiosity, why are you twiddling advbase? Do you have some
high-latency serial lines in there or something? Are you attempting to
get longer detection intervals in case fw1 is not actually down, but fw2
thinks it is? If it's the later, then that's not what it's for.
Personally, I would keep advbase the same on both hosts and twiddle advskew,
set preempt (on both), and see what happens. If something goes wrong, then
it's likely a missing pass rule for carp packets.
--
adam
Steven S
>> Using NTPDATE in cron (30 minutes), I was able to handle this weird
>> behavior.
>>
>> Take a look in your date/time, maybe it's the reason of your strange
>> carp issues.
>
> As to problems with adjtime(2) and SMP machines, there is a small
> diff from tedu@ on tech@ at
> http://marc.theaimsgroup.com/?l=openbsd-tech&m=113592306900483&w=2,
> which stemmed from the discussion on misc@ around the same time,
> involving another SMP machine with severely screwed timekeeping - in
> fact, it was so bad that NTPd couldn't keep up. Ted's diff
> allows NTP to
> keep up with time slew even on very imprecise hosts.
>
> It's a workaround, but might work for you.
I tried the patch, but it didn't apply cleanly against 3.8:-(
I tried booting FW2 with the SP kernel, but the problem still persists. It
doesn't appear to be ntpd related since ntp updates didn't correlate with
carp BACKUP -> MASTER transitions. I'll keep plugging away at it...
-Steve S.
Joachim Schipper
Nonetheless, it does lead to the question if timekeeping, especially
without ntpd, is accurate. You seem to believe this is not the case;
fixing this might well fix the carp problems.
Joachim
Steven S
DL360 G1).
If I bump advbase to '3' on each box everything is more stable. Given this,
I now have a roughly 10 second fail-over time, but that is still acceptable.
Since these are production boxes I'll probably wait until my 3.9 arrives to
see if any of the kern_time/kern_clock changes help. I'll let everyone know
more when I do.
Thanks for all the pointers and assistance!
Steve's corollary to Henning's carp theorem ("carp works."): Unless the
system clock is broken:-)
-Steve S.