Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Intel 82574L Gigabit Ethernet Controller

52 views
Skip to first unread message

Shtorm

unread,
Jul 5, 2010, 7:43:54 AM7/5/10
to
I just got Supermicro SuperServer 5016I-MRF server with two 82574L
controllers on board. According to specs this controller can do two
msi-x vectors for tx and rx, but only one tx-rx pair is used on
8.1-prerelease.

uname -a
FreeBSD router 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0: Tue Jun 29
00:42:32 UTC 2010 root@releng_8:/usr/obj/usr/src/sys/ROUTER amd64

Here is pciconf

em0@pci0:1:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class = network
subclass = ethernet
em1@pci0:2:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class = network
subclass = ethernet

Checking procstat shows 5 interrupts per interface

procstat -at | grep em\[01\]
0 100035 kernel em0 rxq 0 16 sleep -
0 100037 kernel em0 txq 6 16 sleep -
0 100039 kernel em0 rxq 0 16 sleep -
0 100041 kernel em0 txq 0 16 sleep -
0 100044 kernel em1 rxq 0 16 sleep -
0 100046 kernel em1 txq 1 16 sleep -
0 100048 kernel em1 rxq 0 16 sleep -
0 100050 kernel em1 txq 0 16 sleep -
11 100034 intr irq256: em0 7 16 wait -
11 100036 intr irq257: em0 0 16 wait -
11 100038 intr irq258: em0 0 16 wait -
11 100040 intr irq259: em0 0 16 wait -
11 100042 intr irq260: em0 3 16 wait -
11 100043 intr irq261: em1 4 16 wait -
11 100045 intr irq262: em1 5 16 wait -
11 100047 intr irq263: em1 0 16 wait -
11 100049 intr irq264: em1 0 16 wait -
11 100051 intr irq265: em1 2 16 wait -


but vmstat -i shows only one pair is used

%vmstat -i
interrupt total rate
irq1: atkbd0 391 0
irq18: ehci0 uhci5 2 0
irq19: uhci2 uhci4 27 0
irq23: uhci3 ehci1 8463 0
cpu0: timer 107412943 7864
irq256: em0 93311951 6832
irq257: em0 90479067 6624
irq260: em0 2 0
irq261: em1 92966894 6806
irq262: em1 80298240 5879
irq265: em1 1 0
cpu1: timer 107412607 7864
cpu2: timer 107412785 7864
cpu3: timer 107412830 7864
cpu5: timer 107413210 7864
cpu4: timer 107413389 7864
cpu7: timer 107407337 7864
cpu6: timer 107413132 7864
Total 1216363271 89058


Here is my loader.conf

%cat /boot/loader.conf | grep em
if_em_load=YES
hw.em.rxd=4096
hw.em.txd=4096
hw.em.tx_int_delay=512
hw.em.rx_int_delay=512
hw.em.tx_abs_int_delay=1024
hw.em.rx_abs_int_delay=1024
hw.em.enable_msix=1
hw.em.msix_queues=2
hw.em.rx_process_limit=100
hw.em.fc_setting=0


Is it ever possible to activate second tx-rx pair on this card?

_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

Jack Vogel

unread,
Jul 5, 2010, 2:11:01 PM7/5/10
to
Are you defining 'EM_MULTIQUEUE', its off by default and needs to be
defined somewhere by you.

You will only see the two queues used if you have two different connections
operating at once.

Jack

Yuriy A. Korobko

unread,
Jul 5, 2010, 3:06:36 PM7/5/10
to
On Mon, 2010-07-05 at 11:11 -0700, Jack Vogel wrote:
> Are you defining 'EM_MULTIQUEUE', its off by default and needs to be
> defined somewhere by you.
>
> You will only see the two queues used if you have two different connections
> operating at once.
>
> Jack
>
>
> On Mon, Jul 5, 2010 at 4:43 AM, Shtorm <ad...@shtorm.com> wrote:
>
> > I just got Supermicro SuperServer 5016I-MRF server with two 82574L
> > controllers on board. According to specs this controller can do two
> > msi-x vectors for tx and rx, but only one tx-rx pair is used on
> > 8.1-prerelease.
> >

Thanks for you reply Jack,

This box working as a router with up to 60 kpps traffic rate , I believe
there is enough flows to see both queues running.

# netstat -I em0 -w 1
input (em0) output
packets errs idrops bytes packets errs bytes colls
60356 0 0 61659974 46803 0 15527523 0
56273 0 0 56429189 44278 0 15199658 0
55405 0 0 55408969 43654 0 14978427 0

Anyway, will try to rebuild em module with

CFLAGS += -DEM_MULTIQUEUE

and with LEGACY stuff commented out in src/sys/modules/em/Makefile tomorrow.

Is this enough or should I put define in the beginning of src/sys/dev/e1000/if_em.c ?

Jack Vogel

unread,
Jul 6, 2010, 1:02:53 AM7/6/10
to
Cleaner in the Makefile, let me know how it goes.

Jack


On Mon, Jul 5, 2010 at 12:06 PM, Yuriy A. Korobko
<admini...@shtorm.com>wrote:

Shtorm

unread,
Jul 6, 2010, 4:28:53 AM7/6/10
to
On Mon, 2010-07-05 at 22:02 -0700, Jack Vogel wrote:
> Cleaner in the Makefile, let me know how it goes.
>
> Jack
>
>
> On Mon, Jul 5, 2010 at 12:06 PM, Yuriy A. Korobko
> <admini...@shtorm.com>wrote:
>
> > On Mon, 2010-07-05 at 11:11 -0700, Jack Vogel wrote:
> > > Are you defining 'EM_MULTIQUEUE', its off by default and needs to be
> > > defined somewhere by you.
> > >
> > > You will only see the two queues used if you have two different
> > connections
> > > operating at once.
> > >
> > > Jack

Recompiled module (actually whole kernel) with EM_MULTIQUEUE, but got a
some watchdog timeouts.

Here is example of netstat when it happens:

2346 0 0 583370 2642 0 2988379 0
2250 0 0 550961 2384 0 2835276 0
2634 0 0 703410 2733 0 2971417 0
2741 0 0 695884 2748 0 3061573 0
2811 0 0 618520 2274 0 3273868 0
3220 0 0 687145 2068 0 3417857 0
901 0 0 231694 166 0 1565839 0
752 0 0 216135 0 0 270247 0
685 0 0 208745 0 0 243442 0
713 0 0 178089 0 0 230472 0
735 0 0 159555 0 0 178435 0
616 0 0 145222 0 0 179022 0
input (em1) output


packets errs idrops bytes packets errs bytes colls

616 0 0 129608 0 0 120929 0
659 0 0 113806 0 0 105707 0
622 0 0 106247 0 0 107825 0
645 0 0 101593 0 0 38023 0
483 0 0 61681 0 0 8547 0

Watchdog timeout on em1 message on console

0 0 0 0 0 1 19095 0
0 0 0 0 0 0 222 0
0 0 0 0 0 0 0 0
243 0 0 30224 1783 0 1708875 0
576 0 0 62350 105 0 7710 0
467 0 0 49758 4 0 4388 0
500 0 0 53415 23 0 24743 0
437 0 0 50135 21 0 16733 0

After this traffic increased again up to 3 kpps and another watchdog
timeout happened.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:30:48:bc:ab:ca
inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.y
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:30:48:bc:ab:cb
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

em1 have 30 vlans and mpd5 as pppoe server listening on each vlan.

Here is sysctl dev.em

dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.0.5
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.0.%parent: pci1
dev.em.0.nvm: -1
dev.em.0.rx_int_delay: 200
dev.em.0.tx_int_delay: 200
dev.em.0.rx_abs_int_delay: 500
dev.em.0.tx_abs_int_delay: 500
dev.em.0.rx_processing_limit: 100
dev.em.0.link_irq: 2
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.fc_high_water: 18432
dev.em.0.fc_low_water: 16932
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.rx_overruns: 0
dev.em.0.mac_stats.watchdog_timeouts: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd: 732916
dev.em.0.mac_stats.good_pkts_recvd: 732916
dev.em.0.mac_stats.bcast_pkts_recvd: 1238
dev.em.0.mac_stats.mcast_pkts_recvd: 0
dev.em.0.mac_stats.rx_frames_64: 84622
dev.em.0.mac_stats.rx_frames_65_127: 128831
dev.em.0.mac_stats.rx_frames_128_255: 34037
dev.em.0.mac_stats.rx_frames_256_511: 30206
dev.em.0.mac_stats.rx_frames_512_1023: 24919
dev.em.0.mac_stats.rx_frames_1024_1522: 430301
dev.em.0.mac_stats.good_octets_recvd: 0
dev.em.0.mac_stats.good_octest_txd: 0
dev.em.0.mac_stats.total_pkts_txd: 678078
dev.em.0.mac_stats.good_pkts_txd: 678078
dev.em.0.mac_stats.bcast_pkts_txd: 109
dev.em.0.mac_stats.mcast_pkts_txd: 0
dev.em.0.mac_stats.tx_frames_64: 324803
dev.em.0.mac_stats.tx_frames_65_127: 196866
dev.em.0.mac_stats.tx_frames_128_255: 47362
dev.em.0.mac_stats.tx_frames_256_511: 31917
dev.em.0.mac_stats.tx_frames_512_1023: 44997
dev.em.0.mac_stats.tx_frames_1024_1522: 32133
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 0
dev.em.0.interrupts.rx_pkt_timer: 0
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.tx_abs_timer: 0
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 0
dev.em.0.host.breaker_tx_pkt: 0
dev.em.0.host.host_tx_pkt_discard: 0
dev.em.0.host.rx_pkt: 0
dev.em.0.host.breaker_rx_pkts: 0
dev.em.0.host.breaker_rx_pkt_drop: 0
dev.em.0.host.tx_good_pkt: 0
dev.em.0.host.breaker_tx_pkt_drop: 0
dev.em.0.host.rx_good_bytes: 0
dev.em.0.host.tx_good_bytes: 0
dev.em.0.host.length_errors: 0
dev.em.0.host.serdes_violation_pkt: 0
dev.em.0.host.header_redir_missed: 0
dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.0.5
dev.em.1.%driver: em
dev.em.1.%location: slot=0 function=0
dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.1.%parent: pci2
dev.em.1.nvm: -1
dev.em.1.rx_int_delay: 200
dev.em.1.tx_int_delay: 200
dev.em.1.rx_abs_int_delay: 500
dev.em.1.tx_abs_int_delay: 500
dev.em.1.rx_processing_limit: 100
dev.em.1.link_irq: 7
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.dropped: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.fc_high_water: 18432
dev.em.1.fc_low_water: 16932
dev.em.1.mac_stats.excess_coll: 0
dev.em.1.mac_stats.symbol_errors: 0
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.defer_count: 0
dev.em.1.mac_stats.missed_packets: 0
dev.em.1.mac_stats.recv_no_buff: 0
dev.em.1.mac_stats.recv_errs: 0
dev.em.1.mac_stats.crc_errs: 0
dev.em.1.mac_stats.alignment_errs: 0
dev.em.1.mac_stats.coll_ext_errs: 0
dev.em.1.mac_stats.rx_overruns: 0
dev.em.1.mac_stats.watchdog_timeouts: 3
dev.em.1.mac_stats.xon_recvd: 0
dev.em.1.mac_stats.xon_txd: 0
dev.em.1.mac_stats.xoff_recvd: 0
dev.em.1.mac_stats.xoff_txd: 0
dev.em.1.mac_stats.total_pkts_recvd: 638131
dev.em.1.mac_stats.good_pkts_recvd: 638131
dev.em.1.mac_stats.bcast_pkts_recvd: 49276
dev.em.1.mac_stats.mcast_pkts_recvd: 0
dev.em.1.mac_stats.rx_frames_64: 8091
dev.em.1.mac_stats.rx_frames_65_127: 472864
dev.em.1.mac_stats.rx_frames_128_255: 46165
dev.em.1.mac_stats.rx_frames_256_511: 33998
dev.em.1.mac_stats.rx_frames_512_1023: 44560
dev.em.1.mac_stats.rx_frames_1024_1522: 32453
dev.em.1.mac_stats.good_octets_recvd: 0
dev.em.1.mac_stats.good_octest_txd: 0
dev.em.1.mac_stats.total_pkts_txd: 668981
dev.em.1.mac_stats.good_pkts_txd: 668981
dev.em.1.mac_stats.bcast_pkts_txd: 0
dev.em.1.mac_stats.mcast_pkts_txd: 12
dev.em.1.mac_stats.tx_frames_64: 5594
dev.em.1.mac_stats.tx_frames_65_127: 166028
dev.em.1.mac_stats.tx_frames_128_255: 33061
dev.em.1.mac_stats.tx_frames_256_511: 28853
dev.em.1.mac_stats.tx_frames_512_1023: 24279
dev.em.1.mac_stats.tx_frames_1024_1522: 411166
dev.em.1.mac_stats.tso_txd: 0
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.interrupts.asserts: 0
dev.em.1.interrupts.rx_pkt_timer: 0
dev.em.1.interrupts.rx_abs_timer: 0
dev.em.1.interrupts.tx_pkt_timer: 0
dev.em.1.interrupts.tx_abs_timer: 0
dev.em.1.interrupts.tx_queue_empty: 0
dev.em.1.interrupts.tx_queue_min_thresh: 0
dev.em.1.interrupts.rx_desc_min_thresh: 0
dev.em.1.interrupts.rx_overrun: 0
dev.em.1.host.breaker_tx_pkt: 0
dev.em.1.host.host_tx_pkt_discard: 0
dev.em.1.host.rx_pkt: 0
dev.em.1.host.breaker_rx_pkts: 0
dev.em.1.host.breaker_rx_pkt_drop: 0
dev.em.1.host.tx_good_pkt: 0
dev.em.1.host.breaker_tx_pkt_drop: 0
dev.em.1.host.rx_good_bytes: 0
dev.em.1.host.tx_good_bytes: 0
dev.em.1.host.length_errors: 0
dev.em.1.host.serdes_violation_pkt: 0
dev.em.1.host.header_redir_missed: 0

loader.conf variables:

if_em_load=YES
hw.em.rxd=4096
hw.em.txd=4096
hw.em.tx_int_delay=200
hw.em.rx_int_delay=200
hw.em.tx_abs_int_delay=500
hw.em.rx_abs_int_delay=500
hw.em.enable_msix=1
hw.em.msix_queues=2
hw.em.rx_process_limit=100
hw.em.fc_setting=0


And here is vmstat -i

interrupt total rate
irq16: uhci0 2673 2


irq18: ehci0 uhci5 2 0

irq19: uhci2 uhci4 70 0
irq23: uhci3 ehci1 8835 9
cpu0: timer 3743479 3957
irq256: em0 662219 700
irq257: em0 717531 758
irq259: em0 12142 12
irq260: em0 6100 6
irq261: em1 694169 733
irq262: em1 217388 229
irq264: em1 463023 489
irq265: em1 11 0
cpu1: timer 3743411 3957
cpu6: timer 3743408 3957
cpu7: timer 3743407 3957
cpu4: timer 3743408 3957
cpu5: timer 3743407 3957
cpu3: timer 3743408 3957
cpu2: timer 3743409 3957
Total 32731500 34599


If you need more information just let me know, I can use this box for
tests at night until others can handle traffic.

Jack Vogel

unread,
Jul 6, 2010, 8:09:00 PM7/6/10
to
Yow, 30 vlans, but only em1 is using vlans not em0?

Is only em1 having watchdogs? I noticed you appear to
have flow control off, maybe turning it on would help.

I would like to see the log messages from the watchdogs.

Jack

Shtorm

unread,
Jul 7, 2010, 6:03:02 AM7/7/10
to

> Yow, 30 vlans, but only em1 is using vlans not em0?
>
> Is only em1 having watchdogs? I noticed you appear to
> have flow control off, maybe turning it on would help.
>
> I would like to see the log messages from the watchdogs.
> Jack

Yes, em0 - plain untagged traffic to border router, em1 - tagged - one
vlan per 200-300 pppoe clients. Anyway, I saw watchdogs on em0 too,
there is no logs for it because remote syslog server connected via em0
and it looses messages during card reset, will enable local logs to get
some info.

Log files are almost empty, is there any driver-specific debugging
options other than TUNABLE_INT("hw.em.sbp", &em_debug_sbp)? Anyway will
try to set it to 1 and wait for watchdog.

Here is a part from log file I have now

Jul 6 10:32:34 ntp info hostname x.x.x.8 ntpd adjusting local clock by
5.083720s
Jul 6 10:33:07 ntp info hostname x.x.x.8 ntpd adjusting local clock by
4.915903s
Jul 6 10:35:01 auth info hostname x.x.x.8 login login on ttyv2 as root
Jul 6 10:35:01 auth notice hostname x.x.x.8 login ROOT LOGIN (root) ON
ttyv2
Jul 6 10:35:24 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout
-- resetting
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel em1: link state
changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1028: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1020: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1005: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1029: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1021: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1004: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1030: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1022: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1007: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1031: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1023: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1006: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1024: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1016: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1001: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1025: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1017: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1000: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1026: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1018: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1003: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1027: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1019: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1002: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1012: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1013: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1014: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1015: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1032: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1008: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1033: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1009: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1010: link
state changed to DOWN
Jul 6 10:35:24 kern notice hostname x.x.x.8 kernel vlan1011: link
state changed to DOWN
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel em1: link state
changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1028: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1020: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1005: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1029: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1021: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1004: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1030: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1022: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1007: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1031: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1023: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1006: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1024: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1016: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1001: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1025: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1017: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1000: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1026: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1018: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1003: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1027: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1019: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1002: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1012: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1013: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1014: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1015: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1032: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1008: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1033: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1009: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1010: link
state changed to UP
Jul 6 10:35:27 kern notice hostname x.x.x.8 kernel vlan1011: link
state changed to UP
Jul 6 10:37:21 ntp info hostname x.x.x.8 ntpd adjusting local clock by
3.641940s
Jul 6 10:37:46 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout
-- resetting
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel em1: link state
changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1028: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1020: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1005: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1029: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1021: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1004: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1030: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1022: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1007: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1031: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1023: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1006: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1024: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1016: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1001: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1025: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1017: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1000: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1026: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1018: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1003: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1027: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1019: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1002: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1012: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1013: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1014: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1015: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1032: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1008: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1033: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1009: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1010: link
state changed to DOWN
Jul 6 10:37:46 kern notice hostname x.x.x.8 kernel vlan1011: link
state changed to DOWN
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel em1: link state
changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1028: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1020: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1005: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1029: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1021: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1004: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1030: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1022: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1007: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1031: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1023: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1006: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1024: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1016: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1001: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1025: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1017: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1000: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1026: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1018: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1003: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1027: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1019: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1002: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1012: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1013: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1014: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1015: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1032: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1008: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1033: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1009: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1010: link
state changed to UP
Jul 6 10:37:49 kern notice hostname x.x.x.8 kernel vlan1011: link
state changed to UP
Jul 6 10:38:40 kern crit hostname x.x.x.8 kernel Limiting icmp unreach
response from 237 to 200 packets/sec
Jul 6 10:39:10 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout
-- resetting
Jul 6 10:39:10 kern notice hostname x.x.x.8 kernel em1: link state
changed to DOWN

Mike Tancsa

unread,
Jul 7, 2010, 9:36:42 AM7/7/10
to
At 06:03 AM 7/7/2010, Shtorm wrote:

> > Yow, 30 vlans, but only em1 is using vlans not em0?
> >
> > Is only em1 having watchdogs? I noticed you appear to
> > have flow control off, maybe turning it on would help.
> >
> > I would like to see the log messages from the watchdogs.
> > Jack
>
>Yes, em0 - plain untagged traffic to border router, em1 - tagged - one
>vlan per 200-300 pppoe clients. Anyway, I saw watchdogs on em0 too,
>there is no logs for it because remote syslog server connected via em0
>and it looses messages during card reset, will enable local logs to get
>some info.

I only have one board that has a pair of these NICs, but I noticed
that in the BIOS, there is an option to enable or disable the Option
ROM to do things like WOL, some sort of IPMI and other functions /
features I am not familiar with and dont use.


| |ROM for the
onboard |
| Maximize Memory below [Disabled] |network
controllers. |
| Memory Mapped I/O abo [Disabled] |Warning: If
[Disabled] |
| Onboard Video [Enabled] |is selected,
NIC2 can |
| Dual Monitor Video [Disabled] |not be used to
boot or |
| Onboard NIC1 ROM [Enabled] |wake the
system. |
| Onboard NIC2
ROM [Enabled] | |
| Onboard NIC iSCSI
ROM [Disabled] | |
| |
|
| NIC1 MAC Address 001517C84B98 |>< Select
Screen |
| NIC2 MAC Address 001517C84B99 |^v Select
Item |
| |+/- Change
Value |

If you have such features, perhaps try disabling them in the BIOS ?

On mine, the two nics dont come up the same... Not sure if their
capabilities are the same even ?

em2@pci0:0:25:0: class=0x020000 card=0x34ec8086
chip=0x10ef8086 rev=0x05 hdr=0x00
vendor = 'Intel Corporation'


class = network
subclass = ethernet

cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP


em3@pci0:3:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086

rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class = network
subclass = ethernet

cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
cap 11[a0] = MSI-X supports 5 messages in map 0x1c

em2: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x3040-0x305f
mem 0xb1b00000-0xb1b1ffff,0xb1b25000-0xb1b25fff irq 16 at device 25.0 on pci0
em2: Using MSI interrupt
em2: [FILTER]
em2: Ethernet address: 00:15:17:c8:4b:99

em3: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x1000-0x101f
mem 0xb1900000-0xb191ffff,0xb1920000-0xb1923fff irq 16 at device 0.0 on pci3
em3: Using MSI interrupt
em3: [FILTER]
em3: Ethernet address: 00:15:17:c8:4b:98


---Mike


--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mi...@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike

Jack Vogel

unread,
Jul 7, 2010, 12:22:02 PM7/7/10
to
Interesting, didn't know any vendor had combined PCH and 82574 on one board,
yes, they are
completely different, as you can see the one only supports MSI, its a client
NIC.

I would agree with Mike, disable anything having to do with IPMI if it
exists and see if that
helps. Also you did not comment on the flow control setting, how about
turning that on?

Jack

Mike Tancsa

unread,
Jul 7, 2010, 1:18:47 PM7/7/10
to
At 12:22 PM 7/7/2010, Jack Vogel wrote:
>Interesting, didn't know any vendor had combined PCH and 82574 on
>one board, yes, they are
>completely different, as you can see the one only supports MSI, its
>a client NIC.

Its one of the lesser known vendors, Intel ;-) Its a S3420GPLC

<http://www.intel.com/Products/Server/Motherboards/S3420GP/S3420GP-specifications.htm>http://www.intel.com/Products/Server/Motherboards/S3420GP/S3420GP-specifications.htm


---Mike

><mailto:mi...@sentex.net>mi...@sentex.net
>Providing Internet since
>1994 <http://www.sentex.net>www.sentex.net
>Cambridge, Ontario
>Canada <http://www.sentex.net/mike>www.sentex.net/mike

Jack Vogel

unread,
Jul 7, 2010, 2:02:06 PM7/7/10
to
LOL, figures. The poor FreeBSD engineer never gets the new toys :(

I asked and my tester has actually had one of these, so if needed I
imagine I can lay my hands on one.

Jack

Shtorm

unread,
Jul 9, 2010, 8:55:34 AM7/9/10
to
On Wed, 2010-07-07 at 09:22 -0700, Jack Vogel wrote:
> Interesting, didn't know any vendor had combined PCH and 82574 on one board,
> yes, they are
> completely different, as you can see the one only supports MSI, its a client
> NIC.
>
> I would agree with Mike, disable anything having to do with IPMI if it
> exists and see if that
> helps. Also you did not comment on the flow control setting, how about
> turning that on?
>
> Jack

Flow control does not matter, I have timeouts with and without them.

I've updated bios to latest version, all IPMI stuff was already disabled
- we do not use it anyway. If it matters, motherboard have dedicated
ethernet port for IPMI. Log shows watchdog timeouts from em0 and em1.

Jul 8 07:22:26 <kern.crit> server kernel: em1: Watchdog timeout --
resetting
Jul 8 07:22:26 <kern.notice> server kernel: em1: link state changed to
DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1028: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1020: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1005: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1029: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1021: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1004: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1030: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1022: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1007: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1031: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1023: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1006: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1024: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1016: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1001: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1025: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1017: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1000: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1026: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1018: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1003: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1027: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1019: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1002: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1012: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1013: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1014: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1015: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1032: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1008: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1033: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1009: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1010: link state
changed to DOWN
Jul 8 07:22:26 <kern.notice> server kernel: vlan1011: link state
changed to DOWN
Jul 8 07:22:30 <kern.notice> server kernel: em1: link state changed to
UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1028: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1020: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1005: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1029: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1021: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1004: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1030: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1022: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1007: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1031: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1023: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1006: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1024: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1016: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1001: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1025: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1017: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1000: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1026: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1018: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1003: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1027: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1019: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1002: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1012: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1013: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1014: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1015: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1032: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1008: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1033: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1009: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1010: link state
changed to UP
Jul 8 07:22:30 <kern.notice> server kernel: vlan1011: link state
changed to UP
Jul 8 07:23:40 <kern.crit> server kernel: em0: Watchdog timeout --
resetting
Jul 8 07:23:40 <kern.notice> server kernel: em0: link state changed to
DOWN
Jul 8 07:23:43 <kern.notice> server kernel: em0: link state changed to
UP
Jul 8 07:23:56 <ntp.crit> server ntpd[3687]: 2 out of 3 peers valid
Jul 8 07:23:56 <ntp.crit> server ntpd[3687]: bad peer from pool
pool.ntp.org (195.214.215.17)
Jul 8 07:27:15 <kern.crit> server kernel: em0: Watchdog timeout --
resetting
Jul 8 07:27:15 <kern.notice> server kernel: em0: link state changed to
DOWN
Jul 8 07:27:18 <kern.notice> server kernel: em0: link state changed to
UP

Ian Smith

unread,
Jul 9, 2010, 9:58:47 AM7/9/10
to
On Wed, 7 Jul 2010, Shtorm wrote:
>
> > Yow, 30 vlans, but only em1 is using vlans not em0?
> >
> > Is only em1 having watchdogs? I noticed you appear to
> > have flow control off, maybe turning it on would help.
> >
> > I would like to see the log messages from the watchdogs.
> > Jack
>
> Yes, em0 - plain untagged traffic to border router, em1 - tagged - one
> vlan per 200-300 pppoe clients. Anyway, I saw watchdogs on em0 too,
> there is no logs for it because remote syslog server connected via em0
> and it looses messages during card reset, will enable local logs to get
> some info.
>
> Log files are almost empty, is there any driver-specific debugging
> options other than TUNABLE_INT("hw.em.sbp", &em_debug_sbp)? Anyway will
> try to set it to 1 and wait for watchdog.
>
> Here is a part from log file I have now

Deleting the stuff you're most interested in :)

> Jul 6 10:32:34 ntp info hostname x.x.x.8 ntpd adjusting local clock by 5.083720s
> Jul 6 10:33:07 ntp info hostname x.x.x.8 ntpd adjusting local clock by 4.915903s
> Jul 6 10:35:01 auth info hostname x.x.x.8 login login on ttyv2 as root
> Jul 6 10:35:01 auth notice hostname x.x.x.8 login ROOT LOGIN (root) ON ttyv2

> Jul 6 10:35:24 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout -- resetting
[..]


> Jul 6 10:37:21 ntp info hostname x.x.x.8 ntpd adjusting local clock by 3.641940s

> Jul 6 10:37:46 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout -- resetting
[..]


> Jul 6 10:38:40 kern crit hostname x.x.x.8 kernel Limiting icmp unreach
> response from 237 to 200 packets/sec

> Jul 6 10:39:10 kern crit hostname x.x.x.8 kernel em1: Watchdog timeout > -- resetting

Probably completely unrelated, but I can't help noticing those big clock
shifts by ntp over a short period amidst all this. I don't know if that
could affect watchdogs, but is it a regular occurrence during these?

>From your latest, a bit more noise from ntp:

> Jul 8 07:23:40 <kern.crit> server kernel: em0: Watchdog timeout -- resetting

[..]


> Jul 8 07:23:56 <ntp.crit> server ntpd[3687]: 2 out of 3 peers valid
> Jul 8 07:23:56 <ntp.crit> server ntpd[3687]: bad peer from pool pool.ntp.org (195.214.215.17)
> Jul 8 07:27:15 <kern.crit> server kernel: em0: Watchdog timeout -- resetting

Ignore if not relevant.

cheers, Ian

Shtorm

unread,
Jul 9, 2010, 11:07:26 AM7/9/10
to

Yeah, saw this too, it was first boot for this install and I forgot to
run tzsetup during flash image build.

As for the latest log, this box connected to internet via em0, ntpd just
says it have some peers to sync with after interface flap.

Thanks.

Jack Vogel

unread,
Jul 9, 2010, 12:26:50 PM7/9/10
to
LOL, the way the watchdog code works these days it records the clock
at key TX points and then compares that in the timer code, so if your
system is dinking around with the time that could be the cause of this.

Can you change the environment to guarantee a continuous time
stream and then see what happens??

Jack

Ian Smith

unread,
Jul 9, 2010, 12:32:07 PM7/9/10
to
On Fri, 9 Jul 2010, Shtorm wrote:

> Yeah, saw this too, it was first boot for this install and I forgot to
> run tzsetup during flash image build.
>
> As for the latest log, this box connected to internet via em0, ntpd just
> says it have some peers to sync with after interface flap.
>
> Thanks.

'scuse the noise, I'll stick to lurking ..

Ryan Stone

unread,
Jul 9, 2010, 12:52:20 PM7/9/10
to
Hm, I see that you're using the ticks variable. To the best of my
knowledge, changing the current time should not cause ticks to change.
A lot of things, like the callout wheel, would be pretty broken if
that were happening. My knowledge of this stuff is several major
versions out of date, though, so maybe things have changed.

Jack Vogel

unread,
Jul 9, 2010, 1:04:26 PM7/9/10
to
OK, its just that something odd is going on since my testers are not seeing
this happening, fishing for ideas....

Jack

Shtorm

unread,
Jul 9, 2010, 1:24:09 PM7/9/10
to
Yes, I'm not a kernel hacker, but cat understand what C code says, I see
your point. It will take few days until I can get to server room, will
try without ntpd.

Also, I have kern.hz=4000 in loader.conf, as far as I understand tick
length will be 25 ms and timeout for watchdog will be 10*25 ms = 250ms
Is it enough for adapter or I need to increase

#define EM_WATCHDOG (10 * hz)

I can be wrong with this, just do not have appropriate knowledge.

Thanks for your help.

On Fri, 2010-07-09 at 09:26 -0700, Jack Vogel wrote:
> LOL, the way the watchdog code works these days it records the clock
> at key TX points and then compares that in the timer code, so if your
> system is dinking around with the time that could be the cause of this.
>
> Can you change the environment to guarantee a continuous time
> stream and then see what happens??
>

> Jack
>
>

> > > _______________________________________________
> > > freeb...@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > > To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
> >

> > Yeah, saw this too, it was first boot for this install and I forgot to
> > run tzsetup during flash image build.
> >
> > As for the latest log, this box connected to internet via em0, ntpd just
> > says it have some peers to sync with after interface flap.
> >
> > Thanks.
> >
> >
> >

Ryan Stone

unread,
Jul 9, 2010, 1:28:16 PM7/9/10
to
No, defining EM_WATCHDOG as 10 * hz should mean that the watchdog
expires after 10 seconds no matter what your kern.hz is. hz is set to
the number of ticks in a second.

Jack Vogel

unread,
Jul 9, 2010, 1:46:58 PM7/9/10
to
Well, you can always try increasing it, but its unlikely that the
problem is the constant, in the older style watchdog code this
was set to 5 so I am already being lenient :)

Jack

> > > > _______________________________________________
> > > > freeb...@freebsd.org mailing list
> > > > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > > > To unsubscribe, send any mail to "
> freebsd-net...@freebsd.org"
> > >

> > > Yeah, saw this too, it was first boot for this install and I forgot to
> > > run tzsetup during flash image build.
> > >
> > > As for the latest log, this box connected to internet via em0, ntpd
> just
> > > says it have some peers to sync with after interface flap.
> > >
> > > Thanks.
> > >
> > >
> > >

Ian Smith

unread,
Jul 9, 2010, 2:48:55 PM7/9/10
to
On Fri, 9 Jul 2010, Ryan Stone wrote:

> No, defining EM_WATCHDOG as 10 * hz should mean that the watchdog
> expires after 10 seconds no matter what your kern.hz is. hz is set to
> the number of ticks in a second.

Ok, one more probably wild punt .. Shtorm you say HZ=4000, giving:

===


And here is vmstat -i
interrupt total rate
irq16: uhci0 2673 2
irq18: ehci0 uhci5 2 0
irq19: uhci2 uhci4 70 0
irq23: uhci3 ehci1 8835 9
cpu0: timer 3743479 3957
irq256: em0 662219 700
irq257: em0 717531 758
irq259: em0 12142 12
irq260: em0 6100 6
irq261: em1 694169 733
irq262: em1 217388 229
irq264: em1 463023 489
irq265: em1 11 0
cpu1: timer 3743411 3957
cpu6: timer 3743408 3957
cpu7: timer 3743407 3957
cpu4: timer 3743408 3957
cpu5: timer 3743407 3957
cpu3: timer 3743408 3957
cpu2: timer 3743409 3957
Total 32731500 34599

===

a little more variant from 4000 than expected? Originally:

===


but vmstat -i shows only one pair is used
%vmstat -i
interrupt total rate
irq1: atkbd0 391 0

irq18: ehci0 uhci5 2 0

irq19: uhci2 uhci4 27 0
irq23: uhci3 ehci1 8463 0
cpu0: timer 107412943 7864
irq256: em0 93311951 6832
irq257: em0 90479067 6624
irq260: em0 2 0
irq261: em1 92966894 6806
irq262: em1 80298240 5879
irq265: em1 1 0
cpu1: timer 107412607 7864
cpu2: timer 107412785 7864
cpu3: timer 107412830 7864
cpu5: timer 107413210 7864
cpu4: timer 107413389 7864
cpu7: timer 107407337 7864
cpu6: timer 107413132 7864
Total 1216363271 89058

===

Was that with HZ=8000 ? Or what? In any case, em interrupt rates were
a whole lot higher then - but maybe it was just a lot busier on the net?

HZ=4000 ticks are 250ns, not 25ms.

Seems like you're not lacking horsepower .. unless you're using POLLING
(not indicated) or, say, dummynet pipes needing finer-grained output
queue scheduling, it might be worth trying the default HZ=1000 ?

Just curious along similar lines: what says sysctl kern.timecounter ?

Ian Smith

unread,
Jul 9, 2010, 3:05:11 PM7/9/10
to
On Sat, 10 Jul 2010, Ian Smith wrote:
>
> HZ=4000 ticks are 250ns, not 25ms.

Up way too late .. that's 250us of course, thanks Ryan.

Jack Vogel

unread,
Jul 9, 2010, 4:17:22 PM7/9/10
to
On Fri, Jul 9, 2010 at 12:05 PM, Ian Smith <smi...@nimnet.asn.au> wrote:

> On Sat, 10 Jul 2010, Ian Smith wrote:
> >
> > HZ=4000 ticks are 250ns, not 25ms.
>
> Up way too late .. that's 250us of course, thanks Ryan.
>

Even so, very good points Ian, thanks.

Hmmm, wondering if this points to a design flaw in what I
did or just a 'don't do that' kinda thing :)

Jack

Shtorm

unread,
Jul 10, 2010, 11:31:39 AM7/10/10
to
> HZ=4000 ticks are 250ns, not 25ms.
>
> Seems like you're not lacking horsepower .. unless you're using POLLING
> (not indicated) or, say, dummynet pipes needing finer-grained output
> queue scheduling, it might be worth trying the default HZ=1000 ?
>
> Just curious along similar lines: what says sysctl kern.timecounter ?
>
> cheers, Ian
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

You are rigth, Ian, I'm using dumminet for traffic shaping, arount 2000
dynamic pipes (src and dst mask) with speed from 256 kbit up to 100
mbit. I've played with kern.hz a bit setting it to different values
between 1000 and 8000, 4000 gives best for my traffic load, two logs
taken with hz=4000 and hz=8000. And no, I'm not using polling, one of
the great features of intell network cards is interrupt moderation.
There is sysctls for this em driver:

%sysctl dev.em.0 | grep \[tr\]x


dev.em.0.rx_int_delay: 200
dev.em.0.tx_int_delay: 200
dev.em.0.rx_abs_int_delay: 500
dev.em.0.tx_abs_int_delay: 500

I've tried different values, more delay gives less context switches and
reduces cpu load in case of packet forwarding a lot. And you can see I'm
getting watchdog timeouts regardless of this settings, even tried
everything default with empty loader.conf.

Sorry for this, next time will do one thing at a time, not a few in
parallel.

Thank you.

0 new messages