Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Ethernet Switch Framework

111 views
Skip to first unread message

Aleksandr Rybalko

unread,
Jan 20, 2012, 3:13:19 PM1/20/12
to
Hi,

Sorry for cross posting. I just thought this might interesting to
review for subscribers on both MLs.

I glad to introduce working version of Ethernet Switch Framework. of
course, here is so many things to do, but it already work and can help
us in some situations for embedded devices.

here is the patch:
http://my.ddteam.net/files/2012-01-20_switch_framework.patch

It include sys/mips/conf/AR7240, that together with hints file is good
example for typical AR7240 setup.

Since it still don't have any documentation, I will try to explain
it here.


The framework contain 3 basic parts:
1. core module, which handle ioctl calls and interact with driver.
2. Various bus glue (now it is obio(mem), mii(MDIO) and IIC in near
future)
3. drivers. Atheros AR8x16, Broadcom BCM53xx, Ralink RT305xF, Realtek
RTL8305/09
4. FloatPHY, pseudo driver which find master switchX device and ask his
PHY reg's.
5. switchctl utility.

Utility.
Currently can do
switchctl /dev/switch0 (get|set) (reg|port|vlan) [flags]

get/set port: get or set port flags:
"IngressCheck" - put port into VLAN mode, drop packets which have 802.1q
tag with value != PVID
"Q-in-Q" - enable double tag, add second tag to already tagged
packets.
"LAN" and "WAN" - flags, for switches which have special function for
LAN-WAN processing.
"Tagged" and "Untagged" - mark port Tagged/Untagged, used if switch
using Global Tag flag (one flag for all VLANs)
"pvid" - set Port VLAN ID
Example:
switchctl /dev/switch0 set port 2 pvid 2 flags IngressCheck LAN Tagged

get/set vlan V:
add N (tag|untag|forbid) - add port N to VLAN member ports as (tag|
untag| forbid)
del N - delete port N from VLAN member ports
vid N - assign VLAN ID to internal index V
Example:
switchctl /dev/switch0 set vlan 2 vid 12
switchctl /dev/switch0 set vlan 2 add 2 u
switchctl /dev/switch0 set vlan 2 del 1

get/set reg:
Generic access to registers. Have 2 address modifiers:
0x00000000(no modifiers) - access to parent space (parent MDIO bus, if
switchX attaches to miibusX)
0x40000000 - access to switch MDIO bus
0x80000000 - access to switch registers.
Example:
# switchctl /dev/switch0 get reg 0x80000008
Reg 0x80000008 Value = 0x012603e2
# switchctl /dev/switch0 get reg 0x00000008
Reg 0x00000008 Value = 0x0000ffff
# switchctl /dev/switch0 get reg 0x40000001
Reg 0x40000001 Value = 0x00007949
# switchctl /dev/switch0 set reg 0x80000008 0x012603e2
Reg 0x80000008 Value = 0x012603e2 (Old value = 0x012603e2)

FloatPHY
pseudo driver which attach to miibus like normal PHY, but do find
master switchX device and ask his PHY reg's. Main problem with that
driver - is usage of newbus calls between independent device (not a
parent <-> child), since floatphyX query set/get methods of switchX.

Use hints:
"master" - to set master name.
"master_unit" - master unit.
"master_phys" - bitmap of PHY numbers on which get link status/speed.
"flags" - see driver (dev/switch/floatphy.c).
"speed" - initial link speed value, used when no access to master.

I will describe as example 4 situation that current framework is cover:
1. Ralink RT305X SoC, internal switch, one NIC with two paths.
2. Atheros AR7240 SoC, two NIC, but MDIO routed only from second
3. Cavium Octeon CN5010, one NIC with three paths + BCM53115 switch +
some Broadcom PHY

1. Ralink RT305X - is simple one
Attached by obio0 if driver present in kernel.

VLAN features: it have so called global untagged flag, so port can be
member of any VLAN but may be tagged or untagged in all VLAN in same
time.

How to enable: just add following into kernel config file.
------------------------------------------------
device switch
device switch_rt305x
------------------------------------------------

2. Atheros AR7240 - very interesting. have AR8216 internal switch and
two arge NICs.

arge0 MII bus connected to PHY4 which configured as separate PHY(not
attached to switch core). But PHY4 reg's can be accessible only via
switch MDIO bus access.
arge1 MII bus connected to switch MII.
But MDIO bus wired only on arge0, so if arge0 want to know speed and
link status, it must ask switch connected to miibus attached to arge1.
[1] Page 3

VLAN features: Like RT395x use global untagged like flag.

How to enable (example config for AR7240 in patch)

Config:
------------------------------------------------
device mii
device switch
device switch_ar8x16
------------------------------------------------

hints:
------------------------------------------------
# No probe at all
# First MDIO connected to switch which not have real PHY regs
hint.miibus.0.phymask="0x00000000"
# Second MDIO not wired at all
hint.miibus.1.phymask="0x00000000"

# Connect pseudo PHY driver to miibus0
hint.floatphy.0.at="miibus0"
hint.floatphy.0.phyno=0
# floatphy0 will ask switch0
hint.floatphy.0.master="switch"
hint.floatphy.0.master_unit=0
# and get link status from PHYs masked by 0x00000010
hint.floatphy.0.master_phys=0x00000010 # Sense PHY4
hint.floatphy.0.flags=0x00000000
# Default link speed 100 (if no access to master)
hint.floatphy.0.speed=100

# Switch attached to MDIO bus on arge0
hint.switch.0.at="miibus0"
hint.switch.0.phyno=1
# AR8x16 Magic register which can configure PHY4 as a standalone PHY
hint.ar8x16_switch.0.mii_mode=0x012603e2

hint.floatphy.1.at="miibus1"
hint.floatphy.1.phyno=0
hint.floatphy.1.master="switch"
hint.floatphy.1.master_unit=0
# check link on PHY0-PHY3 (link on any rise link on arge1)
hint.floatphy.1.master_phys=0x0000000f # Link Sensing PHY0-PHY3
hint.floatphy.1.flags=0x00000004
# "Link on any PHYs" | "Static link speed"
hint.floatphy.1.speed=1000
------------------------------------------------

3. Cavium Octeon CN5010, one NIC with three paths + BCM53115 switch +
some Broadcom PHY.
Since here is required a lot of magic to attach anything that Cavium
was not expect as "can be attached" (i.e. Cavium SDK limitation), I
made patch which allow to attach one PHY driver per NIC path (per
octe0 iface).

VLAN features: it seems have most clear VLAN implementation, except
some things like remapped some regs which have port bit maps. (seems
forget to think about bigger port count when design small switches)

How to enable:
Config:
------------------------------------------------
device brgphy
device switch
device switch_bcm5325
------------------------------------------------

Hints:
------------------------------------------------
hint.miibus.0.phymask="0x00000000"
hint.miibus.1.phymask="0x00000000"
hint.miibus.3.phymask="0x00000100"
# brgphy will attach here

hint.floatphy.0.at="miibus0"
hint.floatphy.0.phyno=0
hint.floatphy.0.master="switch"
hint.floatphy.0.master_unit=0
# Check link on any ports
hint.floatphy.0.master_phys=0x0000001f # Sense PHY0
hint.floatphy.0.flags=0x00000000
hint.floatphy.0.speed=1000

hint.switch.0.at="miibus1"
hint.switch.0.phyno=30
------------------------------------------------

Huh, hope that is enough for first release of manual :)

Will wait for any comments and suggestions!

We're committing this in 5 days unless anyone objects in any meaningful
way.



1. http://my.ddteam.net/files/Switch_Framework.pdf

WBW
--
Aleksandr Rybalko <r...@freebsd.org>
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

Stefan Bethke

unread,
Jan 20, 2012, 6:35:16 PM1/20/12
to
Thank you for the update, that clears up a number of questions I had.

Here's a couple of questions and comments.

Which devices can this code be run on? For example, can the AR7240 config run on an TL-WR3420?

Would you mind giving an overview of how the various parts fit together, how they are probed and attached? I think I understand from you explanations on IRC, but not everbody had that chance.

Am 20.01.2012 um 21:13 schrieb Aleksandr Rybalko:

> get/set reg:
> Generic access to registers. Have 2 address modifiers:
> 0x00000000(no modifiers) - access to parent space (parent MDIO bus, if
> switchX attaches to miibusX)
> 0x40000000 - access to switch MDIO bus
> 0x80000000 - access to switch registers.

Wouldn't it be better to have a proper API to select the various busses and device register files?

> FloatPHY
> pseudo driver which attach to miibus like normal PHY, but do find
> master switchX device and ask his PHY reg's. Main problem with that
> driver - is usage of newbus calls between independent device (not a
> parent <-> child), since floatphyX query set/get methods of switchX.

The general approach has a number of problems, as far as I understand the miibus code. miibus assumes that only one of the PHYs on the MII is active concurrently (since that's a requirement of the MII/GMII/... bus). If I read your code correctly, you have one miibus that has all the PHYs for all switch ports on them. Won't miibus just isolate all but one PHY?

In your current implementation, you've reimplented ukphy (in a limited fashion). One of the advantages of reusing the mii framework is being able to use all the PHYs, in case a switch chip would include a quirky PHY. Extending floatphy to work as a full proxy for any phy driver looks non-trivial to me due to the API constraints the mii framework imposes.


Stefan

--
Stefan Bethke <s...@lassitu.de> Fon +49 151 14070811

Stefan Bethke

unread,
Jan 22, 2012, 10:31:06 AM1/22/12
to
Am 20.01.2012 um 21:13 schrieb Aleksandr Rybalko:

> It include sys/mips/conf/AR7240, that together with hints file is good
> example for typical AR7240 setup.

IÄm heaving trouble getting this to work. The patch applies cleanly and I can get a kernel compiled and booted, but neither arge0 nor arge1 appear to be functional. I had to roll my own kernel config as your AR7240 hangs before printing anything on my TL-MR3420.

dmesg and devinfo below.


Stefan

CPU platform: Atheros AR7241 rev 1
CPU Frequency=400 MHz
CPU DDR Frequency=400 MHz
CPU AHB Frequency=200 MHz
platform frequency: 400000000
arguments:
a0 = 00000008
a1 = a1f87fb0
a2 = a1f88470
a3 = 00000004
Cmd line:argv is invalid
Environment:
envp is invalid
Cache info:
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e<PerfCount,WatchRegs,MIPS16,EJTAG>
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #1: Thu Jan 1 01:00:00 CET 1970
stb@dummy:/home/stb/working/fe/obj/mipseb/mips.mipseb/home/stb/working/fe/freebsd/sys/TL-MR3420D mips
WARNING: WITNESS option enabled, expect reduced performance.
real memory = 33554432 (32768K bytes)
avail memory = 25567232 (24MB)
random device not loaded; using insecure entropy
nexus0: <MIPS32 root nexus>
nexus0: failed to add child: arge0
nexus0: failed to add child: arge1
clock0: <Generic MIPS32 ticker> on nexus0
Timecounter "MIPS32" frequency 200000000 Hz quality 800
Event timer "MIPS32" frequency 200000000 Hz quality 800
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
gpio0: <Atheros AR71XX GPIO driver> on apb0
gpio0: [GIANT-LOCKED]
gpio0: function_set: 0x0
gpio0: function_clear: 0x0
gpio0: gpio pinmask=0x1943
gpioc0: <GPIO controller> on gpio0
gpiobus0: <GPIO bus> on gpio0
gpioled0: <GPIO led> at pin(s) 0 on gpiobus0
gpioled1: <GPIO led> at pin(s) 1 on gpiobus0
gpioled2: <GPIO led> at pin(s) 3 on gpiobus0
ehci0: <AR71XX Integrated USB 2.0 controller> at mem 0x1b000100-0x1bffffff irq 1 on nexus0
usbus0: set host controller mode
usbus0: EHCI version 1.0
usbus0: set host controller mode
usbus0: <AR71XX Integrated USB 2.0 controller> on ehci0
arge0: <Atheros AR71xx built-in ethernet interface> at mem 0x19000000-0x19000fff irq 2 on nexus0
arge0: Overriding MAC from EEPROM
arge0: No PHY specified, using mask 16
miibus0: <MII bus> on arge0
floatphy0 PHY 0 on miibus0
floatphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
switch0 PHY 1 on miibus0
switch0: 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master
ar8x16_switch0: <Unknown AR8x16 like Ethernet switch (00.00)> on switch0
arge0: Ethernet address: ff:ff:ff:ff:ff:ff
arge1: <Atheros AR71xx built-in ethernet interface> at mem 0x1a000000-0x1a000fff irq 3 on nexus0
arge1: No PHY specified, using mask 15
arge1: Ethernet address: ff:ff:ff:ff:ff:00
spi0: <AR71XX SPI> at mem 0x1f000000-0x1f00000f on nexus0
spibus0: <spibus bus> on spi0
mx25l0: <M25Pxx Flash Family> at cs 0 on spibus0
mx25l0: s25s1032, sector 65536 bytes, 64 sectors
ar71xx_wdog0: <Atheros AR71XX watchdog timer> on nexus0
ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Atheros> at usbus0
uhub0: <Atheros EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2: <Kingston> at usbus0
umass0: <Kingston DataTraveler G3, class 0/0, rev 2.00/1.00, addr 2> on usbus0
umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:0:0:-1: Attached to scbus0
Trying to mount root from ufs:map/rootfs.uzip []...
mountroot: waiting for device map/rootfs.uzip ...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler G3 PMAP> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 15252MB (31236096 512 byte sectors: 255H 63S/T 1944C)
Trying to mount root from ufs:da0s1a []...
warning: no time-of-day clock registered, system time will not be set accurately
Setting hostuuid: aed4c502-193a-11e1-b662-74ea3ae4d920.
Setting hostid: 0x6a714343.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 3849245 free (1085 frags, 481020 blocks, 0.0% fragmentation)
lock order reversal:
1st 0x8095691c ufs (ufs) @ /home/stb/working/fe/freebsd/sys/kern/vfs_mount.c:1245
2nd 0x80956e94 devfs (devfs) @ /home/stb/working/fe/freebsd/sys/kern/vfs_subr.c:2167
KDB: stack backtrace:
db_trace_thread+30 (?,?,?,?) ra c3f597c000000018 sp 0 sz 0
db_trace_self+1c (?,?,?,?) ra c3f597d800000018 sp 0 sz 0
800776d8+34 (?,?,?,?) ra c3f597f0000001a0 sp 0 sz 0
kdb_backtrace+44 (?,?,?,?) ra c3f5999000000018 sp 0 sz 0
801b3bbc+34 (?,?,?,?) ra c3f599a800000020 sp 0 sz 0
witness_checkorder+9cc (?,?,803b1f7c,877) ra c3f599c800000050 sp 0 sz 1
__lockmgr_args+948 (?,?,80956eb4,?) ra c3f59a1800000070 sp 0 sz 1
vop_stdlock+4c (?,?,?,?) ra c3f59a8800000028 sp 0 sz 0
VOP_LOCK1_APV+e4 (?,?,?,?) ra c3f59ab000000020 sp 0 sz 0
_vn_lock+84 (?,?,?,?) ra c3f59ad000000048 sp 0 sz 0
vget+c8 (?,?,?,?) ra c3f59b1800000030 sp 0 sz 0
devfs_allocv+100 (?,?,?,?) ra c3f59b4800000038 sp 0 sz 0
800ba9b4+4c (?,?,?,?) ra c3f59b8000000028 sp 0 sz 0
vflush+6c (?,?,0,80991300) ra c3f59ba8000000f8 sp 0 sz 1
800baa74+54 (?,?,?,?) ra c3f59ca000000028 sp 0 sz 0
dounmount+3f0 (809e8000,?,?,?) ra c3f59cc800000050 sp 100000000 sz 0
sys_unmount+39c (?,?,?,?) ra c3f59d18000000b0 sp 0 sz 0
trap+7f4 (?,?,?,?) ra c3f59dc8000000b8 sp 0 sz 0
MipsUserGenException+10c (?,?,?,404b53c0) ra c3f59e8000000000 sp 0 sz 0
pid 70
Mounting local file systems:.
Setting hostname: whitebox.lassitu.de.
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
floatphy0: found master switch0
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
arge0: link state changed to DOWN
Starting Network: lo0 arge0 arge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
arge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether ff:ff:ff:ff:ff:ff
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::481f:1b46:dbf1:bd78%arge0 prefixlen 64 scopeid 0x2
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: no carrier
arge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether ff:ff:ff:ff:ff:00
inet 44.128.65.33 netmask 0xffffffc0 broadcast 44.128.65.63
inet6 fe80::481f:1b46:dbf1:bd78%arge1 prefixlen 64 scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 100baseTX <full-duplex>
status: active
add net default: gateway 44.128.65.1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Mounting NFS file systems:mount_nfs: diesel: hostname nor servname provided, or not known
.
Creating and/or trimming log files.
No core dumps found.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Setting date via ntp.
Error : hostname nor servname provided, or not known
22 Jan 15:01:04 ntpdate[566]: can't find host diesel

22 Jan 15:01:04 ntpdate[566]: no servers can be used, exiting
Clearing /tmp (X related).
Mounting late file systems:mount_nfs: diesel: hostname nor servname provided, or not known
.
Mounting /etc/fstab filesystems failed,Jan 22 15:01:12 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
Enter full pathname of shell or RETURN for /bin/sh:
# devinfo
nexus0
clock0
apb0
uart0
gpio0
gpioc0
gpiobus0
gpioled0
gpioled1
gpioled2
ehci0
usbus0
uhub0
umass0
arge0
miibus0
floatphy0
switch0
ar8x16_switch0
arge1
spi0
spibus0
mx25l0
ar71xx_wdog0
#

Aleksandr Rybalko

unread,
Jan 22, 2012, 12:51:30 PM1/22/12
to
On Sun, 22 Jan 2012 16:31:06 +0100
Stefan Bethke <s...@lassitu.de> wrote:

> Am 20.01.2012 um 21:13 schrieb Aleksandr Rybalko:
>
> > It include sys/mips/conf/AR7240, that together with hints file is
> > good example for typical AR7240 setup.
>
> IÄm heaving trouble getting this to work. The patch applies cleanly
> and I can get a kernel compiled and booted, but neither arge0 nor
> arge1 appear to be functional. I had to roll my own kernel config as
> your AR7240 hangs before printing anything on my TL-MR3420.

Yeah, I know where is problem, to proper attach switch framework to
arge, arge must be regular NIC. Here is the patch for that:
http://my.ddteam.net/files/2012-01-22_arge.patch
Hope it will apply cleanly.

Patch have fixed both arge problems (problem for allocation of ring
buffer, and stray interrupts) + remove most phymask bits + whitespace
cleanup.

Thank you for testing that Stefan.

P.S. I can't test clear SoC config on my board, because my board id
D-Link DIR-615_E4 with modified U-Boot in it, which able to load only
FW images, but not ELF kernel. So I test it with ZRouter.org FW image
instead.

P.P.S. can you also show me network part of your config and hints files.

P.P.P.S. still working on your previous question about subj, already
begin work on more wide documentation on wiki, but still far enough :)
"http://wiki.freebsd.org/AleksandrRybalko/Switch Framework"
--
Aleksandr Rybalko <r...@freebsd.org>
0 new messages