Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.

Dismiss

VIMAGE

0 views

Skip to first unread message

Pietro Paolini

unread,

May 30, 2013, 6:35:31 AM5/30/13

Hello all,

I am a new bye on the FreeBSD and I am looking at the VIMAGE features experiencing some problems.
I added the options :
VIMAGE
if_bridge

and I removed
STCP

then I recompiled my kernel and install it.

After that, following this tutorial http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf I tried the "Exercise 2" which consist on
the following commands:

vimage -c n1
vimage -c n2
ngctl mkpeer efface ether ether
ngctl mkpeer efface ether ether
ngctl mkpeer em0: bridge lower link0
ngctl name em0:lower bridge0
ngctl connect em0: bridge0: upper link1
ngctl connect ngeth0: bridge0: ether link2
ngctl connect ngeth1: bridge0: ether link3
vimage -i n1 ngeth0 e0

But my virtual interface on the n1 vimage does not receive any packet from the external network while I can see the packet go out from it.

For instance using DHCP, e0 on n1 sends DHCP packets but it does not receive the answers (which are send, I verified it from wireshark), in adding
the ARP request for his IP address (if I try to add it statically) are not received then it can not answer.

At the end of the line the question is: how can I make this "virtual network" and the external real network be able to communicate ?

Thanks in advance.
Pietro.

_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

Joe

unread,

May 30, 2013, 8:29:43 AM5/30/13

1. That link is from 2007. So very much has changed since then.
There are more current links on the internet about this subject. Most
are for 8.X releases.

2. If your running 8.2-RELEASE or 9.1-RELEASE all you need to add is
"options vimage" statement to your kernel source and recompile.

3. There are 2 networking methods available for creating vnet/vimage
jail networks, if_bridge/epair and netgraph. The if_bridge/epair method
is far simpler to config and use then the netgraph method.

4. There are 2 methods of jail setup, the rc.d method where your jail
definition parameters go into the hosts rc.conf and the jail(8) method
where you can place each jails definition parameter in separate files.

5. There are two very important show stopper PRs on vimage,
164763 memory leak and 149050 the rc.d keyword "nojail" problem.
Vimage is a very long way from prime time usage, thats why it's labeled
as highly experimental. Host system freezes and page faults are common.

6. When it comes to running a firewall in a vnet/vimage jail your
limited to IPFW and it has limitations. Dummynet and in kernel NAT cause
system freezes. IPFILTER causes page fault at boot time. PF will run on
the host but not run in the vnet/vimage jail. Here are a bunch of PRs on
vimage firewall problems, 143621, 176092, 161094, 176992, 143808,
148155, 165252, 178480, 178482

Check out these links

http://druidbsd.sourceforge.net/vimage.shtml
http://devinteske.com/vimage-jails-on-freebsd-8
http://lists.freebsd.org/pipermail/freebsd-virtualization/2011-September/000747.html

http://bsdbased.com/2009/12/06/freebsd-8-vimage-epair-howto
http://zewaren.net/site/?q=node/78

Mark Moellering

unread,

May 30, 2013, 9:46:39 AM5/30/13

I would like to thank Pietro for asking the question and Joe for
answering, as I was looking into vimage myself. This sort of thing
really helps a lot of people who are exploring FreeBSD and new features.

Pietro Paolini

unread,

May 30, 2013, 11:21:17 AM5/30/13

Thanks so much, really interesting and good links but I can't again accomplish my task - I followed http://bsdbased.com/2009/12/06/freebsd-8-vimage-epair-howto as a base tutorial -

I would like simulate more client for my PC using this technology, and that's my topology:

1 - Two epairs created
2 - Two jail created
3 - Assign one intf for jail
4 - Add the real interface to a bridge where I put all the interfaces

-------
JAIL1
0b
-------

-------
JAIL2
1b
-------

-------
HOST
0a
1a
em0 --> REAL
-------

Where {0,1}a is the first pair and {0,1}b is the second and em0 is my real interface (it has an IP address), then I got to the problem, I tried
to :

jexec 2 dhclient epair1b

And I can see the DHCP packet with the correct MAC address going out, the server reply (I have a sniffer pc) but the transaction does not end successfully, what is really strange is that if I attach tcpdump on em0 I can NOT see the answer server sends while when I try

dhclient em0

I can see the packet going in and out and the DHCP transaction finish successfully. Do you have any idea about how can I accomplish my
target ? Maybe I am using the wrong technology ?

I would not surprised if I make a error on my configuration but what really interest me is if I CAN do that using jail.

Thanks a lot, and in advance ! :-)

Pietro.

Joe

unread,

May 30, 2013, 12:18:49 PM5/30/13

Pietro;

You really have to provide a lot more info about your host system and
jail configuration.

What version of FreeBSD are you running?

Which method do you use to create your jails, rc.d/rc.conf or jail(8).

How are your creating the if_bridge/epair network, IE entering commands
by hand or using a script? Post the script.

Can you post the output of ifconfig command after you have your jail
network created?

How are you handling the "rc.d nojail keyword" problem?

Have you manually assigned a private LAN ip address and default route to
the epairXb interface inside of the vnet jail?

DHCP will not work from inside of the vnet jail. Your ISP only assigns a
single dymamic IP address per each account. You already used your
assigned ip address for your host. If the computer your running the
vnet/vimage jail on is on a local area network and the gateway host is
running a DHCP server to dynamically assign private ip address to
computers on the LAN, then DHCP in the vnet/vimage jail may work.

To the broader question, why do you think you need a vnet/vimage jail in
the first place. You wrote "simulate more client for my PC", If client
means hosting paying clients then you sure don't want to be using vimage
because it highly experimental and NOT reliable.

Jails are a security feature that encloses a process in a container. By
process I mean "postfix email server" or "apache web server". If your
"client" means "processes", then this is what the non-vnet/vimage jail
is for.

Teske, Devin

unread,

May 30, 2013, 12:25:04 PM5/30/13

On May 30, 2013, at 3:35 AM, Pietro Paolini wrote:

Hello all,

I am a new bye on the FreeBSD and I am looking at the VIMAGE features experiencing some problems.
I added the options :
VIMAGE
if_bridge

and I removed
STCP

then I recompiled my kernel and install it.

After that, following this tutorial http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf I tried the "Exercise 2" which consist on
the following commands:

vimage -c n1
vimage -c n2
ngctl mkpeer efface ether ether
ngctl mkpeer efface ether ether

Don't you just love autocorrect? (does the same thing to me… turns "eiface" into "efface")

ngctl mkpeer em0: bridge lower link0

Looks good.

ngctl name em0:lower bridge0

I usually do my "connect" before the "name"… but shouldn't matter. Should work all the same.

ngctl connect em0: bridge0: upper link1

This looks wrong to me.

I'd expect:

ngctl connect em0: bridge0:lower upper link1

# The way I see most people using vimage… it seems like almost always a mistake to not hook the upper (unless you have very good reason to make the base machine inaccessible from the jails and vice-versa).

Some missing points…

# Make sure the bridged-interface is active
ifconfig em0 up

# Set promiscous mode and
# don't override src addr
ngctl msg em0: setpromisc 1
ngctl msg em0: setautosrc 0

ngctl connect ngeth0: bridge0: ether link2
ngctl connect ngeth1: bridge0: ether link3

Let's re-work this.

Earlier you did 2x "mkpeer" to pre-create your eiface nodes. I don't do this, as it later requires a "connect" for each created peer.

What you can do instead is _not_ perform those earlier commands (shown below):

ngctl mkpeer eiface ether ether
ngctl mkpeer eiface ether ether

(followed by the above "connect" statements)

But *instead*, you can actually do a contextual "mkpeer" that connects the nodes as they are created.

For example… instead of this:

ngctl mkpeer eiface ether ether
ngctl mkpeer eiface ether ether

ngctl connect ngeth0: bridge0: ether link2

ngctl connect ngeth0: bridge0: ether link3

(which doesn't look right to me, because I'm used to creating an ether edge between bridge:lower and the eiface)

You can instead do:

ngctl mkpeer em0:lower eiface link2 ether
ngctl mkpeer em0:lower eiface link3 ether

This simplifies 4 statements into 2.

NOTE: I can't remember if "em0:lower" can be replaced with the new name of "bridge0" (that you created with "ngctl name em0:lower bridge0"). My scripts continue to use the raw name in the mkpeer statements even after a rename of the lower-link. You *may* be able to instead say "ngctl mkpeer bridge0: eiface link2 ether" etc.)

vimage -i n1 ngeth0 e0

But my virtual interface on the n1 vimage does not receive any packet from the external network while I can see the packet go out from it.

Try hooking changing your PHY->bridge hook to the lower and hooking your PHY upper into the bridge.

For instance using DHCP, e0 on n1 sends DHCP packets but it does not receive the answers (which are send, I verified it from wireshark), in adding
the ARP request for his IP address (if I try to add it statically) are not received then it can not answer.

At the end of the line the question is: how can I make this "virtual network" and the external real network be able to communicate ?

Thanks in advance.

Wondering if you've given my solution a try…

http://druidbsd.sf.net/vimage.shtml
--
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Teske, Devin

unread,

May 30, 2013, 12:38:07 PM5/30/13

On May 30, 2013, at 9:18 AM, Joe wrote:

Pietro Paolini wrote:

freebsd-...@freebsd.org<mailto:freebsd-...@freebsd.org> mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org<mailto:freebsd-questi...@freebsd.org>"

I imagine that most vimage solutions do not do what mine does… (http://druidbsd.sourceforge.net/download.shtml#vimage)

Which is to actually give each bridged interface a unique MAC address.

This makes each vimage truly appear as a separate host. This makes DHCP from a vimage possible because the DHCP server sees discretely unique systems, through-and-through.

To the broader question, why do you think you need a vnet/vimage jail in the first place. You wrote "simulate more client for my PC", If client means hosting paying clients then you sure don't want to be using vimage because it highly experimental and NOT reliable.

The reports of vimages unreliability must be coupled with the feature-set you desire.

I say this because yesterday afternoon… we shipped our first-ever production platform to a customer -- based entirely on FreeBSD 8.1 + vimage + zfs + sysutils/zxfer + multipath + nullfs + nfs. No single point of failure.

We ran vimage in the lab for about 2.5 years before arriving at the point that we were ready to put it into production. (NOTE: FreeBSD-8.1 is about 36 months old).

So to say that it's not reliable enough for production… it depends on your definition of production. From the FreeBSD Foundation side… I'd say that this sounds about correct (because "production" means to the FreeBSD Foundation… able to work with all GENERIC features). However, in our case "production" means… able to work with the features *we* need.

The features VIMAGE doesn't work with today are PF and a few other things (I saw Joe post in a different reply that "nooptions SCTP" may not be required anymore… so progress seems to be afoot).

But with respect to stability and reliability… if you can get them to run… they're solid…

Just don't do any of the following:

1. Expose /dev/mem to the jail via a ruleset

2. Expose /dev/kmem (same problem as #1 … you could find yourself in a kernel panic)

NOTE: Both of the above are self-inflicted gunshot wounds. These devices are not exposed by default (and the only reason to do so is to try to get things like Xorg or "netstat -nr" to work within the vimage; side-note: "route -n get default" works).

Oh… and then there's the issue that when you stop a vimage, some KVA pages are supposedly lost.

Well… in true production, we don't make it a habit of stopping vimages (once they are up, they stay up). Usually the only reason for a vimage to go down is if the base machine goes down or you need to change the mounts… in either case, the KVA pages will be refreshed on a reboot of the base machine… so it's really not an issue to the way we run production.
--
Devin

Jails are a security feature that encloses a process in a container. By process I mean "postfix email server" or "apache web server". If your "client" means "processes", then this is what the non-vnet/vimage jail is for.

_______________________________________________

freebsd-...@freebsd.org<mailto:freebsd-...@freebsd.org> mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Pietro Paolini

unread,

May 31, 2013, 4:48:19 AM5/31/13

Many thanks for the answer Devin,
when I try to use that last command I receive:

ngctl connect em0: bridge0:lower upper link1

ngctl: send msg: Invalid argument

What's wrong ?

Pietro Paolini

unread,

May 31, 2013, 4:53:18 AM5/31/13

On May 30, 2013, at 6:38 PM, "Teske, Devin" <Devin...@fisglobal.com> wrote:

>
> On May 30, 2013, at 9:18 AM, Joe wrote:
>
>> Pietro Paolini wrote:

>>>> freebsd-...@freebsd.org mailing list

>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>> To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

>> freebsd-...@freebsd.org mailing list

>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"
>
> _____________
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

I am running a 9.1-RELEASE with VIMAGE option enabled, I don't see any problem with my DHCP server because I am in a custom env then DHCP is running on a machine than I configured.

I used both ways

http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf

or

http://bsdbased.com/2009/12/06/freebsd-8-vimage-epair-howto

All I wish is to connect my external network on the real ethernet interface to my virtual network stack (jail, visage, as you wish call it).

Teske, Devin

unread,

May 31, 2013, 6:05:39 PM5/31/13

On May 31, 2013, at 1:48 AM, Pietro Paolini wrote:

Let's start from scratch on a freshly booted box…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
[sudo] Password:
There are 4 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0
Name: em1 Type: ether ID: 00000003 Num hooks: 0
Name: ngctl1719 Type: socket ID: 00000004 Num hooks: 0
Name: msk0 Type: ether ID: 00000001 Num hooks: 0

Ok… we have an "ether" type node for each of our physical adapters (these are provided by ng_ether(4); you didn't have to do anything to get these nodes).

We also have a single "socket" type node. This is the "ngctl" connection to the netgraph subsystem (you can learn more by reading ng_socket(4)).

Here's the corresponding hardware behind em0, em1, and msk0:

===

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ grep '$em\|e1000phy\|mskc\?$[[:digit:]]' /var/run/dmesg.boot
mskc0: <Marvell Yukon 88E8050 Gigabit Ethernet> port 0xdc00-0xdcff mem 0xfcffc000-0xfcffffff irq 16 at device 0.0 on pci5
msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0
msk0: Ethernet address: xx:xx:xx:xx:xx:xx
miibus0: <MII bus> on msk0
e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0
e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
mskc0: [ITHREAD]
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec80-0xecbf mem 0xfebe0000-0xfebfffff irq 16 at device 4.0 on pci7
em0: [FILTER]
em0: Ethernet address: xx:xx:xx:xx:xx:xx
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec00-0xec3f mem 0xfeba0000-0xfebbffff,0xfeb80000-0xfeb9ffff irq 18 at device 6.0 on pci7
em1: [FILTER]
em1: Ethernet address: xx:xx:xx:xx:xx:xx
em0: link state changed to UP

===

Next, let's make a bridge (think of it as a big software switch that we're going to hook a bunch of interfaces; created, physical, or otherwise).

Since I'm doing this over an SSH connection (a mistake I made earlier today), I'm not going to touch em0 (the adapter my SSH connection is using). Creating the bridge on an actively configured PHY will knock it off the net. This is not to say you can't have an active configuration on a bridged interface… just that the creation of the bridge (something you should only do once each time you boot) will disrupt an active connection.

So…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl mkpeer em1: bridge lower link0

NOTE: No output == Success.

===

Now let's look at our handiwork…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl info em1:lower
Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link0 em1 ether 00000003 lower

Ok, we see that the lower peer hook of the em1 ether-node goes off to something named "link0".

To see where link0 is off-to… we need a full listing (back to "ngctl ls -l").

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 5 total nodes:
Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link0 em1 ether 00000003 lower
Name: em0 Type: ether ID: 00000002 Num hooks: 0
Name: em1 Type: ether ID: 00000003 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
lower <unnamed> bridge 00000007 link0
Name: ngctl1762 Type: socket ID: 0000000b Num hooks: 0
Name: msk0 Type: ether ID: 00000001 Num hooks: 0

Matching "link0" in the first column to "link0" in the last-column, we can see that this lower-link0 is to a bridge (with no name).

NOTE: When you're digesting the above output… it helps to imagine whitespace in between the nodes with their respective hooks and other nodes. Future pastes below will introduce such whitespace to make it easier to read.

===

Right now, the only way to refer to the bridge is by way of "em1:lower" (because we created the bridge right on the lower hook of the em1 ether-node).

At this point, let's talk about naming. Giving our bridge a name is entirely optional, but greatly clarifies the output of both "ngctl ls -l" and "ngctl dot".

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl name em1:lower em1bridge
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 5 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0

Name: em1 Type: ether ID: 00000003 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
lower em1bridge bridge 00000007 link0

Name: ngctl1831 Type: socket ID: 0000001a Num hooks: 0

Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link0 em1 ether 00000003 lower

Name: msk0 Type: ether ID: 00000001 Num hooks: 0

The new "em1bridge" name acts as an alias to "em1:lower" in future ngctl commands. For example, "ngctl info em1:lower" and "ngctl info em1bridge" can now be used interchangeably and produce the same results.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl info em1bridge:
Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link0 em1 ether 00000003 lower
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl info em1:lower
Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link0 em1 ether 00000003 lower

===

We're not done with the bridge yet. Because we foresee the possibility that it might be nice to be able to communicate with the jail that we're going to later hook into this bridge… we should hook the physical adapter's "upper" hook into the bridge.

If you don't do this, you won't be able to (for example) ping a jail from the host where the host has only the PHY and the jail has only a (yet uncreated) eiface. Regardless of the fact that the bridge uses the PHY and the jail uses the bridge, to communicate with an IP that is configured on the base host, you must hook the upper.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl connect em1: em1:lower upper link1

If you want to use the alias I set up earlier (of "em1bridge") that works too (just don't forget the colon at the end of the alias):

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl connect em1: em1bridge: upper link1

Here's the results:

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 5 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0

Name: em1 Type: ether ID: 00000003 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
upper em1bridge bridge 0000002a link1
lower em1bridge bridge 0000002a link0

Name: ngctl1874 Type: socket ID: 00000030 Num hooks: 0

Name: em1bridge Type: bridge ID: 0000002a Num hooks: 2
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link1 em1 ether 00000003 upper
link0 em1 ether 00000003 lower

Name: msk0 Type: ether ID: 00000001 Num hooks: 0

NOTE: Some of the Peer ID's have changed, because I wanted to test that the alias could be used; I used "sudo ngctl shutdown em1bridge:" and re-executed up to the point where I connect the em1:upper into the bridge… except this time using the alias of "em1bridge" instead of "em1:lower" (indeed, you can use them interchangeably).

===

Ok… We've now done the hard part… which was to create and configure a bridge that is usable by any new nodes we connect to it and also (if you hooked the upper portion of em1 back into its own lower which is acting as the bridge) the base machine can communicate with any of the forth-coming jails (if on the same subnet at least).

There's an easy step that shouldn't be skipped though…

Before you can truly use this bridge with any other interfaces…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ifconfig em1 up
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1: setpromisc 1
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1: setautosrc 0

A bridge cannot send packets out if the interface is down.
A bridge cannot work properly without promiscuous mode.
A bridge cannot send out packets for different addresses unless you turn off "setautosrc"

===

Let's create our first virtual NIC and connect it to the bridge.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl mkpeer em1bridge: eiface link2 ether

This command did two things. It created a new "eiface" node (see ng_eiface(4)), and connected it to the bridge.

Let's have a look:

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 6 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0

Name: em1 Type: ether ID: 00000003 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
upper em1bridge bridge 0000002a link1
lower em1bridge bridge 0000002a link0

Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
ether em1bridge bridge 0000002a link2

Name: ngctl2800 Type: socket ID: 00000036 Num hooks: 0

Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link2 ngeth0 eiface 00000035 ether
link1 em1 ether 00000003 upper
link0 em1 ether 00000003 lower

Name: msk0 Type: ether ID: 00000001 Num hooks: 0

The list of hooks for our bridge (em1bridge) is growing, and now we see a new node (ngeth0) with one hook into that bridge.

===

ASIDE: If you wanted to script this… here's how you can test for an unused link:

Right now, we have link0, link1, and link2 for the bridge. If a link exists for a bridge, the following command will return some info about the link and return success (whereas if the link does not exist, the command will return an error and exit with error-status):

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 0
Rec'd response "getstats" (4) from "[2a]:":
Args: {}
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 1
Rec'd response "getstats" (4) from "[2a]:":
Args: {}
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 2
Rec'd response "getstats" (4) from "[2a]:":
Args: {}
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 3
ngctl: send msg: Socket is not connected
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 4
ngctl: send msg: Socket is not connected
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl msg em1bridge: getstats 5
ngctl: send msg: Socket is not connected

As you can see from the above output… we get errors for link3, link4, and link5, because they don't exist. Naturally, testing $? exit status after each of these commands would show how this can be scripted (HINT: throw stdout/stderr to /dev/null and test $?).

===

At this point… you say "ifconfig":

dte...@oos0a.lbxrich.vicor.com<mailto:dte...@oos0a.lbxrich.vicor.com> ~ $ ifconfig
msk0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
ether xx:xx:xx:xx:xx:xx
media: Ethernet autoselect
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether xx:xx:xx:xx:xx:xx
inet xx.xx.xx.xx netmask 0xffffff80 broadcast xx.xx.xx.xx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether xx:xx:xx:xx:xx:xx
media: Ethernet autoselect
status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000
ngeth0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:00:00:00:00:00

===

Ok, there are two problems with the network interface.

1. It has a NULL MAC address (00:00:00:00:00:00). Good luck communicating on the Internet (remember, we disabled setautosrc -- we intend to make up a MAC address that is unique).

2. The name leaves something to be desired (if we're going to use this with a vimage jail, it would be nice if the interface had the jail name in it, so that when you do an "ngctl ls -l" or an "ngctl dot" … you're going to see the jail name so it becomes clear which jails are hooked to which PHY's through which bridges).

===

Let's tackle the easier one first… let's rename this new interface.

You and I already know that this interface that we want to rename is "ngeth0"… but you can actually extract the name from the link in the bridge.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl show -n em1bridge:link2
Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1

First, we rename it in netgraph (this does not affect the output of ifconfig -- and again, we do this to make "ngctl ls -l" and "ngctl dot" more palatable):

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl name em1bridge:link2 ng0_myjail
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 6 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0

Name: em1 Type: ether ID: 00000003 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
upper em1bridge bridge 0000002a link1
lower em1bridge bridge 0000002a link0

Name: ngctl2843 Type: socket ID: 00000046 Num hooks: 0

Name: ng0_myjail Type: eiface ID: 00000035 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
ether em1bridge bridge 0000002a link2

Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link2 ng0_myjail eiface 00000035 ether
link1 em1 ether 00000003 upper
link0 em1 ether 00000003 lower

Name: msk0 Type: ether ID: 00000001 Num hooks: 0

Looking good. However, ifconfig hasn't changed…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ ifconfig
...
ngeth0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:00:00:00:00:00

We want to rename the interface with ifconfig for a different reason.

We renamed the interface with netgraph earlier so that netgraph outputs would be nice and easy to digest.

This time, we rename with ifconfig so that we can layer jails onto the same rootdir.

The naming convention (which is the same naming convention I use for renaming on the netgraph side) is:

ng#_name

The # always starts at zero for each jail where "name" is the name of the jail.

Again… I use this scheme so that I can layer jails onto the same root-dir; /etc/rc.conf is then populated with things like:

ifconfig_ng0_myjail=...
ifconfig_ng0_myrouter=...
ifconfig_ng1_myrouter=...
ifconfig_ng0_anotherjail=...

So that when you say "service netif start" inside the vnet jail… it applies the right settings.

So… we rename with ifconfig:

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ifconfig ngeth0 name ng0_myjail
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ ifconfig
...
ng0_myjail: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:00:00:00:00:00

===

We're almost ready to shove this interface into a jail (which we haven't created yet).

But… we come back to that NULL MAC address.

NOTE: Forming your own MAC address, or even coming up with your own formula should not be taken lightly.

Here's a formula I use (which is based on several RFC's for MAC address formation):

NOTE: In this context, ${_bridge} is em1 and $LINKNUM is 2

# Set the MAC address of the new interface
# using a sensible algorithm to prevent
# conflicts on the network.
#
# MAC LP:LL:LB:BB:BB:BB
# P 2, 6, A, or E but usually 2
# NOTE: Indicates "privately administered" MAC
# L ng_bridge(4) link number (1-65535)
# B Same as bridged interface
#
_bridge_ether=$( ifconfig ${_bridge} ether |
awk '/ether/{print $2}' )
_ether_devid="${_bridge_ether#??:??:?}"
n=$LINKNUM
_quad=$(($n & 15))
case "${_quad}" in
10) _quad=a;; 11) _quad=b;; 12) _quad=c;;
13) _quad=d;; 14) _quad=e;; 15) _quad=f;;
esac
_ether_devid=":${_quad}${_ether_devid}"
n=$(($n >> 4))
_quad=$(($n & 15))
case "${_quad}" in
10) _quad=a;; 11) _quad=b;; 12) _quad=c;;
13) _quad=d;; 14) _quad=e;; 15) _quad=f;;
esac
_ether_devid="${_quad}${_ether_devid}"
n=$(($n >> 4))
_quad=$(($n & 15))
case "${_quad}" in
10) _quad=a;; 11) _quad=b;; 12) _quad=c;;
13) _quad=d;; 14) _quad=e;; 15) _quad=f;;
esac
_ether_devid="2:${_quad}${_ether_devid}"
n=$(($n >> 4))
_quad=$(($n & 15))
case "${_quad}" in
10) _quad=a;; 11) _quad=b;; 12) _quad=c;;
13) _quad=d;; 14) _quad=e;; 15) _quad=f;;
esac
_ether_devid="${_quad}${_ether_devid}"
n=$(($n >> 4))

After which… ${_ether_devid} holds a properly formed MAC address that can (in every case I've tested) "get out".

Here's what I do to set it:

ifconfig ng0_myjail ether "${_ether_devid}"

Here's an example of how the MAC address was translated from the physical adapter to the ng_eiface(4) interface:

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ ifconfig em1; ifconfig ng0_myjail
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:0e:0c:ab:1b:76
media: Ethernet autoselect
status: no carrier
ng0_myjail: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:00:2c:ab:1b:76

===

OK… we're now ready to shove that interface into a vimage jail.

But…

First we need a vimage jail. (this is not a tutorial on how to create, manage, build, or do anything else with jails, vimage-jails, or vps-jails *other* than give it a netgraph based interface)

I'm going to use my existing base machine as a fake jail (by pointing my jail's rootdir at "/").

NOTE: Certain sysctl's have to be set appropriately before you fire up the jail to make this vimage jail able to do "more" on the net.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo sysctl security.jail.set_hostname_allowed=1 security.jail.sysvipc_allowed=1 security.jail.socket_unixiproute_only=1
security.jail.set_hostname_allowed: 1 -> 1
security.jail.sysvipc_allowed: 1 -> 1
security.jail.socket_unixiproute_only: 0 -> 1

NOTE: Unless you intend to reboot to restore the defaults later… you might want to take down those previous values for restoration *after* we fire up the "vimage" jail.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo jail -i -c vnet name=myjail host.hostname=myjail path=/ persist
1
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ jls
JID IP Address Hostname Path
1 - myjail /

OK… we have a running jail (with the vnet property, making it a "vimage" jail -- which can accept network interfaces).

===

Right now our jail has no network interfaces (well, it has an unconfigured lo0).

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo jexec myjail ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>

So let's pass the netgraph created interface into the jail…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ifconfig ng0_myjail vnet 1
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo jexec myjail ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
ng0_myjail: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:00:2c:ab:1b:76

Sweet!

===

Almost there…

Let's go into /etc/rc.conf, give it an IP, and start the network…

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo sysrc ifconfig_ng0_myjail="inet 192.168.1.1 netmask 255.255.255.0"
/etc/rc.conf: ifconfig_ng0_myjail: -> inet 192.168.1.1 netmask 255.255.255.0
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ grep ng0 /etc/rc.conf
ifconfig_ng0_myjail="inet 192.168.1.1 netmask 255.255.255.0"
dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo jexec myjail service netif start
Starting Network: lo0 ng0_myjail.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000
ng0_myjail: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:00:2c:ab:1b:76
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255

Now we're cookin' with gasoline!

===

Optionally go configure your base machine with an IP and have fun.

--
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Teske, Devin

unread,

May 31, 2013, 11:26:16 PM5/31/13

A quick conclusion…

Because we've built this all on top of netgraph… we can … graph it.

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl dot | dot -Tsvg -o netgraph-scu0a.svg

I then uploaded the file to the web and here it is:

http://druidbsd.sourceforge.net/download/netgraph-scu0a.svg

You should compare this directly to the output of "ngctl ls -l":

dte...@scu0a.jbsd.vicor.com<mailto:dte...@scu0a.jbsd.vicor.com> ~ $ sudo ngctl ls -l
There are 6 total nodes:
Name: em0 Type: ether ID: 00000002 Num hooks: 0

Name: em1 Type: ether ID: 00000003 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
upper em1bridge bridge 0000002a link1
lower em1bridge bridge 0000002a link0

Name: ng0_myjail Type: eiface ID: 00000035 Num hooks: 1
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
ether em1bridge bridge 0000002a link2

Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
link2 ng0_myjail eiface 00000035 ether
link1 em1 ether 00000003 upper
link0 em1 ether 00000003 lower

Name: ngctl8676 Type: socket ID: 00000049 Num hooks: 0

Name: msk0 Type: ether ID: 00000001 Num hooks: 0

You'll notice that when you graph the layout with "ngctl dot", the nodes are rendered as boxes displaying their "Peer Name" up top, their "Peer Type" in the lower-left, and their "Peer ID" in the bottom-right.

The edges from one node to another contains two octagons. These are the "Local hook" and "Peer hook".

Pietro Paolini

unread,

Jun 3, 2013, 9:33:48 AM6/3/13

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> [sudo] Password:
>> There are 4 total nodes:
>> Name: em0 Type: ether ID: 00000002 Num hooks: 0
>> Name: em1 Type: ether ID: 00000003 Num hooks: 0
>> Name: ngctl1719 Type: socket ID: 00000004 Num hooks: 0
>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0
>>
>> Ok… we have an "ether" type node for each of our physical adapters (these are provided by ng_ether(4); you didn't have to do anything to get these nodes).
>>
>> We also have a single "socket" type node. This is the "ngctl" connection to the netgraph subsystem (you can learn more by reading ng_socket(4)).
>>
>> Here's the corresponding hardware behind em0, em1, and msk0:
>>
>> ===
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ grep '$em\|e1000phy\|mskc\?$[[:digit:]]' /var/run/dmesg.boot

>> mskc0: <Marvell Yukon 88E8050 Gigabit Ethernet> port 0xdc00-0xdcff mem 0xfcffc000-0xfcffffff irq 16 at device 0.0 on pci5
>> msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0
>> msk0: Ethernet address: xx:xx:xx:xx:xx:xx
>> miibus0: <MII bus> on msk0
>> e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0
>> e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
>> mskc0: [ITHREAD]
>> em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec80-0xecbf mem 0xfebe0000-0xfebfffff irq 16 at device 4.0 on pci7
>> em0: [FILTER]
>> em0: Ethernet address: xx:xx:xx:xx:xx:xx
>> em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec00-0xec3f mem 0xfeba0000-0xfebbffff,0xfeb80000-0xfeb9ffff irq 18 at device 6.0 on pci7
>> em1: [FILTER]
>> em1: Ethernet address: xx:xx:xx:xx:xx:xx
>> em0: link state changed to UP
>>
>> ===
>>
>> Next, let's make a bridge (think of it as a big software switch that we're going to hook a bunch of interfaces; created, physical, or otherwise).
>>
>> Since I'm doing this over an SSH connection (a mistake I made earlier today), I'm not going to touch em0 (the adapter my SSH connection is using). Creating the bridge on an actively configured PHY will knock it off the net. This is not to say you can't have an active configuration on a bridged interface… just that the creation of the bridge (something you should only do once each time you boot) will disrupt an active connection.
>>
>> So…
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1: bridge lower link0

>>
>> NOTE: No output == Success.
>>
>> ===
>>
>> Now let's look at our handiwork…
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower

>> Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link0 em1 ether 00000003 lower
>>
>>
>> Ok, we see that the lower peer hook of the em1 ether-node goes off to something named "link0".
>>
>> To see where link0 is off-to… we need a full listing (back to "ngctl ls -l").
>>
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> There are 5 total nodes:
>> Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link0 em1 ether 00000003 lower
>> Name: em0 Type: ether ID: 00000002 Num hooks: 0
>> Name: em1 Type: ether ID: 00000003 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> lower <unnamed> bridge 00000007 link0
>> Name: ngctl1762 Type: socket ID: 0000000b Num hooks: 0
>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0
>>
>>
>> Matching "link0" in the first column to "link0" in the last-column, we can see that this lower-link0 is to a bridge (with no name).
>>
>> NOTE: When you're digesting the above output… it helps to imagine whitespace in between the nodes with their respective hooks and other nodes. Future pastes below will introduce such whitespace to make it easier to read.
>>
>> ===
>>
>> Right now, the only way to refer to the bridge is by way of "em1:lower" (because we created the bridge right on the lower hook of the em1 ether-node).
>>
>> At this point, let's talk about naming. Giving our bridge a name is entirely optional, but greatly clarifies the output of both "ngctl ls -l" and "ngctl dot".
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1:lower em1bridge
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> There are 5 total nodes:
>> Name: em0 Type: ether ID: 00000002 Num hooks: 0
>>
>> Name: em1 Type: ether ID: 00000003 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> lower em1bridge bridge 00000007 link0
>>
>> Name: ngctl1831 Type: socket ID: 0000001a Num hooks: 0
>>
>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link0 em1 ether 00000003 lower
>>
>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0
>>
>> The new "em1bridge" name acts as an alias to "em1:lower" in future ngctl commands. For example, "ngctl info em1:lower" and "ngctl info em1bridge" can now be used interchangeably and produce the same results.
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1bridge:

>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link0 em1 ether 00000003 lower

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower

>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link0 em1 ether 00000003 lower
>>
>> ===
>>
>> We're not done with the bridge yet. Because we foresee the possibility that it might be nice to be able to communicate with the jail that we're going to later hook into this bridge… we should hook the physical adapter's "upper" hook into the bridge.
>>
>> If you don't do this, you won't be able to (for example) ping a jail from the host where the host has only the PHY and the jail has only a (yet uncreated) eiface. Regardless of the fact that the bridge uses the PHY and the jail uses the bridge, to communicate with an IP that is configured on the base host, you must hook the upper.
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1:lower upper link1

>>
>> If you want to use the alias I set up earlier (of "em1bridge") that works too (just don't forget the colon at the end of the alias):
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1bridge: upper link1
>>
>> Here's the results:
>>
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> There are 5 total nodes:
>> Name: em0 Type: ether ID: 00000002 Num hooks: 0
>>
>> Name: em1 Type: ether ID: 00000003 Num hooks: 2
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> upper em1bridge bridge 0000002a link1
>> lower em1bridge bridge 0000002a link0
>>
>> Name: ngctl1874 Type: socket ID: 00000030 Num hooks: 0
>>
>> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 2
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link1 em1 ether 00000003 upper
>> link0 em1 ether 00000003 lower
>>
>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0
>>
>>
>> NOTE: Some of the Peer ID's have changed, because I wanted to test that the alias could be used; I used "sudo ngctl shutdown em1bridge:" and re-executed up to the point where I connect the em1:upper into the bridge… except this time using the alias of "em1bridge" instead of "em1:lower" (indeed, you can use them interchangeably).
>>
>> ===
>>
>> Ok… We've now done the hard part… which was to create and configure a bridge that is usable by any new nodes we connect to it and also (if you hooked the upper portion of em1 back into its own lower which is acting as the bridge) the base machine can communicate with any of the forth-coming jails (if on the same subnet at least).
>>
>> There's an easy step that shouldn't be skipped though…
>>
>> Before you can truly use this bridge with any other interfaces…
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ifconfig em1 up
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setpromisc 1
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setautosrc 0

>>
>> A bridge cannot send packets out if the interface is down.
>> A bridge cannot work properly without promiscuous mode.
>> A bridge cannot send out packets for different addresses unless you turn off "setautosrc"
>>
>> ===
>>
>> Let's create our first virtual NIC and connect it to the bridge.
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1bridge: eiface link2 ether

>>
>> This command did two things. It created a new "eiface" node (see ng_eiface(4)), and connected it to the bridge.
>>
>> Let's have a look:
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> There are 6 total nodes:
>> Name: em0 Type: ether ID: 00000002 Num hooks: 0
>>
>> Name: em1 Type: ether ID: 00000003 Num hooks: 2
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> upper em1bridge bridge 0000002a link1
>> lower em1bridge bridge 0000002a link0
>>
>> Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> ether em1bridge bridge 0000002a link2
>>
>> Name: ngctl2800 Type: socket ID: 00000036 Num hooks: 0
>>
>> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3
>> Local hook Peer name Peer type Peer ID Peer hook
>> ---------- --------- --------- ------- ---------
>> link2 ngeth0 eiface 00000035 ether
>> link1 em1 ether 00000003 upper
>> link0 em1 ether 00000003 lower
>>
>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0
>>
>> The list of hooks for our bridge (em1bridge) is growing, and now we see a new node (ngeth0) with one hook into that bridge.
>>
>> ===
>>
>> ASIDE: If you wanted to script this… here's how you can test for an unused link:
>>
>> Right now, we have link0, link1, and link2 for the bridge. If a link exists for a bridge, the following command will return some info about the link and return success (whereas if the link does not exist, the command will return an error and exit with error-status):
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 0

>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 1

>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 2

>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 3

>> ngctl: send msg: Socket is not connected

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 4

>> ngctl: send msg: Socket is not connected

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 5

>> ngctl: send msg: Socket is not connected
>>
>> As you can see from the above output… we get errors for link3, link4, and link5, because they don't exist. Naturally, testing $? exit status after each of these commands would show how this can be scripted (HINT: throw stdout/stderr to /dev/null and test $?).
>>
>> ===
>>
>> At this point… you say "ifconfig":
>>

>> dte...@oos0a.lbxrich.vicor.com ~ $ ifconfig

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl show -n em1bridge:link2

>> Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1
>>
>>
>> First, we rename it in netgraph (this does not affect the output of ifconfig -- and again, we do this to make "ngctl ls -l" and "ngctl dot" more palatable):
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1bridge:link2 ng0_myjail
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

>> dte...@scu0a.jbsd.vicor.com ~ $ ifconfig

>> ...
>> ngeth0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ether 00:00:00:00:00:00
>>
>> We want to rename the interface with ifconfig for a different reason.
>>
>> We renamed the interface with netgraph earlier so that netgraph outputs would be nice and easy to digest.
>>
>> This time, we rename with ifconfig so that we can layer jails onto the same rootdir.
>>
>> The naming convention (which is the same naming convention I use for renaming on the netgraph side) is:
>>
>> ng#_name
>>
>> The # always starts at zero for each jail where "name" is the name of the jail.
>>
>> Again… I use this scheme so that I can layer jails onto the same root-dir; /etc/rc.conf is then populated with things like:
>>
>> ifconfig_ng0_myjail=...
>> ifconfig_ng0_myrouter=...
>> ifconfig_ng1_myrouter=...
>> ifconfig_ng0_anotherjail=...
>>
>> So that when you say "service netif start" inside the vnet jail… it applies the right settings.
>>
>> So… we rename with ifconfig:
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ngeth0 name ng0_myjail
>> dte...@scu0a.jbsd.vicor.com ~ $ ifconfig

>> dte...@scu0a.jbsd.vicor.com ~ $ ifconfig em1; ifconfig ng0_myjail

>> em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
>> ether 00:0e:0c:ab:1b:76
>> media: Ethernet autoselect
>> status: no carrier
>> ng0_myjail: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ether 02:00:2c:ab:1b:76
>>
>> ===
>>
>> OK… we're now ready to shove that interface into a vimage jail.
>>
>> But…
>>
>> First we need a vimage jail. (this is not a tutorial on how to create, manage, build, or do anything else with jails, vimage-jails, or vps-jails *other* than give it a netgraph based interface)
>>
>> I'm going to use my existing base machine as a fake jail (by pointing my jail's rootdir at "/").
>>
>> NOTE: Certain sysctl's have to be set appropriately before you fire up the jail to make this vimage jail able to do "more" on the net.
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo sysctl security.jail.set_hostname_allowed=1 security.jail.sysvipc_allowed=1 security.jail.socket_unixiproute_only=1

>> security.jail.set_hostname_allowed: 1 -> 1
>> security.jail.sysvipc_allowed: 1 -> 1
>> security.jail.socket_unixiproute_only: 0 -> 1
>>
>> NOTE: Unless you intend to reboot to restore the defaults later… you might want to take down those previous values for restoration *after* we fire up the "vimage" jail.
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo jail -i -c vnet name=myjail host.hostname=myjail path=/ persist
>> 1
>> dte...@scu0a.jbsd.vicor.com ~ $ jls

>> JID IP Address Hostname Path
>> 1 - myjail /
>>
>> OK… we have a running jail (with the vnet property, making it a "vimage" jail -- which can accept network interfaces).
>>
>> ===
>>
>> Right now our jail has no network interfaces (well, it has an unconfigured lo0).
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig

>> lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
>> options=3<RXCSUM,TXCSUM>
>>
>> So let's pass the netgraph created interface into the jail…
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ng0_myjail vnet 1
>> dte...@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig

>> lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
>> options=3<RXCSUM,TXCSUM>
>> ng0_myjail: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ether 02:00:2c:ab:1b:76
>>
>> Sweet!
>>
>> ===
>>
>> Almost there…
>>
>> Let's go into /etc/rc.conf, give it an IP, and start the network…
>>

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo sysrc ifconfig_ng0_myjail="inet 192.168.1.1 netmask 255.255.255.0"

>> /etc/rc.conf: ifconfig_ng0_myjail: -> inet 192.168.1.1 netmask 255.255.255.0

>> dte...@scu0a.jbsd.vicor.com ~ $ grep ng0 /etc/rc.conf

>> ifconfig_ng0_myjail="inet 192.168.1.1 netmask 255.255.255.0"

>> dte...@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail service netif start

>> Starting Network: lo0 ng0_myjail.
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> options=3<RXCSUM,TXCSUM>
>> inet 127.0.0.1 netmask 0xff000000
>> ng0_myjail: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ether 02:00:2c:ab:1b:76
>> inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
>>
>> Now we're cookin' with gasoline!
>>
>> ===
>>
>> Optionally go configure your base machine with an IP and have fun.
>
> A quick conclusion…
>
> Because we've built this all on top of netgraph… we can … graph it.
>

> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl dot | dot -Tsvg -o netgraph-scu0a.svg

>
> I then uploaded the file to the web and here it is:
>
> http://druidbsd.sourceforge.net/download/netgraph-scu0a.svg
>
> You should compare this directly to the output of "ngctl ls -l":
>

> dte...@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l

Hello Devin,

If you live in the same city I will invite you for a couple of beer (I have to pay of course!) - I live in the Netherlands then let me know -

I followed your tutorial with the expected result: I can ping the em1 interface but I still have the same problems of before regarding external to internal networks communication.

Please note that on my original host (no jail) the default gateway is 192.168.1.254, that's important for what I am going to do.

I added a default route like :

route add default -interface ng0_myjail

and then I try to :

jexec myjail ping 8.8.8.8

I analyzed the wireshark capture and I can see that an ARP request for the 192.168.254 (with the MAC address of our virtual NIC as source, as expected) go out my freebsd env - which is on a virtualbox - and it gets the answer but when I read the ARP table of the jail I can see it as incomplete. It seems that packet going out but the answer is not received and it is confirmed when I try to sniff with tcpdump on my FREEBSD, I can't see any ARP request going in both from my physical and virtual NIC, the same if I try :

ping 192.168.1.254

Then I can see ping reply coming from wireshark but not from tcpdump on FreeBDS.

Wireshark is attached on the host machine on the physical interface where VB is attached in Bridged Mode, my original physical interface on FreeBSD took the IP address from DHCP without problem then the problem is related to the jail.

do I try do accomplish a task which is not possible with JAIL or something is wrong in my configuration or worst, in my brain :P ?

Thanks in advance,
Pietro.

Teske, Devin

unread,

Jun 3, 2013, 12:31:26 PM6/3/13

On Jun 3, 2013, at 6:33 AM, Pietro Paolini wrote:

>
> On Jun 1, 2013, at 5:26 AM, "Teske, Devin" <Devin...@fisglobal.com> wrote:
>
>>
>> On May 31, 2013, at 3:05 PM, Teske, Devin wrote:
>>
>>>
>>> On May 31, 2013, at 1:48 AM, Pietro Paolini wrote:
>>>
>>>>
>>>> On May 30, 2013, at 6:25 PM, "Teske, Devin" <Devin...@fisglobal.com> wrote:
>>>>
>>>>>
>>>>> On May 30, 2013, at 3:35 AM, Pietro Paolini wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> I am a new bye on the FreeBSD and I am looking at the VIMAGE features experiencing some problems.
>>>>>> I added the options :
>>>>>> VIMAGE
>>>>>> if_bridge
>>>>>>
>>>>>> and I removed
>>>>>> STCP
>>>>>>
>>>>>> then I recompiled my kernel and install it.
>>>>>>

>>>>>> After that, following this tutorial https://urldefense.proofpoint.com/v1/url?u=http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=xe0XNgnKBiT9v8HzxwWwnNMOVN3YdEHmTsIZfFoQA9Y%3D%0A&s=b271820faf310ac274ded8c3135b4931f2a5278e78ec1a66ce6a5ab971ff64f5 I tried the "Exercise 2" which consist on

>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.sourceforge.net/download/netgraph-scu0a.svg&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=xe0XNgnKBiT9v8HzxwWwnNMOVN3YdEHmTsIZfFoQA9Y%3D%0A&s=0a6ee990c4871ecc2a5b69879d11f65798fe70d8948fb9d4592bd915b1fd0199

If I ever make it out there, I'll certainly shoot you an e-mail.

> I followed your tutorial with the expected result: I can ping the em1 interface but I still have the same problems of before regarding external to internal networks communication.
>
> Please note that on my original host (no jail) the default gateway is 192.168.1.254, that's important for what I am going to do.
>
> I added a default route like :
>
> route add default -interface ng0_myjail
>

Why not:

jexec myjail route add -net default 192.168.1.254

??
--
Devin

> and then I try to :
>
> jexec myjail ping 8.8.8.8
>
> I analyzed the wireshark capture and I can see that an ARP request for the 192.168.254 (with the MAC address of our virtual NIC as source, as expected) go out my freebsd env - which is on a virtualbox - and it gets the answer but when I read the ARP table of the jail I can see it as incomplete. It seems that packet going out but the answer is not received and it is confirmed when I try to sniff with tcpdump on my FREEBSD, I can't see any ARP request going in both from my physical and virtual NIC, the same if I try :
>
> ping 192.168.1.254
>
> Then I can see ping reply coming from wireshark but not from tcpdump on FreeBDS.
>
> Wireshark is attached on the host machine on the physical interface where VB is attached in Bridged Mode, my original physical interface on FreeBSD took the IP address from DHCP without problem then the problem is related to the jail.
>
> do I try do accomplish a task which is not possible with JAIL or something is wrong in my configuration or worst, in my brain :P ?
>
> Thanks in advance,
> Pietro.
>
>
>
>
>
>

0 new messages