netgraph network setup for jail(8) vnet jails.

Joe

unread,

May 18, 2013, 11:38:18 AM5/18/13

to

Hello list

I cant get to the internet using this netgraph setup script.
I sure would appreciate giving this console log a look over for
errors. My netgraph knowledge level is not sufficient to see what is
wrong. The goal is to run this script to setup and break down a netgraph
network for a single vnet jail at a time. rl0 is the real nic interface
device name of the nic facing the internet. This box is on my lan and
the gateway box does NAT for all lan boxes. The host running this script
can ping the internet ok.

Thank you very much for your help.

The host's kernel has modules with vimage & ipfw compiled in.

From the host
# /root >ifconfig
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
options=2008<VLAN_MTU,WOL_MAGIC>
ether 00:0c:6e:09:8b:74
inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The jails config file
# /root >cat /usr/local/etc/vnet/vdir4
vdir4 {
host.hostname = "vdir4";
path = "/usr/jails/vdir4";
mount.fstab = "/usr/local/etc/fstab/vdir4";
vnet;
persist;
}

The netgraph script
# /root >cat /usr/local/bin/vnet.ng.test
#!/bin/sh
# snip comments for displaying here
# This script is based on this /usr/share/examples/netgraph/virtual.lan

# Give the name of ethernet interface.
ETHER_INTF="rl0"

# List the names of virtual nodes and their IP addresses. Use ':'
# character to separate node name from node IP address and netmask.

#TARGET_TOPOLOGY="c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24"
TARGET_TOPOLOGY="vdir4|10.0.2.20/24"

# MAC manufacturer prefix. This can be modified according to needs.
MAC_PREFIX="00:1d:92"

# Temporary file is important for proper execution of script.
TEMP_FILE="/var/tmp/virtual.lan.tmp"

virtual_lan_start() {

# Load netgraph KLD's as necessary.

for KLD in ng_ether ng_bridge ng_eiface; do
if ! kldstat -v | grep -qw ${KLD}; then
echo -n "Loading ${KLD}.ko... "
kldload ${KLD} || exit 1
echo "done"
fi
done

# Reset all interfaces and jails. If temporary file can not be found
# script assumes that there is no previous configuration.

if [ ! -e ${TEMP_FILE} ]; then
echo "No previous configuration(${TEMP_FILE}) found to clean-up."
else
echo -n "Cleaning previous configuration..."
virtual_lan_stop
echo "done"
fi

# Create temporary file for usage. This file includes generated
# interface names and jail names. All bridges, interfaces and jails
# are written to file while created. In clean-up process written
# objects are cleaned (i.e. removed) from system.

if [ -e ${TEMP_FILE} ]; then
touch ${TEMP_FILE}
fi

echo -n "Verifying ethernet interface existence..."
# Verify ethernet interface exist.
if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then
echo "Error: interface ${ETHER_INTF} does not exist"
exit 1
fi

ifconfig ${ETHER_INTF} up || exit 1
echo "done"

# Get current number of bridge interfaces in the system. This number
# is used to create a name for new bridge.
BRIDGE_COUNT=`ngctl l | grep bridge | wc -l | sed -e "s/ //g"`
BRIDGE_NAME="bridge${BRIDGE_COUNT}"

# Create new ng_bridge(4) node and attach it to the ethernet interface.
# Connect ng_ether:lower hook to bridge:link0 when creating bridge and
# connect ng_ether:upper hook to bridge:link1 after bridge name is set.

echo "Creating bridge interface: ${BRIDGE_NAME}..."
ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1
ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1
ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1
echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected."

# In the above code block two hooks are connected to bridge interface,
# therefore LINKNUM is set to 2 indicating total number of connected
# hooks on the bridge interface.
LINKNUM=2

# Write name of the bridge to temp file. Clean-up procedure will use
# this name to shutdown bridge interface.
echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE}

# Attach vnet jail.
for NODE in ${TARGET_TOPOLOGY}; do

# Virtual nodes are defined in TARGET_TOPOLOGY variable. They
# have the form of 'nodeName|IPaddr'. Below two lines split
# node definition to get node name and node IP.

NODE_NAME=`echo ${NODE} | awk -F"|" '{print $1}'`
NODE_IP=`echo ${NODE} | awk -F"|" '{print $2}'`

# Create virtual node (jail) with given name
echo -n "Creating virtual node (jail) ${NODE_NAME}..."

jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME}
if [ $? -ne 0 ]; then
echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}."
virtual_lan_stop
exit 2
fi
echo "done"

# Write name of the jail to temp file. Clean-up procedure will
# use this name to remove jail.

echo "node ${NODE_NAME}" >> ${TEMP_FILE}

# Create a ng_eiface object for virtual node. ng_eiface
# object has a hook that can be connected to one of bridge
# links. After creating interface get its automatically
# generated name for further usage.

echo "Creating eiface interface for virtual node ${NODE_NAME}."
ngctl mkpeer eiface ether ether
EIFACE=`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'`
echo "Interface ${EIFACE} is created."

# Write name of the interface to temp file. Clean-up procedure
# will use this name to shutdown interface.

echo "interface ${EIFACE}" >> ${TEMP_FILE}

# Move virtual interface to virtual node. Note that Interface
# name will not be changed at the end of this movement. Moved
# interface can be seen at the output of ifconfig command in
# jail: 'jexec jailname ifconfig'

echo "Moving ${EIFACE} to ${NODE_NAME}"
ifconfig ${EIFACE} vnet ${NODE_NAME}

# Make lo0 interface localhost.
jexec ${NODE_NAME} ifconfig lo0 localhost

# Generate a random mac address for virtual interface. First
# three octets can be changed by user. Last three octets are
# generated randomly.
M4=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`
M5=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`
M6=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`

MAC=`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}`

# Set the link address (mac address) of virtual interface in
# virtual node to randomly generated MAC.
echo "Setting MAC address of ${EIFACE} to '${MAC}'"
jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC

# Either IPv4 or IPv6 can be used in this script. Ifconfig
# IP setting syntax differs slightly for two IP versions.
# For version 4 'inet' keyword is used whereas for version 6
# 'inet6' is used. Below line tries to decide which IP version
# is given and sets IPVER to 'inet' or 'inet6'.

IPVER=`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \
if( NF==4 && $1>0 && $1<256 && $2<256 && $3<256 && \
last[1]<256) print "inet"; else print "inet6"}'`

# Set IP address of virtual interface in virtual node.
echo "Setting IP address of ${EIFACE} to '${NODE_IP}'"
jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP}

# Connect virtual interface to bridge interface. Syntax is :
# ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK.
# Interface has one hook named 'ether' and below line connects
# ether hook to bridge's first unconnected link.

echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..."

ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \
|| exit 1
echo "done"

# Now, bridge has one more connected link thus link count is
# incremented.
LINKNUM=`expr ${LINKNUM} + 1`
done
echo "Virtual LAN established successfully!"

}

# Stop routine.
virtual_lan_stop() {

if [ ! -e ${TEMP_FILE} ]; then
echo "Nothing to stop! ${TEMP_FILE}: temp file not found"
else
echo -n "Shutdown bridge interface.."
OBJECTS=`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'`
for BRIDGE in ${OBJECTS}; do
ngctl shutdown ${BRIDGE}: >/dev/null 2>&1
done
echo "done"

echo -n "Shutdown all eiface interfaces..."
OBJECTS=`cat ${TEMP_FILE} | grep interface | awk '{print $2}'`
for INTERFACE in ${OBJECTS}; do
ngctl shutdown ${INTERFACE}: >/dev/null 2>&1
done
echo "done"

echo -n "Removing all jails..."
OBJECTS=`cat ${TEMP_FILE} | grep node | awk '{print $2}'`
for NODE in ${OBJECTS}; do
jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE}
done
echo "done"

echo "Removing tempfile ${TEMP_FILE}"
rm ${TEMP_FILE}
fi
echo "Virtual LAN objects removed successfully!"

}

# Main entry point.

case $# in
1)
case $1 in
start)
echo -n "Creating default target topology:"
echo " ${TARGET_TOPOLOGY}"
virtual_lan_start
;;
stop)

if [ ! -e ${TEMP_FILE} ]; then
echo -n "Noting to stop! ${TEMP_FILE}:"
echo " temp file not found"
else
virtual_lan_stop
fi
;;
help)
virtual_lan_usage
exit 1
;;
*)
virtual_lan_usage
exit 1

esac
;;
2)
case $1 in
start)
TARGET_TOPOLOGY=$2
echo -n "Creating target topology:"
echo "${TARGET_TOPOLOGY}"
virtual_lan_start
;;
*)
virtual_lan_usage
exit 1
esac
;;

*)
virtual_lan_usage
exit 1
esac

# /root >vnet.ng.test start
Creating default target topology: vdir4|10.0.2.20/24
Loading ng_ether.ko... done
Loading ng_bridge.ko... done
Loading ng_eiface.ko... done
No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up.
Verifying ethernet interface existence...done
Creating bridge interface: bridge0...
Bridge bridge0 is created and rl0 is connected.
Creating virtual node (jail) vdir4...vdir4: created
done
Creating eiface interface for virtual node vdir4.
Interface ngeth0 is created.
Moving ngeth0 to vdir4
Setting MAC address of ngeth0 to '00:1d:92:df:92:8e'
Setting IP address of ngeth0 to '10.0.2.20/24'
Connecting ngeth0:ether to bridge0:link2...done
Virtual LAN established successfully!

# /root >ngctl ls -l
There are 5 total nodes:
Name: rl0 Type: ether ID: 00000001 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
upper bridge0 bridge 00000006 link1

lower bridge0 bridge 00000006 link0

Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
link2 ngeth0 eiface 0000000a ether

link1 rl0 ether 00000001 upper

link0 rl0 ether 00000001 lower

Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
ether bridge0 bridge 00000006 link2

Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0

# /root >jexec vdir4 tcsh
vdir4 / >ping -c 1 8.8.178.135
PING 8.8.178.135 (8.8.178.135): 56 data bytes
ping: sendto: No route to host

--- 8.8.178.135 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
vdir4 / >exit
exit

# /root >vnet.ng.test stop
Shutdown bridge interface..done
Shutdown all eiface interfaces...done
Removing all jails...vdir4: removed
done
Removing tempfile /var/tmp/virtual.lan.tmp
Virtual LAN objects removed successfully!

# /root >jls
JID IP Address Hostname Path

# /root >ngctl ls -l
There are 3 total nodes:
Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0
Name: rl0 Type: ether ID: 00000001 Num hooks: 0
Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0

_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

Teske, Devin

unread,

May 18, 2013, 7:29:45 PM5/18/13

to

Sorry for top-post, but just wanted to add a quick note:

The output of "ngctl dot" would be very helpful to others in debugging your setup.
--
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Joe

unread,

May 18, 2013, 8:51:41 PM5/18/13

to

Teske, Devin wrote:
> Sorry for top-post, but just wanted to add a quick note:
>
> The output of "ngctl dot" would be very helpful to others in debugging your setup.

graph netgraph {
edge [ weight = 1.0 ];
node [ shape = record, fontsize = 12 ] {
"1" [ label = "{rl0:|{ether|[1]:}}" ];
"5" [ label = "{bridge0:|{bridge|[5]:}}" ];
"9" [ label = "{ngeth0:|{eiface|[9]:}}" ];
"e" [ label = "{ngctl2355:|{socket|[e]:}}" ];
};
subgraph cluster_disconnected {
bgcolor = pink;
"e";
};
node [ shape = octagon, fontsize = 10 ] {
"1.upper" [ label = "upper" ];
"1.lower" [ label = "lower" ];
};
{
edge [ weight = 2.0, style = bold ];
"1" -- "1.upper";
"1" -- "1.lower";
};
node [ shape = octagon, fontsize = 10 ] {
"5.link2" [ label = "link2" ];
"5.link1" [ label = "link1" ];
"5.link0" [ label = "link0" ];
};
{
edge [ weight = 2.0, style = bold ];
"5" -- "5.link2";
"5" -- "5.link1";
"5" -- "5.link0";
};
"5.link1" -- "1.upper";
"5.link0" -- "1.lower";
node [ shape = octagon, fontsize = 10 ] {
"9.ether" [ label = "ether" ];
};
{
edge [ weight = 2.0, style = bold ];
"9" -- "9.ether";
};
"9.ether" -- "5.link2";
};

Teske, Devin

unread,

May 18, 2013, 10:48:31 PM5/18/13

to

I rendered your output by saving it in a file ("joe.dot") and then running:

dot -Tsvg -o joe.svg < joe.dot

I then uploaded "joe.svg" to my website:

http://druidbsd.sf.net/download/joe.svg

Compare your output to any of the following:

http://druidbsd.sf.net/download/warden0.jbsd.svg
http://druidbsd.sourceforge.net/download/folsom.svg

It looks like everything is connected properly.

A couple thoughts off the top of my head:

a. Did you enable promiscuous mode on rl0 via ngctl? (in your script perhaps?)

b. Have you tried giving ngeth0 a new MAC address? (I do this through ngctl too, but I imagine ifconfig from within the jail could achieve the same thing)
--
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Joe

unread,

May 23, 2013, 9:42:02 AM5/23/13

to

Teske, Devin wrote:
> snip...............

>
> I rendered your output by saving it in a file ("joe.dot") and then running:
>
> dot -Tsvg -o joe.svg < joe.dot
>
> I then uploaded "joe.svg" to my website:
>
> http://druidbsd.sf.net/download/joe.svg
>
> Compare your output to any of the following:
>
> http://druidbsd.sf.net/download/warden0.jbsd.svg
> http://druidbsd.sourceforge.net/download/folsom.svg
>
> It looks like everything is connected properly.
>
> A couple thoughts off the top of my head:
>
> a. Did you enable promiscuous mode on rl0 via ngctl? (in your script perhaps?)
>
> b. Have you tried giving ngeth0 a new MAC address? (I do this through ngctl too, but I imagine ifconfig from within the jail could achieve the same thing)
> --
> Devin
>
>

Yes I enabled promiscuous mode and setautosrc 0 on rl0 via ngctl.
I can find no documentation on why this is done. Can you point me to some?

Yes I gave the jail a unique MAC address.

I tried to generate my own network map, but having problem.

ngctl dot > file.dot works.
dot -Tsvg -o file.svg < file.dot
gives me "command dot not found".
Tried ngctl dot -Tsvg -o file.svg < file.dot
and -T is illegal option.
What am I doing wrong?

Thanks for your help
Joe

Trond Endrestøl

unread,

May 24, 2013, 1:14:23 AM5/24/13

to

Please install graphics/graphviz, either from ports or from packages.

> Tried ngctl dot -Tsvg -o file.svg < file.dot
> and -T is illegal option.
> What am I doing wrong?
>
> Thanks for your help
> Joe