[minix3] Help about 2 nics on Minix

[Stefano Cordio]

Hi guys!
I’m Stefano Cordio from University of Rome “Tor Vergata” (Italy), I’m
a GSoC 2010 student and this summer I will work on a new firewall to
Minix3.

I'm glad to work with you!

I would like to configure a machine with two network cards on Minix
3.1.7 r6844. Based on what I read in the group, do it on VMware is not
possible because the lance driver does not support two devices, then I
would configure a physical machine.
My question is: how can I figure out which drivers support more
devices of the same type without looking at their source code? I would
leave this as a last resort :)
There is any text that explains it and that I have missed in the man
pages/group/wiki?

Alternatively, I found two different network adapters that are
supported by Minix (RTL 8169 and RTL 8139) but I have some
configuration problem.

If I enable only a device at a time through netconf, the driver is
loaded and the device is working properly (i.e. it gets the address
from the DHCP server on the local network and the traffic comes out).

But if I change /etc/inet.conf in this way

eth0 rtl8169 0 (default);
eth1 rtl8139 0;

drivers seem to be running (I see it through ps ax) and the device
files are created but no network device obtains the address from the
DHCP server. If I try to configure IP address and routing table
manually, it does not seem to work.

Any suggestions?

Thanks in advance for any advice!

Bye,

Stefano

--
You received this message because you are subscribed to the Google Groups "minix3" group.
To post to this group, send email to min...@googlegroups.com.
To unsubscribe from this group, send email to minix3+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/minix3?hl=en.

[Antoine LECA]

Stefano Cordio wrote:
> I'm glad to work with you!

Ciao Stefano,
I believe this LOOOOONG mail will give you quite a big hand.

> I would like to configure a machine with two network cards on Minix
> 3.1.7 r6844. Based on what I read in the group, do it on VMware is not
> possible because the lance driver does not support two devices,

AFAIK the lance driver is one of the few which DOES enable two devices.
The one which is not that clear is the (gigabit) e1000, where I do not
know is having two ports is a supported configuration; hints are, this
was not tested (Niek, fell free to correct me.)

The real problem, and you'll have it with physical adapters as well, is
that the PCI setup stuff in MINIX 3 (and netconf.sh) is targetted toward
only one NIC, so the second adapter needs manual set up.

The only way I succeeded was setting up the parameters of each card
separately, using the DRIVER0 and DRIVER1 environment variables (set up
with edparams) with something like DRIVERn=pci:1:2:3, where 1.2.3 is the
PCI address of each adapter. A real nuisance here is that I cannot say
to you which are the correct values to put instead of 1.2.3; to add the
insult to the injury, there are variations among version in the way
VMWare distribute the NIC cards among the PCI buses; and unfortunately,
while Minix' PCI driver knows the values (since it is able to enumerate
the bus), I do not know any way to print them; lspci is a useful tool
here in the sense that it was worse before, but it only get the VID:PID
pair from the PCI device, not the address... A second nuisance is that
while this scheme is implemented for several driver like dp8390 with
RTL8029 or rtl8139 (where I succeeded starting it, to be then stopped
because the driver does not handle several port simultaneously, more
details below), it is lacking for both lance and e1000...


Now, while I played quite a bit with that, I also believe the future
here is to change a bit the ways NIC drivers are handled (which is one
reason why I did not write all this in a wiki, if they should become
obsolete the day I finish the article, as it already occured.)
Nowadays, when you have two different NICs, say nicfoo and nicbar,
inet.conf will look like
eth0 nicfoo 0 { default;};
eth1 nicbar 0;
inet is talking with nicfoo and nicbar, which are two different drivers,
and can work at the same time without interference.
On the other hand, if you have two NIC but of the same kind nicbaz,
inet.conf will now look like
eth0 nicbaz 0 { default;};
eth1 nicbaz 1;
there will be only ONE driver (nicbaz), and inet will send requests with
an additional parameter (DL_PORT) being 0 or 1. HOWEVER, since the
driver is blocked while sending or receiving, while you send one packet
on nicbaz:0 you cannot send to nicbaz:1 :-(


We had this discussion some month ago (end Dec and again February) with
David van Moolenbroek, at the bare minimum, please have a look at
<msgid:9d1d88a6-89f9-4e06...@l26g2000yqd.googlegroups.com>
http://groups.google.com/group/minix3/msg/8f5127e61804c366).
My proposal is to do the same as done with at_wini, ie to drop the
nicbaz above and to setup the two cards as two independant drivers, and
inet will deal independantly with each card. This brings several advantages:
- drivers are easier to program, since we do not have to deal with
multiple ports, with race conditions etc; they are also lighter since
many index retrieving become unnecessary; merging two drivers codebase
or changing the name of a driver does not break anything any more;
- should one card become dead, that driver is stopped while another one
continue working undisturbed; similarly one can restart one interface
from scratch (restarting the driver) while the other stands quiet (note
that this would need substancial changes to inet though)
- we can easily select the driver at start up time based on PCI
enumeration, just passing it the current PCI address as discovered,
without having to worry about multiple port of the same driver, which is
the very problem you are facing in the first place; this minimizes the
changes needed, since the basic PCI framework does not change, just the
way the xx_pcibus, xx_pcidev, xx_pcifunc is set up, ie from command line
arguments rather than from environment variables parsing.

However I did not see updates from Jaswinder Singh who is reported to
work on this issue officially (while I am just a hacker.)

> My question is: how can I figure out which drivers support more
> devices of the same type without looking at their source code?

This is a sidenote and is in no way specific to networking code, but
documentation is a real problem with Minix these days: it regularly
lacks behind, so there is a number of features whcih either are
completely absent from both the manual and the wiki, while there are
several others (mainly carried over from Minix 2 days) where the
documentation is not in phase; I even know cases where the comments at
the top of the source files is misleading. The bottom line is, you
probably should learn to look into the source code ASAP...

> Alternatively, I found two different network adapters that are
> supported by Minix (RTL 8169 and RTL 8139) but I have some
> configuration problem.

With both a 8139 and a 8169, then I think it should work; I myself have
similar hardware and had some success; but I had to unmount since it was
a temporary experiment, and the officially published Minix driver code
and model changed in a way I cannot match with my hacks.
However, I do not believe two 8139 or two 8169 in the same machine will
work out of the box, I guess the drivers did not expect such a case and
mix states; in fact, it was while I was hacking my driver I found all
the stuff about multiple ports...

> If I enable only a device at a time through netconf, the driver is
> loaded and the device is working properly (i.e. it gets the address
> from the DHCP server on the local network and the traffic comes out).
>
> But if I change /etc/inet.conf in this way
>
> eth0 rtl8169 0 (default);
> eth1 rtl8139 0;
>
> drivers seem to be running (I see it through ps ax) and the device
> files are created but no network device obtains the address from the
> DHCP server.

OK, I believe I got this one...

First thing first, you have to understand all the stuff above above
configuration (sorry, I insist ;-) ). As I explained above, to set up
two NICs the automatic way does not work and you have at least to make
clear (in the system environement, that is at boot monitor or edparams)
the PCI address of each device.
In your case, assuming the rtl8169 is at 1.6.9 etc., it means
<envvar_rtl8169>0 = pci:1.6.9
<envvar_rtl8139>0 = pci:1.3.9

That is all good and fine; BUT the fact is, the name of the rtl8139
environment variable (end of rtl8139.h) is "RTLETH"##n; while the name
of the rtl8169 environment variable (in rtl8169.c, before the actual
code) is "RTLETH"##n; of course, this cannot work flawlessly! Change one
of them, recompile, enter the right values and I guess you will be done.


Antoine

[Stefano Cordio]

Hi Antonie,
thanks for your reply, your words are very useful!

Based on what you say, both of my scenarios could work, so first I
must identify the PCI address of the two network adapters in the both
cases (virtual and real) and then, in the second, I must change the
driver source code to obtain two different environment variables.

> The bottom line is, you probably should learn to look into the source code ASAP...

I am able to look into the source code, but in this period I would
study the inet source code and I would not add the drivers to the list
of things to do, unless it is absolutely necessary :)


Thanks again, I wish you a good day!


Bye,
Stefano

[Antoine LECA]

I wrote earlier:

> Stefano Cordio wrote:
>> I would like to configure a machine with two network cards on Minix
>> 3.1.7 r6844. Based on what I read in the group, do it on VMware is not
>> possible because the lance driver does not support two devices,
>
> AFAIK the lance driver is one of the few which DOES enable two devices.

I have good news here: I needed to test something on VMware (@Ben: it is
about the problem with the Dell I did not solve already, I am still
working on it) and while I was there, I did test that setup, and I
successfully enabled it! :-)

<snip>

> The real problem, and you'll have it with physical adapters as well, is
> that the PCI setup stuff in MINIX 3 (and netconf.sh) is targetted toward
> only one NIC, so the second adapter needs manual set up.
>
> The only way I succeeded was setting up the parameters of each card
> separately, using the DRIVER0 and DRIVER1 environment variables (set up
> with edparams) with something like DRIVERn=pci:1:2:3, where 1.2.3 is the
> PCI address of each adapter. A real nuisance here is that I cannot say
> to you which are the correct values to put instead of 1.2.3; to add the
> insult to the injury, there are variations among version in the way
> VMWare distribute the NIC cards among the PCI buses;

On recent version of VMWare (Player/Workstations), the effective
parameters seem to be bus=0, dev=0x10+i, func=0, where i is 0 for the
first virtual NIC, 1 for the second etc.
The resulting environment setup is then
d0p0s0> LANCE0=pci:0.16.0
d0p0s0> LANCE1=pci:0.17.0

But do not 'save' that unless you have the updated driver!

> A second nuisance is that
> while this scheme is implemented for several driver like dp8390 with
> RTL8029 or rtl8139 (where I succeeded starting it, to be then stopped
> because the driver does not handle several port simultaneously, more

> details below), it is lacking for lance

Attached is a patch which I ported from another driver, which enable the
parsing of "pci:*.*.*" for LANCEx; please try; if you or any people are
reporting success, then the VU team might consider it for inclusion?

Of course, the stuff about the requirement to use two system environment
variables still hold; furthermore, it should be obvious you need to have
two lines in /etc/inet.conf


If you study it, you may consider it somewhat bad style since I
essentially short-circuited the update_conf() stuff: this is becasue
unless there is some established standard for setting up the NIC
parameters, I followed the pci:* "standard" used for other drivers,
which directly contradicts the scheme used for the lance driver :-(


I guess one can increase the lance driver to support 3 or 4 ports (hence
that much "cards" in VMware); besides the obvious change in lance.h
#define EC_PORT_NR_MAX 2+1+1

I think you only need to increase adequately the size of ec_conf[]
ec_conf_t ec_conf[]= /* Card addresses */
{
/* I/O port, IRQ, Buffer address, Env. var, Buf selector. */
{ 0x1000, 9, 0x00000, "LANCE0" },
{ 0xD000, 15, 0x00000, "LANCE1" },

{ 0x1D000, 25, 0x00000, "LANCE2" },
{ 0x2D000, 35, 0x00000, "LANCE3" },

};


Antoine

[Stefano Cordio]

Hello Antoine,
I also have some good news:
I found the presence of the "pci_debug" environment variable during
the analisys of the PCI driver source code, it enables the printing of
the driver messages that can be read in /var/log/messages and then the
PCI devices address can be located.

Consequently, testing on the physical machine, I found the address of
the two PCI network cards, a RTL8139 (address: 0.13.0) and a RTL8169
(address: 0.14.0), I modified the driver source rtl8169 changing the
"RTLETH" environment variable in "RTLETH8169", then I set environment
variables as follows:

RTLETH8169 = pci: 0.13.0
RTLETH = pci: 0.14.0

So I changed /etc/inet.conf like this:

eth0 rtl8139 0 (default;);
eth1 rtl8169 0;

ip0 obtains the address by DHCP but ip1 doesn't obtain the address,
even if I manually set the ip1 address I can not do a local ping to
that interface...

Now I will carefully read your post and I see what I can do!

Thanks!

Stefano

[Sylvia Xu]

Hello!

  I'm studying this problem recently.And I have increased the rtl3189 to support two nicsin Xen,with a inet.conf file below:

 
eth0 rtl8139 0 (default;);

eth1 rtl8139 1;

 

     Default NIC is configured to virtual mode,and the second as bridge mode.The DHCP allocated the first nic with an ip of 192.168.122.190.  and I manually configured the second nic's ip to 172.16.166.46 and gateway as 172.16.166.1 ,to be in the same net as my physical host in.

#ifconfig -a

/dev/ip0: address 192.168.122.190 netmask 255.255.255.0 mtu 1500

/dev/ip1: address 172.16.166.46 netmask 255.255.255.0 mtu 1500

 

    I 've successfully ping through the 192.168.122.119 host(another vitual machine on Xen),and the  172.16,166,114 host(a vitual machine on VMW ).But the the problem comes the communication among the the three machine becoming very very slow.Even I used telnet command to connect 192.168.122.119 and 172.16,166,114, they worked but I have to wait on my shell to see the command I typed on the tty!!!

    Is this because the driver's ports shareing or the network supporting?

    Maybe two nics with same driver is really not a good idea. 

2010/4/30 Antoine LECA <Antoine...@gmail.com>

[Antoine LECA]

Sylvia Xu wrote:
> I'm studying this problem recently.And I have increased the rtl3189 to
> support two nicsin Xen,with a inet.conf file below:
>
> eth0 rtl8139 0 (default;);
> eth1 rtl8139 1;

Interesting experiment, thanks for sharing.

> the the problem comes the communication among the the three machine becoming
> very very slow.

How do you enable the two NICs? do you use RTLETHx variables?


Assuming you do not use that system, a wholly untested idea to try to
use two distinct drivers (to see if it is indeed the culprit) is to
change in /usr/etc/rc the code which starts the driver, inserting
between the "for driver in lance... done" loop and the "up inet" command
two lines which read
up rtl8139 -label rtl_8139_a -period 5HZ -args "<...>"
up rtl8139 -label rtl_8139_b -period 5HZ -args "<...>"
(adjust args the way you need to start correctly each NIC)

Then change inet.conf to read
eth0 rtl_8139_a 0 (default;);
eth1 rtl_8139_b 0;

As I said, just guesswork, so expect bugs and possible roadblockers...


Quick notes:
- using 0 in each port is the way to go: each driver are independant so
each are operating port #0
- I changed the rtl8139 to rtl_8139 just to avoid the catch in the "for"
loop; in fact the for loop can be completely wiped, it is effectively
replaced by the two "up" lines above
- RTLETH0 will not work since both drivers will read the _same_
environment variable; if you are using it, I guess the correct way is to
modify the driver to use argv[1] if present to take preference over the
RL_ENVVAR mechanism in rl_pci_conf

[Stefano Cordio]

Hi,
my mentors told me to leave this issue for now because in the near future
they will take care of the resolution by a big fix on inet. They want to
provide support for multiple network interfaces of the same type probably
throwing an instance of the driver for each device.

Cheers,
Stefano

[Antoine LECA]

Stefano Cordio wrote:
> my mentors told me to leave this issue for now because in the near future
> they will take care of the resolution by a big fix on inet. They want to
> provide support for multiple network interfaces of the same type probably
> throwing an instance of the driver for each device.

I understand. I am sorry to have wasted your time with my utterly stupid
ideas which are going the wrong way. I apologize for that.


Antoine

[Stefano Cordio]

Antoine LECA wrote:
> I understand. I am sorry to have wasted your time with my utterly stupid
> ideas which are going the wrong way. I apologize for that.

Antonie, no problem! I think it is normal in a community like this that each
proposes their own ideas and have the opportunity to test them with others!
Probably the tests we have done have been helpful to the developers of Minix
to find the best approach to follow!
Thanks again for taking your time to my problem!

Stefano