Slow vxWorks boot - shared memory related?
Ben
We are running vxWorks 5.5.1 on two MVME5110 boards that interact
across a VME backplane. We've used the vxWorks shared memory routines
(VxMP) to implement much of our application.
Being at the latter stages of our project's implementation phase, there
is one vxWorks-related bug which has become increasingly urgent for us
to fix, which is this: after the vxWorks images are loaded, there is an
unacceptably-long delay that occurs between then and when the
application itself is able to run. (It's unacceptable in the sense that
our application has to meet a start-up time constraint as a project
requirement).
Below is a sample log of our master board starting up, with a note
added by me to show where the long delay (of about 40 seconds) occurs:
---------------------------------------------------
VxWorks System Boot
Copyright 1984-2002 Wind River Systems, Inc.
CPU: Motorola MVME5110-2261 - MPC 7410
Version: VxWorks5.5.1
BSP version: 1.2/2
Creation date: Aug 7 2005, 13:36:21
Press any key to stop auto-boot...
1 0
auto-booting...
boot device : tffs=0,0
unit number : 0
processor number : 0
host name : HOST1
file name : /tffs0/vxWorks
inet on ethernet (e) : 192.168.0.1:ffffff00
inet on backplane (b): 192.168.100.1:ffffff00
host inet (h) : 192.168.0.100
user (u) : vxworks
ftp password (pw) : vxworks
flags (f) : 0x8
target name (tn) : host1
other (o) : fei
Attaching to TFFS... done.
Loading /tffs0/vxWorks...1946032
Starting at 0x100000...
Attaching shared memory objects at 0x4100... done
*****DELAY OCCURS HERE, APPROX 40 SECONDS******
Attaching to SM net with memory anchor at 0x4100...
smSetup:Warning! previous parameters differ!
SM address: 192.168.100.1
Attached TCP/IP interface to fei unit 0
Attaching interface lo0...done
Initializing backplane net with anchor at 0x4100... smSetup:Warning!
previous parameters differ!
done.
Backplane anchor at 0x4100... Backplane IP address = 192.168.100.1
Attaching network interface sm0... done.
Adding 4696 symbols for standalone.
->
.... application loads normally
---------------------------------------------------
We suspect that it is something that is related to the shared-memory
network components, given that the delay occurs between the shared
memory objects being attached and the connection to the SM network
taking place. However, we've tinkered with including/excluding/altering
several vxWorks components (shared memory and other) to no avail so
far.
Some other notes about our setup:
- The problem occurs in an identical way whether the images are loaded
from flash (TFFS) or from a host machine via FTP.
- Excluding "auto addresss setup" or "default address for backplane"
from the "shared memory network components" folder doesn't seem to aid
the problem.
I've read several threads about older vxWorks bugs that are related to
blocked ports (when booting via FTP) causing a slow boot time, but I'm
confident that this isn't the same problem. Could it be related to the
way that our bootroms are configured? Any insights/questions/thoughts
are very welcome and appreciated!
Thanks kindly,
Ben
PS I would post our prjParams.h here but it's quite lengthy - if
there's any specific queries you might have about our configuration,
please feel free to ask.
Johan Borkhuis
> Hello all,
>
> We are running vxWorks 5.5.1 on two MVME5110 boards that interact
> across a VME backplane. We've used the vxWorks shared memory routines
> (VxMP) to implement much of our application.
>
> Being at the latter stages of our project's implementation phase, there
> is one vxWorks-related bug which has become increasingly urgent for us
> to fix, which is this: after the vxWorks images are loaded, there is an
> unacceptably-long delay that occurs between then and when the
> application itself is able to run. (It's unacceptable in the sense that
> our application has to meet a start-up time constraint as a project
> requirement).
I have seen the same happen also, but it was on a system without shared
memory. It was not a real problem for us, so we just left it.
How much memory do you have? It might be related to memory
initialization and/or MMU init or something like that.
What you could do to check what is happening:
- Check the ethernet ports to see if any traffic is coming out of your
board.
- See if you can add some debug info in the VxWorks startup, to see
where the software is hanging.
Kind regards,
Johan
--
o o o o o o o . . . _____J_o_h_a_n___B_o_r_k_h_u_i_s___
o _____ || http://www.borkhuis.com |
.][__n_n_|DD[ ====_____ | jo...@borkhuis.com |
>(________|__|_[_________]_|________________________________|
_/oo OOOOO oo` ooo ooo 'o!o!o o!o!o`
== VxWorks FAQ: http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html ==
macabbi
and then it timeouts, hence the long delay.
my $.05
vxworks
Ben
I started out this morning looking at the network traffic that was
emanating from the ethernet ports on our boards (and also our IPMC761
I/O expansion module), and I didn't uncover anything particularly
unusual.
However, I have made some interesting findings by adding some trace
statments to prjConfig.c. I've discovered that the delay occurs when
usrNetworkInit () calls usrEndLibInit (). We are currently using END as
a base for our shared memory network driver, in accordance with the
WindRiver patch t22-cp1-smend - you can download this from WindSurf
online support. When I remove END support from our kernel, the start-up
is much, much quicker - however, it breaks our network and shared
memory network configuration. I'm going to have a play around and see
if I can revert to a BSD-based shared memory network.
Being relatively new to VME debugging, I'm unsure how to go about
"checking the VME bus traffic"... how would one go about doing this?
Thanks again,
Ben
Martin Raabe
Ben schrieb:
> Being relatively new to VME debugging, I'm unsure how to go about
> "checking the VME bus traffic"... how would one go about doing this?
VMEbus is an asynchronous bus, which makes it happen that there is a bus
controller.
This bus controller looks at bus accesses.
So when there is an access to a bus address, which does not exist, the
bus controller sets up a time out (bus timeout), to abandon the access
to the bus.
This is done in hardware via a signal called bus error.
If that occurs, there is a wrong address setting in the configuration of:
the Shared Memory
the VMEbus board, where you start the VxWorks on
the VMEbus board, you want to get connected to
So if the configured VMEboards are not plugged in or have not finished
the initialisation of their shared memory software, the bus error may
occur.
The bus error can happen many times depending of the number of VMEbus
accesses. So it may happen that it takes some seconds, where the VMEbus
traffic prevents your bootstarp from finishing.
Usually you can use a VMEbus tracing hardware, which is not that often
in your shelf due to high costs.
The other method is to take an osciloscope and watch the buserr line on
the VMEbus.
Hope it helps.
--
BaSystem Martin Raabe
E: Martin.Raabe<at>B-a-S-y-s-t-e-m<dot>de
Johan Borkhuis
> Thanks for the advice Johan and macabbi,
>
> I started out this morning looking at the network traffic that was
> emanating from the ethernet ports on our boards (and also our IPMC761
> I/O expansion module), and I didn't uncover anything particularly
> unusual.
>
> However, I have made some interesting findings by adding some trace
> statments to prjConfig.c. I've discovered that the delay occurs when
> usrNetworkInit () calls usrEndLibInit (). We are currently using END as
> a base for our shared memory network driver, in accordance with the
> WindRiver patch t22-cp1-smend - you can download this from WindSurf
> online support. When I remove END support from our kernel, the start-up
> is much, much quicker - however, it breaks our network and shared
> memory network configuration. I'm going to have a play around and see
> if I can revert to a BSD-based shared memory network.
You made some good progress here. What you could do to narrow it down
more is look for the function usrEndLibInit and see where in this
function the most time is spend. I don't know the exact location, but
you can find this file somewhere under target/config IIRC.
ssubbarayan
Very neat explaination Martin.Thanks for helping to understand.
Regards,
s.subbarayan
Jostein Trones at
on our system I had a very similar experience - its not VME however, but
cPCI.
In our case, it was due to no network being connected. If I removed the
network plug (or if it was unterminated), the boot-sequence was considerably
longer. If it was connected, the boot was quick..
Don't know if this is relavant to you.
Good luck anyways,
-jostein
"Ben" <ben.c...@gmail.com> wrote in message
news:1125379943....@g44g2000cwa.googlegroups.com...
Ben
haven't actually had to go to an actual hardware level to investigate
anything yet, but when I do that seems like a good tactic to start
with.
As far as my END problem goes, I think I have a reasonably clear idea
of what's going on now. As part of usrEndLibInit, the MVME5100 BSP
function sysFei82557EndLoad is called to initialise the END devices.
This, in turn, calls fei82557EndLoad (pParamStr), which is contained in
the vxWorks-provided driver for the END style 82557 Ethernet interface
(target/src/drv/end/fei82557End.c).
If I have a network cable plugged in to the board's Ethernet ports, and
connected to something that is switched on (eg. my development PC), the
driver initialises in about 2-3 seconds (I inserted some trace
statements into sysFei82557EndLoad to check). However, if the Ethernet
ports on the boards that are being initialised by the END driver are
not plugged in to anything, then a delay of 15-20 seconds occurs. I'm
guessing that somewhere in fei82557EndLoad, the driver tries to find an
"active" physical connection (maybe at the data link layer, or
something), although after a certain period it obviously times out and
continues on regardless. While I can see the source code for the driver
(in fei82558End.c), it seems to be pre-compiled into vxWorks so I'm
unable to play around with it.
As a workaround, I'm currently experimenting with disabling the
Ethernet ports that we don't need to use (by removing them from the
endDevTbl in our BSP's configNet.h), and simply ensuring that the
Ethernet port we do want to use (which is located on our MVME761
transition module, accessible via an IPMC761 PMC card) is plugged in to
a "live" connection.
We've also sent some queries off to our local WindRiver channels, so
hopefully we may here something from them at some stage. Any further
thoughts still appreciated! :-)
Cheers,
Ben
Ben
that this is, in fact, exactly the same problem that you're talking
about. Thanks for that.
Ben
Bill Pringlemeir
> If I have a network cable plugged in to the board's Ethernet ports,
> and connected to something that is switched on (eg. my development
> PC), the driver initialises in about 2-3 seconds (I inserted some
> trace statements into sysFei82557EndLoad to check). However, if the
> Ethernet ports on the boards that are being initialised by the END
> driver are not plugged in to anything, then a delay of 15-20 seconds
> occurs. I'm guessing that somewhere in fei82557EndLoad, the driver
Auto-negotiation. 10-base T or 10-base 2, 100Mbps or 10 Mbps. If it
is possible, configure your network card (probably configNet.h) to
match your hardware. This is often possible if the system design has
the network configuration under your control. Obviously this won't
work if you are making a generic network device (like a hub). Ergo,
you must configure this.
If your network has a heartbeat this will also speed the detection.
Also, some hardware has a bit to detect connectivity. vxWorks does
not have an interface to get this, but you can write your own code to
access it.
hth,
Bill Pringlemeir.
--
A mind that is stretched to a new idea never returns to its original
dimension - Anonymous
vxWorks FAQ, "http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html"
Ben
Thanks for the explanation - that does seem to be on the money. I've
had a look deeper into the vxWorks-supplied driver
src/drv/end/fei82557End.c and it seems that it is indeed the
auto-negotation phase which is the issue.
My problem is that the highest level code which I can alter and
re-compile is at the BSP level (in config/mv5100/sysFei82557End.c), and
I don't have any options from there which are passed down to the driver
level to statically set connection speed/duplex settings etc. (Is there
a way to re-compile vxWorks drivers in /target/src/drv? I tried
altering them and rebuilding the vxWorks images/bootroms but the
changes don't seem to be included.) So, it seems that my one and only
workaround is to ensure that I have a live network connection plugged
in to all our Ethernet ports. The thing is, this isn't practical in
some scenarios pertaining to our project specifically, but we'll just
have to deal with that... we're wondering if it's possible to make a
kind of dummy Ethernet terminator?
The paramater string that is passed to the driver function
fei82557EndLoad contains a "flags" entry, and I was wondering whether
this might be able to be used for setting some driver options; however,
in inimitable vxWorks style, it seems that the flags parameter is "Not
Implemented".
Thanks again,
Ben
Ben
I'm not sure what I was thinking when I wrote that... nothing probably!
We've mocked up a quick loopback RJ45 plug to do the job, seems to work
fine.
Ben
Johan Borkhuis
> My problem is that the highest level code which I can alter and
> re-compile is at the BSP level (in config/mv5100/sysFei82557End.c), and
> I don't have any options from there which are passed down to the driver
> level to statically set connection speed/duplex settings etc. (Is there
> a way to re-compile vxWorks drivers in /target/src/drv? I tried
> altering them and rebuilding the vxWorks images/bootroms but the
> changes don't seem to be included.)
Try including the file src/drv/end/fei82557End.c into your project. This
should rebuild the driver, and it will not include the procompiled
driver version that is in the VxWorks library.
Ben
off as much out of the vxWorks bootup time that I can, and I'm now
trying to optimise our actual application.
Thanks for the advice everyone - I've learned a lot!
Ben