Configuration loss on Real-Time?

[wrath_of_con]

THERE WAS A SUDDEN LOSS OF ALL CONFIGURATION DATA ON OUR REAL-TIME
DEVICE (PXI-8145RT). Max would no longer see it. I could no longer
connect to the Real-Time device via LabVIEW. I could not Ping the
device from our network.

What would cause such a thing? As far as I can tell, there was no
sudden power loss or surge, no user intervention that caused it, etc.

This is at least the 2nd time something like this has happened with
our Real-Time system. We had to clear the Flash memory and reload all
the software and reset the IP address.

I'm not getting warm and fuzzies about this when our system suddenly
requires reconfiguration from scratch.

- Con

[GValdes]

Hi, Con.

I can think of one thing that would cause a similar behavior. I'm
thinking that another computer on the network is trying to get the
same IP address that the 8145 is already using, and that kicks the
controller out of the network and possibly halts it. If that was the
case, you might want to check with your IT department and make sure
that they are reserving the IP address, so no other computer can try
and use it.
Now, if this was the actual cause of this behavior, I wouldn't think
that the only way of getting the controller back is by reformatting
the Compact Flash card. You should just be able to reset the IP using
the DIP switches on the front. That will set the IP address to
0.0.0.0 and you should be able to reassign an IP to it. Have you
tried doing this? Another thing you could try is, prior to format the
Flash, make a backup of it; that way, after you format you can just
copy the files and save time downloading the software back to the
controller. Finally, make sure you are formatting the Flash to be FAT
file system.
What kind of VI are you running in RT? Do you have any data
acquisition boards? I wonder if the VI is running out of memory or
disk space.

If you can think of any other piece of info that could be useful,
please, let me know.

Gustavo Valdes

[aasenjo]

Con,

I can recommend a couple of things to look for if you run into this
issue again. There are a few scenarios that might cause the symptoms
you are seeing, and, if there is really something wrong with the
system, we’d like to know about it anyway:

Obvious things to look for:

- You could have a startup application that uses up 100% of the
CPU.
- Your target might have been configured for a different sub-net
while it was running.
- Your target might have crashed repeatedly, falling back into
safe mode (although it should still be visible)

Not so obvious things:

- File corruption, for whatever reason, affecting files that get
accessed during start-up of the system. This could cause the system to
fall back into an un-configured, null-IP state.
- Reboot due to a crash and then fail to obtain an IP through
DHCP, if you are using DHCP.

Next time the target falls into such a state I would suggest the
following:

1) Reboot it and try to find it in MAX
2) Disable any start-up applications using the hardware switches
on the controller.
3) Since this is a target that does not have a video output, use
a Serial null-modem cable and a terminal utility (hyperterminal would
do) to check the output console. For this you would need to enable
Serial Redirect. The User Manual for the controller tells you what
switch to use to enable Serial Redirect:
<a
href=http://digital.ni.com/manuals.nsf/websearch/6EF72E1DA17CB52A86256A8B006399F2?OpenDocument&node=132100_US>”PXI-8140RT
Series User Manual”</a>

I use these settings:

Bps: 9600
Data bits: 8
Parity: none
Stop bits: 1
Flow Control: Hardware

4) The serial redirect output should at least give you an
indication of what’s going on, whether it’s failing to get an IP,
using the wrong IP, hanging, running out of memory or even if it’s
just continually rebooting at startup.
5) Force the target to boot into Safe-Mode. The User Manual
mentions how to do this. This should make the target show up again,
assuming it is within the same subnet or that it has the valid IP that
you are looking for.
6) If the controller comes back to life, try opening an FTP
session to it (you could use the FTP client in MAX or Internet
Explorer if necessary) and get the following files from the system:

- C:\ni-rt.ini
- C:\persist.pal (if it exists)
- C:\ni-rt\system\rtlog.txt (if it exists)
- C:\ni-rt\system\CONFIG3.MXC (if it exists)
- C:\ni-rt\system\config3.mxs (if it exists)
- C:\ni-rt\system\mxsjar.ini (if it exists)

We might be able to tell what happened from these files. Additionally,
it would be good to add more information like the version of LabVIEW
Real-Time you are using, boards and drivers, whether you are using a
start-up application or not, if using DHCP, and any other details
regarding the setup.

7) Worst case scenario would require you to take the Compact
flash card out of the controller and get the files (before deleting
them) by using a Compact Flash reader on a PC.

Booting the controller into safe-mode should be a no-excuse setting in
which you should be able to see the controller from MAX, or ping it,
assuming the basic networking conditions are met.

While your application is running you could check for available disk
space by using the Volume Info VI (File-IO palette). You could check
memory usage by running the Real-Time System Manager (RT 7.1) or using
the following utility:

<a
href=http://sine.ni.com/apps/we/niepd_web_display.display_epd4?p_guid=BEABFF12019F1037E034080020E74861&p_node=%20DZ52103&p_source=External>”Programmatically
Tracking Memory in LabVIEW Real-Time”</a>

Let us know,

Alejandro

LabVIEW Real-Time R&D

[wrath_of_con]

Thanks for all your information as well. I will keep it nearby and
include the information you requested if this happens again.

For now, maybe this will help, here's what happened: This may be
repeating some things, but I'm trying to mention everything that I
recall, in case it helps you imagine the scenario.

LabVIEW 7.1 with Real-Time
(1) 8145RT controller card
(2) 7831R FPGA cards (mainly used as IO)
(1) RS232 card (not used at the moment)

I was connecting to the RT system over our network and was developing
VI's that are being loaded on the Real-Time controller and that access
some IO on the FPGA cards. I have been developing similar vi's and
loading them and running them just fine. I was not developing Startup
vi's and turning the PXI chassis on/off or anything like that. I
would just run the vi's, they would download, run, rinse and repeat.
I will check our IT dept to see if another comp was trying to obtain
that IP address, but other than that, no one touched the system that I
know of.

Friday, while I was switching Operating Environments from RT to
LabVIEW for Windows, I noticed the options for the FPGA emulator and
FPGA device were no longer available from the drop down menu on
LabVIEW. Came back Tuesday to the same. The box was sitting there, on
but frozen and not responding. Ping did not see it. FTP did not see
it. MAX did not see it. Rebooting did not work. Turning it off/on did
not work. Trying to reconfigure through MAX did not work.

This has happened before and one of our engineer placed a call and the
option of clearing the flash memory was presented. So thats what we
did this time.

Should this happen again, I will try to get the files you mentioned.

Thanks for your assistance.

- Con

[wrath_of_con]

Thanks for your suggestions Gustavo!

I will check with our IT dept about having another computer trying to
obtain that IP address - but still I would think I would be able to
reassign an IP address through Max and keep going. But this froze
everything. No network communication signal light flashing on the
Ethernet connection (it was a solid green and thats all). The User1
light was a solid amber.

As far as what type of VIs - for now they are mostly straightforward
vi's that read information from an FPGA 7831R card and perform some
calculations.

I guess the thing that concerns me (ALOT) is that once the Real-Time
system is deployed, its supposed to be standalone and running! Not
needing to be taken apart and the Flash memory cleared with some 3rd
party device! You know what I mean?

Thanks!

- Con

As far as running out of space - not that I know of (do you know of a
way to check). The VI's I was developing at the time were not writing
to files or creating huge arrays or anything like that.

[GValdes]

Con,

As Alejandro recommended on another <a
href="http://exchange.ni.com/servlet/ProcessRequest?RHIVEID=101&RPAGEID=137&HOID=506500000005000000C3D40100&HIsQuestionPoster=No&HTHREAD=000120003&UCATEGORY_0=_50_%24_6_&UCATEGORY_S=0">thread</a>,
feel free to send those files if you get the same type of crash again.

Gustavo

[aasenjo]

I apologize for starting another thread, it took a while before I got
to posting my comments.

Thanks for linking to the other thread Gustavo.

Alejandro