CDC processing windows disappeared...

[patric...@nasa.gov]

Hey,

We were processing data for ~5 hours, some of the data sources had
switched in and out (sim, real-time, playback), but just before our
real test the CDS window 'disappeared'. I've uploaded screen shots of
the IOM at the time of the 'crash' and was hoping for some
interpretation of what I'm seeing. The files names are:

IOM_err1_101909_p1.bmp
IOM_err1_101909_p2.bmp
iads iom_cds error_101809.bmp

Since these are screen captures the P1 and P2 files show the message
window in varoius states. The P1 shows the messages that started
happening and P2 shows where it 'crashed'. We recovered by bringing
up the CDS via the console. The file from 101809 is where we had the
same issue the day prior. This is using IADS version 5.3.2.

Any thoughts and/or recommendations?

[Michael Jones]

I see that you are running appx 5200 parameters at nearly 1 megasample/
sec. Are these rates 'normal' for you?

When you get an opportunity to run again, watch your memory
utilizations for the ComputeDataServer.exe on the CDS. How long does
it take for memory usage to stabalize? Does it stabalize?

How much physical RAM is available on your CDS hardware? What OS are
you running?

My apologies for bombarding you with questions,
Mike

On Oct 19, 4:40 pm, "patrick....@dfrc.nasa.gov"

[patric...@nasa.gov]

Mike,

I've uploaded another screen shot that has the details. (CDS_Info.bmp)

We're still looking at the memory usage.

> > Any thoughts and/or recommendations?- Hide quoted text -
>
> - Show quoted text -

[patric...@nasa.gov]

and yes those are our normal parameters and rates.

On Oct 19, 6:20 pm, Michael Jones <mjo...@iads-soft.com> wrote:

[Michael Jones]

One thing from that last pic jumps out at me as odd, but I'm going to
have to see what Mike Burt has to say about it.
It shows the requested data source packet size as 113876, but the
actual is 4718608. This may be normal but it sure looks odd to me.

While flowing data to the CDS as if for a mission, How long does it
take the memory utilization of the ComputeDataServer.exe to stabalize
and what does it stabalize to? If/When the problem occurs, what is
happening on the CDS console window?

On Oct 20, 9:24 am, "patrick....@dfrc.nasa.gov"

> > - Show quoted text -- Hide quoted text -

[Mike Burt]

Hi Patrick:

Looked over the screen shots and did not necessarily see anything
abnormal...besides of course the ERROR messages on the IOM that were
probably caused by the problems at the CDS. Not sure how long you had
been running the CDS when taking the snapshot but CPU and memory
utilization looked ok...

Have a few questions:

When you state: "We were processing data for ~5 hours, some of the
data sources had switched in and out (sim, real-time, playback)"...are
you also switching time sources or are you attached to some universal
'range' time that maintains a consistent time flow when switching data
sources?

When you state: "but just before our real test the CDS window
'disappeared'."...were you in the process of performing some tasks in
preparation of the 'real test' when the CDS disappeared?

Is this something you just started observing...i.e. had you been
running in this fashion with the same project(s) in the past without
incident? If so is there anything that changed recently that may
explain the difference in behavior?

Thank you for your cooperation in trying to track this down.

[patric...@nasa.gov]

The last snap shot was after the CDS had been running a short period
of time.

When the data sources were being switched in the time was constant
range time.

There were no tasks being performed prior to the CDS crashing (the
primary source had been switched in for some time); it was a sit and
wait period.

I was told that the room was being staffed so I'm unsure what the
users were up to.

We've been updating the config file with displays, but no major
imports/exports.

We've going to run this config file on a test system and see if we can
duplicate the result.

[Mike Burt]

> We've going to run this config file on a test system and see if we can
> duplicate the result.

That sounds like the best direction to move at this point...if you can
duplicate the problem we should be able to isolate...please let us
know what you find.

If you want to forward me the config file I can review and see if I
observe anything obvious...

Thanks