wave server problem 2006-02-10 10:10:15 <Buddy Becker>

[Branden C. Christensen]

wave server problem 2006-02-10 10:10:15 <Buddy Becker>




I've been having a problem with the Socorro network earthworm server
on
eworm.nmt.edu. Just about every day the wave server process will stop
working. Restarting earthworm will get everything working again until
the
next day. Here is an excerpt from the wave server log file:


20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages
lost.
20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787
messages
lost.
20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
20060210_UTC_08:51:22 SendHeader: failed for tank
/export/home/eworm/wave_tank/lem_ehz.tnk
SendReqDataRaw(): failed
_writeTraceDataRaw(): failed
20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on
tank[24]
20060210_UTC_08:51:22 Error sending error message reply to client.
ServerThread: Could not supply requested answer to client. Sent error
message instead.
20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
iReadResult = -1
_writeTraceDataAscii(): ftell[-1] failed for tank
/export/home/eworm/wave_tank/lem_ehz.tnk
_writeTraceDataAscii(): failed
20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
tank[24]
ServerThread: Could not supply requested answer to client. Sent error
message instead.
_writeTraceDataAscii(): ftell[-1] failed for tank
/export/home/eworm/wave_tank/lem_ehz.tnk
_writeTraceDataAscii(): failed
20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
tank[24]
20060210_UTC_08:51:22 Error sending error message reply to client.
ServerThread: Could not supply requested answer to client. Sent error
message instead.
20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
iReadResult = -1


I've checked to make sure the filesystem has room. I've tried deleting
all the wave tank files. My temporary solution has been to
automatically
check for these errors and restart earthworm when needed. I'm not
really
sure what to do next.

Any insight would be appreciated.

Thanks,
Buddy







Re: wave server problem 2006-02-10 10:17:19 <Paul Friberg>



Hi Buddy,

In order to help debug this problem it would be helpful for you
provide the .d file, the size of the tanks in question, the version
of EW you are running with, and the OS/version of the host which
is having this problem. The more info the better. Thanks in advance.

Some big issues w.r.t write failures are: 1. size of the tank (some
OS/versions have file size limitations) and 2. are there any system
level
disk problems (dmesg on solaris)?

DId anything in the configuration change recently?

Regards,

Paul

Buddy Becker wrote:
> I've been having a problem with the Socorro network earthworm server on
> eworm.nmt.edu. Just about every day the wave server process will stop
> working. Restarting earthworm will get everything working again until the
> next day. Here is an excerpt from the wave server log file:
>
>
> 20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
> 20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages lost.
> 20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787 messages
> lost.
> 20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
> 20060210_UTC_08:51:22 SendHeader: failed for tank
> /export/home/eworm/wave_tank/lem_ehz.tnk
> SendReqDataRaw(): failed
> _writeTraceDataRaw(): failed
> 20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on tank[24]
> 20060210_UTC_08:51:22 Error sending error message reply to client.
> ServerThread: Could not supply requested answer to client. Sent error
> message instead.
> 20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
> iReadResult = -1
> _writeTraceDataAscii(): ftell[-1] failed for tank
> /export/home/eworm/wave_tank/lem_ehz.tnk
> _writeTraceDataAscii(): failed
> 20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on tank[24]
> ServerThread: Could not supply requested answer to client. Sent error
> message instead.
> _writeTraceDataAscii(): ftell[-1] failed for tank
> /export/home/eworm/wave_tank/lem_ehz.tnk
> _writeTraceDataAscii(): failed
> 20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on tank[24]
> 20060210_UTC_08:51:22 Error sending error message reply to client.
> ServerThread: Could not supply requested answer to client. Sent error
> message instead.
> 20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
> iReadResult = -1
>
>
> I've checked to make sure the filesystem has room. I've tried deleting
> all the wave tank files. My temporary solution has been to automatically
> check for these errors and restart earthworm when needed. I'm not really
> sure what to do next.
>
> Any insight would be appreciated.
>
> Thanks,
> Buddy
>
>
>


--
Paul Friberg - p.fr...@isti.com - http://www.isti.com
(845) 256-9290 Office - (845) 256-9299 FAX
ISTI==Instrumental Software Technologies, Inc.
Systems Integration and Software Development Specialists







Re: wave server problem 2006-02-10 11:45:52 <Buddy Becker>




Earthworm version: 6.3

System specs:
eworm% uname -a
SunOS eworm 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-60

attached is the .d file and the file it includes.

dmesg didn't have anything useful to say.

There have been no recent changes, it just started happening one day.
That kind of leads me to suspect hardware problems, but the machine is
still running fine in all other ways.

Thanks,
Buddy

> Hi Buddy,
>
> In order to help debug this problem it would be helpful for you
> provide the .d file, the size of the tanks in question, the version
> of EW you are running with, and the OS/version of the host which
> is having this problem. The more info the better. Thanks in advance.
>
> Some big issues w.r.t write failures are: 1. size of the tank (some
> OS/versions have file size limitations) and 2. are there any system level
> disk problems (dmesg on solaris)?
>
> DId anything in the configuration change recently?
>
> Regards,
>
> Paul
>
> Buddy Becker wrote:
>> I've been having a problem with the Socorro network earthworm server on
>> eworm.nmt.edu. Just about every day the wave server process will stop
>> working. Restarting earthworm will get everything working again until
>> the
>> next day. Here is an excerpt from the wave server log file:
>>
>>
>> 20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
>> 20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages
>> lost.
>> 20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787 messages
>> lost.
>> 20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
>> 20060210_UTC_08:51:22 SendHeader: failed for tank
>> /export/home/eworm/wave_tank/lem_ehz.tnk
>> SendReqDataRaw(): failed
>> _writeTraceDataRaw(): failed
>> 20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on tank[24]
>> 20060210_UTC_08:51:22 Error sending error message reply to client.
>> ServerThread: Could not supply requested answer to client. Sent error
>> message instead.
>> 20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>> iReadResult = -1
>> _writeTraceDataAscii(): ftell[-1] failed for tank
>> /export/home/eworm/wave_tank/lem_ehz.tnk
>> _writeTraceDataAscii(): failed
>> 20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>> tank[24]
>> ServerThread: Could not supply requested answer to client. Sent error
>> message instead.
>> _writeTraceDataAscii(): ftell[-1] failed for tank
>> /export/home/eworm/wave_tank/lem_ehz.tnk
>> _writeTraceDataAscii(): failed
>> 20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>> tank[24]
>> 20060210_UTC_08:51:22 Error sending error message reply to client.
>> ServerThread: Could not supply requested answer to client. Sent error
>> message instead.
>> 20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>> iReadResult = -1
>>
>>
>> I've checked to make sure the filesystem has room. I've tried deleting
>> all the wave tank files. My temporary solution has been to
>> automatically
>> check for these errors and restart earthworm when needed. I'm not
>> really
>> sure what to do next.
>>
>> Any insight would be appreciated.
>>
>> Thanks,
>> Buddy
>>
>>
>>
>
>
> --
> Paul Friberg - p.fr...@isti.com - http://www.isti.com
> (845) 256-9290 Office - (845) 256-9299 FAX
> ISTI==Instrumental Software Technologies, Inc.
> Systems Integration and Software Development Specialists
>
>
>

(wave_serverV.d)

(wave_tank.nmt)







Re: wave server problem 2006-02-11 08:44:19 <Paul Friberg>



Hi Buddy,

Thanks for the additional info. Much appreciated. To all
others, if you report a software problem, please also
provide as much information as you can, config files,
log messages, OS and EW versions etc.

I have not seen this problem
before, but the log seems to indicate a problem with tank

/export/home/eworm/wave_tank/lem_ehz.tnk

so you could have some data that is out of whack from that
station/channel. WaveServerV doesn't like out of sequence
data or improperly formed packets so something could be
wrong with the data stream.

Also, it seems to be having problems with the ftell()
system call. This call returns -1 if one of the following
error conditions apply (from the ftell man page):

ERRORS
The ftell() and ftello() functions will fail if:

EBADF The file descriptor underlying stream is not an open
file descriptor.

ESPIPE
The file descriptor underlying stream is associated
with a pipe, a FIFO, or a socket.

The ftell() function will fail if:

EOVERFLOW
The current file offset cannot be represented
correctly in an object of type long.

The ftello() function will fail if:

EOVERFLOW
The current file offset cannot be represented
correctly in an object of type off_t.


I suspect an EOVERFLOW condition. What does "ls -l" tell you about
the tank in question? It could also be a new bug.

Also, Lynn Dietz recommends blowing away not just the bad tank, but
also the bad index file if/when you clean things up.

Regards,

Paul

Buddy Becker wrote:
> Earthworm version: 6.3
>
> System specs:
> eworm% uname -a
> SunOS eworm 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-60
>
> attached is the .d file and the file it includes.
>
> dmesg didn't have anything useful to say.
>
> There have been no recent changes, it just started happening one day.
> That kind of leads me to suspect hardware problems, but the machine is
> still running fine in all other ways.
>
> Thanks,
> Buddy
>
>
>>Hi Buddy,
>>
>>In order to help debug this problem it would be helpful for you
>>provide the .d file, the size of the tanks in question, the version
>>of EW you are running with, and the OS/version of the host which
>>is having this problem. The more info the better. Thanks in advance.
>>
>>Some big issues w.r.t write failures are: 1. size of the tank (some
>>OS/versions have file size limitations) and 2. are there any system level
>>disk problems (dmesg on solaris)?
>>
>>DId anything in the configuration change recently?
>>
>>Regards,
>>
>>Paul
>>
>>Buddy Becker wrote:
>>
>>>I've been having a problem with the Socorro network earthworm server on
>>>eworm.nmt.edu. Just about every day the wave server process will stop
>>>working. Restarting earthworm will get everything working again until
>>>the
>>>next day. Here is an excerpt from the wave server log file:
>>>
>>>
>>>20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
>>>20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages
>>>lost.
>>>20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787 messages
>>>lost.
>>>20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
>>>20060210_UTC_08:51:22 SendHeader: failed for tank
>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>SendReqDataRaw(): failed
>>>_writeTraceDataRaw(): failed
>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on tank[24]
>>>20060210_UTC_08:51:22 Error sending error message reply to client.
>>>ServerThread: Could not supply requested answer to client. Sent error
>>>message instead.
>>>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>>>iReadResult = -1
>>>_writeTraceDataAscii(): ftell[-1] failed for tank
>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>_writeTraceDataAscii(): failed
>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>>>tank[24]
>>>ServerThread: Could not supply requested answer to client. Sent error
>>>message instead.
>>>_writeTraceDataAscii(): ftell[-1] failed for tank
>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>_writeTraceDataAscii(): failed
>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>>>tank[24]
>>>20060210_UTC_08:51:22 Error sending error message reply to client.
>>>ServerThread: Could not supply requested answer to client. Sent error
>>>message instead.
>>>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>>>iReadResult = -1
>>>
>>>
>>>I've checked to make sure the filesystem has room. I've tried deleting
>>>all the wave tank files. My temporary solution has been to
>>>automatically
>>>check for these errors and restart earthworm when needed. I'm not
>>>really
>>>sure what to do next.
>>>
>>>Any insight would be appreciated.
>>>
>>>Thanks,
>>>Buddy
>>>
>>>
>>>
>>
>>
>>--
>> Paul Friberg - p.fr...@isti.com - http://www.isti.com
>> (845) 256-9290 Office - (845) 256-9299 FAX
>> ISTI==Instrumental Software Technologies, Inc.
>>Systems Integration and Software Development Specialists
>>
>>
>>


--
Paul Friberg - p.fr...@isti.com - http://www.isti.com
(845) 256-9290 Office - (845) 256-9299 FAX
ISTI==Instrumental Software Technologies, Inc.
Systems Integration and Software Development Specialists







Re: wave server problem 2006-02-13 10:38:06 <Buddy Becker>




The log complains about a different take each time. I've already tried
deleting the tank it mentions and I've tried deleting all tanks plus
the
indexes.

Just to make sure the wave tanks weren't getting too large to handle I
reduced their size a bit. Now they are all 122M and the problem is
still
happening.

-Buddy

> Thanks for the additional info. Much appreciated. To all
> others, if you report a software problem, please also
> provide as much information as you can, config files,
> log messages, OS and EW versions etc.
>
> I have not seen this problem
> before, but the log seems to indicate a problem with tank
>
> /export/home/eworm/wave_tank/lem_ehz.tnk
>
> so you could have some data that is out of whack from that
> station/channel. WaveServerV doesn't like out of sequence
> data or improperly formed packets so something could be
> wrong with the data stream.
>
> Also, it seems to be having problems with the ftell()
> system call. This call returns -1 if one of the following
> error conditions apply (from the ftell man page):
>
> ERRORS
> The ftell() and ftello() functions will fail if:
>
> EBADF The file descriptor underlying stream is not an open
> file descriptor.
>
> ESPIPE
> The file descriptor underlying stream is associated
> with a pipe, a FIFO, or a socket.
>
> The ftell() function will fail if:
>
> EOVERFLOW
> The current file offset cannot be represented
> correctly in an object of type long.
>
> The ftello() function will fail if:
>
> EOVERFLOW
> The current file offset cannot be represented
> correctly in an object of type off_t.
>
>
> I suspect an EOVERFLOW condition. What does "ls -l" tell you about
> the tank in question? It could also be a new bug.
>
> Also, Lynn Dietz recommends blowing away not just the bad tank, but
> also the bad index file if/when you clean things up.
>
> Regards,
>
> Paul
>
> Buddy Becker wrote:
>> Earthworm version: 6.3
>>
>> System specs:
>> eworm% uname -a
>> SunOS eworm 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-60
>>
>> attached is the .d file and the file it includes.
>>
>> dmesg didn't have anything useful to say.
>>
>> There have been no recent changes, it just started happening one day.
>> That kind of leads me to suspect hardware problems, but the machine is
>> still running fine in all other ways.
>>
>> Thanks,
>> Buddy
>>
>>
>>>Hi Buddy,
>>>
>>>In order to help debug this problem it would be helpful for you
>>>provide the .d file, the size of the tanks in question, the version
>>>of EW you are running with, and the OS/version of the host which
>>>is having this problem. The more info the better. Thanks in advance.
>>>
>>>Some big issues w.r.t write failures are: 1. size of the tank (some
>>>OS/versions have file size limitations) and 2. are there any system
>>> level
>>>disk problems (dmesg on solaris)?
>>>
>>>DId anything in the configuration change recently?
>>>
>>>Regards,
>>>
>>>Paul
>>>
>>>Buddy Becker wrote:
>>>
>>>>I've been having a problem with the Socorro network earthworm server on
>>>>eworm.nmt.edu. Just about every day the wave server process will stop
>>>>working. Restarting earthworm will get everything working again until
>>>>the
>>>>next day. Here is an excerpt from the wave server log file:
>>>>
>>>>
>>>>20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
>>>>20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages
>>>>lost.
>>>>20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787
>>>> messages
>>>>lost.
>>>>20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
>>>>20060210_UTC_08:51:22 SendHeader: failed for tank
>>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>>SendReqDataRaw(): failed
>>>>_writeTraceDataRaw(): failed
>>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on
>>>> tank[24]
>>>>20060210_UTC_08:51:22 Error sending error message reply to client.
>>>>ServerThread: Could not supply requested answer to client. Sent error
>>>>message instead.
>>>>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>>>>iReadResult = -1
>>>>_writeTraceDataAscii(): ftell[-1] failed for tank
>>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>>_writeTraceDataAscii(): failed
>>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>>>>tank[24]
>>>>ServerThread: Could not supply requested answer to client. Sent error
>>>>message instead.
>>>>_writeTraceDataAscii(): ftell[-1] failed for tank
>>>>/export/home/eworm/wave_tank/lem_ehz.tnk
>>>>_writeTraceDataAscii(): failed
>>>>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>>>>tank[24]
>>>>20060210_UTC_08:51:22 Error sending error message reply to client.
>>>>ServerThread: Could not supply requested answer to client. Sent error
>>>>message instead.
>>>>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>>>>iReadResult = -1
>>>>
>>>>
>>>>I've checked to make sure the filesystem has room. I've tried deleting
>>>>all the wave tank files. My temporary solution has been to
>>>>automatically
>>>>check for these errors and restart earthworm when needed. I'm not
>>>>really
>>>>sure what to do next.
>>>>
>>>>Any insight would be appreciated.
>>>>
>>>>Thanks,
>>>>Buddy
>>>>
>>>>
>>>>
>>>
>>>
>>>--
>>> Paul Friberg - p.fr...@isti.com - http://www.isti.com
>>> (845) 256-9290 Office - (845) 256-9299 FAX
>>> ISTI==Instrumental Software Technologies, Inc.
>>>Systems Integration and Software Development Specialists
>>>
>>>
>>>
>
>
> --
> Paul Friberg - p.fr...@isti.com - http://www.isti.com
> (845) 256-9290 Office - (845) 256-9299 FAX
> ISTI==Instrumental Software Technologies, Inc.
> Systems Integration and Software Development Specialists
>
>
>








Re: wave server problem 2006-02-13 14:10:05 <David Kragness>



Looking at your original logfile snippet:

>20060210_UTC_08:47:12 Wave_serverV: Client closed socket.
>20060210_UTC_08:50:27 wave_serverV: Circular queue lapped. 1 messages
>lost.
>
>
Indicates that the internal queue is (over) full at this time.

>20060210_UTC_08:50:57 wave_serverV: Circular queue lapped. 787
>messages
>lost.
>
>
Indicates you got 787 more relevant tracebuf messages than you could
process in the 30 seconds since the last message, so either you
suddenly
got a spirt of tracebuf traffic, or wave_server hung on a mutex, and
no
messages got processed in that time. (or it could be some combination
of the two.)

People have seen behavior before, where the time clock jumps on a
datalogger, tracebufs with out-of-order timestamps get generated, and
it
causes
strange things to happen. I believe that the wave_serverV code in the
v6.3
earthworm release has code to filter out out-of-order tracebuf data,
but
if you
are running any decimation or filter modules in your earthworm
environment that
produce tracebufs that the wave_server sees, that may cause a flurry
of
unexpected traffic.

>20060210_UTC_08:51:19 Wave_serverV: Client closed socket.
>20060210_UTC_08:51:22 SendHeader: failed for tank
>/export/home/eworm/wave_tank/lem_ehz.tnk
>
>
> SendReqDataRaw(): failed

>_writeTraceDataRaw(): failed
>20060210_UTC_08:51:22 server_thread.c: _writeTrDRaw() failed on
>tank[24]
>20060210_UTC_08:51:22 Error sending error message reply to client.
>ServerThread: Could not supply requested answer to client. Sent error
>message instead.
>
>
This would seem to be more of a socket issue, where it just can't
send
the data.
Unfortunately that error message does not include the return code
from
the send() call.
However, there are no mutexes held during network transmissions, so
even
if it took a
long time to transfer the data, that shouldn't have an affect on
normal
wave_server
operation, and would not cause the "queue lapped" messages.

>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>iReadResult = -1
>
>
Client closed the socket. Unimportant.

>_writeTraceDataAscii(): ftell[-1] failed for tank
>/export/home/eworm/wave_tank/lem_ehz.tnk
>_writeTraceDataAscii(): failed
>
>
>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>tank[24]
>ServerThread: Could not supply requested answer to client. Sent error
>message instead.
>
>
As indicated by Paul, this is saying that ftell failed for the tank.
It
is protected by the
tank mutex, so there should be no other data access to that tank at
that
time, and the
data files are ALWAYS OPEN. This would seem to point to a disk
corruption or a
memory error. Unfortunately that message does not print out errno at
the time of the failure.
If the ftell() call for some reason took a LONG time, then that could
cause the
"queue lapped" messages if wave_server was hung waiting for the mutex
so
it could
write new data to that same tank.

>_writeTraceDataAscii(): ftell[-1] failed for tank
>/export/home/eworm/wave_tank/lem_ehz.tnk
>_writeTraceDataAscii(): failed
>20060210_UTC_08:51:22 server_thread.c: _writeTrDAsc2() failed on
>tank[24]
>20060210_UTC_08:51:22 Error sending error message reply to client.
>ServerThread: Could not supply requested answer to client. Sent error
>message instead.
>20060210_UTC_08:51:22 Wave_serverV:ListenForMsg: Bad socket read.
>iReadResult = -1
>
>
>
Same thing happened again.

It seems odd that the failures always occur on ftell. Did you double
check the log file to make sure you
don't get any error messages or warnings on startup, or anywhere
prior
to the blowups. I'm not familiar
with the ftell code, but I would think that it wouldn't even require
a
disk read, because it just returning
the current offset within the given file. It would be interesting to
see what errno is after the ftell call,

You could try adding a debug statement at the ftell (serve_trace.c
approx line 402 - function _writeTraceDataAscii),
and see what errno is after the ftell.

good luck,
David Kragness


Buddy Becker wrote:

>The log complains about a different take each time. I've already tried
>deleting the tank it mentions and I've tried deleting all tanks plus the
>indexes.
>
>Just to make sure the wave tanks weren't getting too large to handle I
>reduced their size a bit. Now they are all 122M and the problem is still
>happening.
>
>-Buddy
>
>