SMART24 to Earthworm

[Kyle R. Jones]

Hello Earthworm Community,

       First of all I have read though the few other posts concerning this same scheme and it seems to me that either this configuration isn't used much or It is really easy and I am just doing something wrong. Let me explain my setup. I have eight geotech digitizers as a part of our Infrasound array being telemetered via radio. We currently have the digitizers sending data back to a central computer running the geohub software. This software saves the data as wfdisc files. It is tedious to travel to the array to copy data files so we are trying to export these data streams back to our main facility and feel that it would be easier to set up an earthworm system to collect these streams and export them to our other earthworm system running in our main facility. 

I have set up a virtual machine on windows XP running earthworm on Ubuntu. i know this seems strange because earthworm will run on windows but I don't want to disrupt the functions of the XP machine. This way I can move the virtual earthworm system to any computer. Our digitizers are running the latest firmware 1.54 and we have the latest configuration utility, I believe it is 1.32.

Attached is a screenshot of the digitizer configuration with a summary below:

Site Name: S1004

Data Frame Size: 10

Compression: Canadian

Data Backfill: Enabled, FIFO

Authenticate: No

Triggered Data: No

Only Channel 6 (c6p) is enabled with location 03

IP address: 192.168.100.119 port 16005

Enable Earthworm Properties: Yes

Enable Server Mode: Yes

Module ID: 67 (This is for import_generic.d)

Installation ID: 158 (Sandia Seismo-Acoustic Network SSAN)

Array Name: FACT

I looked at the WAVE_RING with sniffring and sniffwave and here is the output.

earthworm@earthwormVM:/opt/earthworm/run/params$ sniffwave WAVE_RING

sniffwave: inRing flushed 7 packets of 112448 bytes total.

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

sniffwave: retrieved message too big (16064) for msg

earthworm@earthwormVM:/opt/earthworm/run/params$ sniffring WAVE_RING

1367947820 Received <inst:158> <mod:142> <type:  3> <seq:  1> <Length:    16>

1367947820 6500

1367947823 Received <inst:158> <mod: 67> <type: 19> <seq:  6> <Length: 16064>

1367947835 Received <inst:158> <mod:142> <type:  3> <seq:  2> <Length:    16>

1367947835 6500

I have attached the log files and all of the parameter files. One thing in particular is the naming convention. Here is an example from wave_serverV:

Tank    S1004 c6p  FACT 03  4096  INST_WILDCARD MOD_WILDCARD       512         10000         /opt/earthworm/tanks/S1004.tnk

Something isn't quite right here. I may have some parameter not set correctly. Any help would be appreciated.

Cheers,

   Kyle R. Jones

Sandia National Labs

[Philip Crotwell]

I don't know how to configure this particular module (or hardware), but it looks like you are getting decompressed tracebufs into the wave ring that are of size 16064, but earthworm has an upper limit of 4096 bytes.

We ran into this issue with reftek digitizers on a very quiet, very old, very low signal station using C2 compression, and ended up with decompressed tracebufs that were bigger than 4096. We solved it by turning up the gain on the digitizer (to 32) and switching to C0 compression, which meant the reftek did less efficient compression and so put fewer samples in its packets and hence the tracebufs were below the limit.

Not sure if this help, but I feel your pain. IMHO the hard upper limit is kind of a holdover from days of limited computer power. It would be nice if the xyz2ew modules would check for this on insertion, but that is not always the case.

Philip

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jon Rusho]

Kyle,

 Try changing the data frame size to 1 or 2 seconds.   That should drop the tracebuf2 message size to something more reasonable.  I've run into this before, and changing the data frame size dealt with the issue.

--jon

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
===================================
Jon Rusho
Seismic Network Engineer
University of Utah Seismograph Stations

j...@seis.utah.edu
801-585-5523

[Kyle R. Jones]

Thank you both for the helpful suggestions. Unfortunately that didn't seem to solve the problem. I tried going to 1 second and 2 seconds with and without compression without success. It did reduce the packet size but only to 8064.Still not under the 4096 limit. Any other thoughts?

Thanks,
     Kyle

[Jon Rusho]

What sample rate are you using?   I'm guessing it's high, to be creating a packet that large.   For testing, try reducing the sample rate and see what happens.

--jon

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kyle R. Jones]

Jon,
    I just dropped the sample rate from 200 Hz to 40 Hz with 1 second data packets, no compression and no backfill and earthworm still says the message size is 8064. That seems strange to me.

Kyle

[Kyle R. Jones]

Correction. I forgot that you have to log out of the digitizer when you change sample rates. It is now streaming perfectly at 40 Hz. Now I need to figure out how to make it work at 200 Hz....

[Kyle R. Jones]

Ok, now having remembered to restart the digitizer I put the sample rate back to 200 Hz with 2 second packets and everything is just fine. Thanks for the ALL the help and suggestions.

Kyle

[Philip Crotwell]

Hi

It seems like there are enough systems with this very same problem that it might be nice to be able to address this in a global way instead of patching every digitizer module. There are obviously two ways to do this, one is change the max size of a tracebuf. My understanding is that Paul at least thought this was fraught with peril because there were so many pieces of code in earthworm created under the "max 4096" assumption. Maybe here be dragons, but in the long run it would be a big improvement to remove this this type of arbitrary limitation from earthworm.

In the shorter term, perhaps a "tracebuf splitter" would be useful. Then, even if a digitizer creates tracebufs that are larger than 4096, they can be split into two or more smaller ones before being stuffing into the wave ring. I actually have code to do this splitting that I put together for Steve and Paul at UW. It is in Java and was pulling tracebufs that were too large out of a winston database. It is probably not directly usable in earthworm, but if an enterprising C programmer wanted to steal this and translate, I would not object.

Creating something like this would also have the advantage that if you wanted to run earthworm with a max tracebuf size of 512, you could split 4096 tracebufs. Not sure if this is really useful.

Tracebuf splitting source cpde is here starting on line 313:
https://code.google.com/p/seisfile/source/browse/src/main/java/edu/sc/seis/seisFile/earthworm/TraceBuf2.java

Philip

On Wed, May 8, 2013 at 2:19 PM, Kyle R. Jones <krj...@sandia.gov> wrote:

Ok, now having remembered to restart the digitizer I put the sample rate back to 200 Hz with 2 second packets and everything is just fine. Thanks for the ALL the help and suggestions.

Kyle

[mwit...@memphis.edu]

I'm not sure what dragons are there either but I can imagine all kinds of
unintended consequences with rings, buffers, etc. Plus bigger messages
increase latency which is bad for rapid notifications (1 second packets are
a good thing). If you only need to archive data and maybe make some triggers
for later analysis, then it does seem reasonable to want to have bigger
tracebufs. But it should never happen accidently; the user should have to
consciously allow it and configure accordingly. Sort of like, don't run
sendmail unless you really know what you're doing.

Mitch

Center for Earthquake Research and Information (CERI)
University of Memphis Ph: 901-678-4940
Memphis, TN 38152 Fax: 901-678-4734

[Philip Crotwell]

True, but the problem now is that it is possible to have digitizers configured in a way that the operator thinks is reasonable, the data makes it all the way from the field into the lab, and then earthworm throws it away because it is too big. Low latency is very desirable, but I will take higher latency data over no data any day.

I would argue that if low latency is important, then it is much more important to get that setting right at the digitizer, rather than rejecting it after it has already made it into the wave ring in earthworm. After all, at 40 sps, a 4096 tracebuf can contain 100 seconds of data, which is totally legal but hardly seems low latency to me.

Philip

To post to this group, send an email to earthworm_forum@googlegroups.com

To unsubscribe from this group, send an email to

earthworm_forum+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups
"Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an

email to earthworm_forum+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.

To post to this group, send an email to earthworm_forum@googlegroups.com

To unsubscribe from this group, send an email to

earthworm_forum+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_forum+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.

To post to this group, send an email to earthworm_forum@googlegroups.com

To unsubscribe from this group, send an email to

earthworm_forum+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

--- You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_forum+unsubscribe@googlegroups.com.

[mwit...@memphis.edu]

I agree with you. But if you install a digitizer with what appears to be
reasonable settings, and then earthworm blindly accepts the data and crashes
everything else, then I think I'd prefer no data from the newly configured
das at the expense of not crashing everything else.

Mitch

Center for Earthquake Research and Information (CERI)
University of Memphis Ph: 901-678-4940
Memphis, TN 38152 Fax: 901-678-4734

>>> https://code.google.com/p/**seisfile/source/browse/src/**
>>> main/java/edu/sc/seis/**seisFile/earthworm/TraceBuf2.**java<https://code.google.com/p/seisfile/source/browse/src/main/java/edu/sc/seis/seisFile/earthworm/TraceBuf2.java>

>>>
>>> Philip
>>>
>>>
>>> On Wed, May 8, 2013 at 2:19 PM, Kyle R. Jones <krj...@sandia.gov> wrote:
>>>
>>> Ok, now having remembered to restart the digitizer I put the sample rate
>>>> back to 200 Hz with 2 second packets and everything is just fine. Thanks
>>>> for the ALL the help and suggestions.
>>>>
>>>> Kyle
>>>>
>>>> --
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Earthworm Community Forum" group.
>>>>

>>>> To post to this group, send an email to earthworm_forum@googlegroups.**
>>>> com <earthwo...@googlegroups.com>

>>>>
>>>> To unsubscribe from this group, send an email to

>>>> earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>

>>>>
>>>> For more options, visit this group at

>>>> http://groups.google.com/**group/earthworm_forum?hl=en<http://groups.google.com/group/earthworm_forum?hl=en>

>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google Groups
>>>> "Earthworm Community Forum" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an

>>>> email to earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>
>>>> .
>>>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .

>>>>
>>>>
>>>>
>>>>
>>> --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Earthworm Community Forum" group.
>>>

>>> To post to this group, send an email to earthworm_forum@googlegroups.**
>>> com <earthwo...@googlegroups.com>

>>>
>>> To unsubscribe from this group, send an email to

>>> earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>

>>>
>>> For more options, visit this group at

>>> http://groups.google.com/**group/earthworm_forum?hl=en<http://groups.google.com/group/earthworm_forum?hl=en>

>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "Earthworm Community Forum" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>
>>> .
>>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>> .

>>>
>>>
>>>
>>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Earthworm Community Forum" group.
>>

>> To post to this group, send an email to earthworm_forum@googlegroups.**com<earthwo...@googlegroups.com>

>>
>> To unsubscribe from this group, send an email to

>> earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>

>>
>> For more options, visit this group at

>> http://groups.google.com/**group/earthworm_forum?hl=en<http://groups.google.com/group/earthworm_forum?hl=en>

>>
>> --- You received this message because you are subscribed to the Google
>> Groups "Earthworm Community Forum" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to earthworm_forum+unsubscribe@**googlegroups.com<earthworm_forum%2Bunsu...@googlegroups.com>
>> .
>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>> .

>>
>>
>>
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>

> To post to this group, send an email to earthwo...@googlegroups.com

>
> To unsubscribe from this group, send an email to

> earthworm_for...@googlegroups.com

>
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.