Earthworm running but not creating events?

106 views
Skip to first unread message

Samuel Harms

unread,
Sep 13, 2022, 4:07:19 PM9/13/22
to Earthworm Community Forum
Hi, I'm running Earthworm 7.10 under RHEL 8.5. The server and all the modules seem to be configured and running. I have tank files but no arc or ml files. I've checked the eqassemble, eqcoda, eqbuf, hypo2000_mgr params and don't see anything wrong.

Some oddities I've found are:
  1. In the statmgr logs I get missed message warnings. Is this just because I don't have an email server? Eg: 
    • X/binder Missed msg(s)  inst:1 mod:150 typ:8  region:1020
    • X/eqassemble Missed msg(s)  i1 m150 t8  region:1005
  2. Pick_FP keeps getting restarted every 10 mins or so. 
    • X/pick_FP (MOD_PICK_FP) module dead
    • 20220913_UTC_18:33:01 Statmgr: sent restart request for pick_FP pid 1042723
    • 20220913_UTC_18:33:02 startstop: successfully restarted <pick_FP> <pid: 1043616
       3. I only have about 2 minutes of tank files. There are lots of tanks coming in as I have my sensitivity set to very high (for now). Is it normal to have such short duration? Approx 30 .tnk files are in the directory.

Any help is appreciated!
Thanks!

General info:
        Hostname-OS:            X - Linux 4.18.0-348.12.2.el8_5.x86_64
        Start time (UTC):       Tue Sep 13 18:08:09 2022
        Current time (UTC):     Tue Sep 13 18:08:22 2022
        Disk space avail:       139481812 kb
        Ring  1 name/key/size:  WAVE_RING / 1000 / 16384 kb
        Ring  2 name/key/size:  WAVE_STAGE_RING / 1025 / 16384 kb
        Ring  3 name/key/size:  PICK_RING / 1005 / 4096 kb
        Ring  4 name/key/size:  HYPO_RING / 1015 / 2048 kb
        Startstop's Log Dir:    /app/earthworm/log/
        Startstop's Params Dir: /app/earthworm/params/
        Startstop's Bin Dir:    /app/earthworm//v7.10/bin
        Startstop Version:      v7.10 2019-08-13 (64 bit)

         Process  Process               Class/    CPU
          Name      Id      Status     Priority   Used    Argument
         -------  -------   ------     --------   ----    --------
       startstop 1042114    Alive         ??/ 0 00:00:00  -
         statmgr 1042115    Alive         ??/ 0 00:00:00  <thworm/params/statmgr
        slink2ew 1042116    Alive         ??/ 0 00:00:00  <hworm/params/slink2ew
    wftimefilter 1042117    Alive         ??/ 0 00:00:00  <m/params/wftimefilter
         pick_FP 1042118    Alive         ??/ 0 00:00:00  <thworm/params/pick_FP
       binder_ew 1042119    Alive         ??/ 0 00:00:00  <worm/params/binder_ew
      eqassemble 1042120    Alive         ??/ 0 00:00:00  <orm/params/eqassemble
        localmag 1042121    Alive         ??/ 0 00:00:00  <hworm/params/localmag
    wave_serverV 1042122    Alive         ??/ 0 00:00:00  <m/params/wave_serverV
         ew2file 1042123    Alive         ??/ 0 00:00:00  <thworm/params/ew2file
      copystatus 1042124    Alive         ??/ 0 00:00:00  WAVE_RING HYPO_RING
      copystatus 1042125    Alive         ??/ 0 00:00:00  PICK_RING HYPO_RING

Stefan Lisowski

unread,
Sep 13, 2022, 4:21:34 PM9/13/22
to earthwo...@googlegroups.com, Samuel Harms


On 9/13/22 2:45 PM, Samuel Harms wrote:
> Hi, I'm running Earthworm 7.10 under RHEL 8.5. The server and all the
> modules seem to be configured and running. I have tank files but no arc
> or ml files. I've checked the eqassemble, eqcoda, eqbuf, hypo2000_mgr
> params and don't see anything wrong.
>
> Some oddities I've found are:
>
> 1. In the statmgr logs I get missed message warnings. Is this just
> because I don't have an email server? Eg:

You would get these same warnings to your email if you had an email
server. So, no the missed messages are talking about Earthworm messages,
not Emails

> * X/binder Missed msg(s)  inst:1 mod:150 typ:8  region:1020
> * X/eqassemble Missed msg(s)  i1 m150 t8  region:1005

Look in earthworm.d, what is module 150?

Is that module ID duplicated in any module?

> 2. Pick_FP keeps getting restarted every 10 mins or so.
>
> *
> o X/pick_FP (MOD_PICK_FP) module dead
> o 20220913_UTC_18:33:01 Statmgr: sent restart request for pick_FP
> pid 1042723
> o 20220913_UTC_18:33:02 startstop: successfully restarted
> <pick_FP> <pid: 1043616

Make sure the .desc file that has expected heartbeats is double or so
what the sent heartbeats are in the .d file. Make sure there aren't two
modules using pick fp's module name in another .d file

>        3. I only have about 2 minutes of tank files. There are lots of
> tanks coming in as I have my sensitivity set to very high (for now). Is
> it normal to have such short duration? Approx 30 .tnk files are in the
> directory.

You can increase the megabytes of your tanks. If you do this you'll need
to shut down the wave server, change the config, delete all the old tank
files, and then start the waveserver back up.

> /Any help is appreciated!/
> Thanks!
> *
> *
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>
> To post to this group, send an email to earthwo...@googlegroups.com
>
> To unsubscribe from this group, send an email to
> earthworm_for...@googlegroups.com
>
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en
> <http://groups.google.com/group/earthworm_forum?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to earthworm_for...@googlegroups.com
> <mailto:earthworm_for...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/earthworm_forum/a7ea49b2-c967-4f93-9d38-80ec5472ed08n%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/a7ea49b2-c967-4f93-9d38-80ec5472ed08n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--








*To follow ISTI news and updates please subscribe to our newsletter
"the isti letter" at  https://www.isti.com/newsletter-sign-up
<https://www.isti.com/newsletter-sign-up>.*





S H

unread,
Sep 15, 2022, 3:07:02 PM9/15/22
to Earthworm Community Forum

You would get these same warnings to your email if you had an email
server. So, no the missed messages are talking about Earthworm messages,
not Emails
So what are these messages? Anything to worry about? 


Look in earthworm.d, what is module 150?
It's Pick_FP, the same module that is rebooting... Maybe the reason I'm not getting my files.


Is that module ID duplicated in any module?
I've scanned through them all and I don't think so...  Do you mean the "MyModuleID" of the different modules?


> 2. Pick_FP keeps getting restarted every 10 mins or so.


Make sure the .desc file that has expected heartbeats is double or so
what the sent heartbeats are in the .d file. Make sure there aren't two
modules using pick fp's module name in another .d file
I'm assuming this is in the statmgr.desc file, where I have the following "tsec: 120  page: 0  mail: 0"
In pick_FP.d I have 
"HeartbeatInt            30     # Heartbeat interval, in seconds,
 RestartLength          100"
Seems right, unless I'm looking at the wrong .desc file.



>        3. I only have about 2 minutes of tank files. There are lots of
> tanks coming in as I have my sensitivity set to very high (for now). Is
> it normal to have such short duration? Approx 30 .tnk files are in the
> directory.

You can increase the megabytes of your tanks. If you do this you'll need
to shut down the wave server, change the config, delete all the old tank
files, and then start the waveserver back up.

Where is this parameter found?

Stefan Lisowski

unread,
Sep 15, 2022, 4:40:00 PM9/15/22
to earthwo...@googlegroups.com


On 9/14/22 5:58 PM, S H wrote:

You would get these same warnings to your email if you had an email
server. So, no the missed messages are talking about Earthworm messages,
not Emails
So what are these messages? Anything to worry about?
yes, we need to worry about them



Look in earthworm.d, what is module 150?
It's Pick_FP, the same module that is rebooting... Maybe the reason I'm not getting my files.


Is that module ID duplicated in any module?
I've scanned through them all and I don't think so...  Do you mean the "MyModuleID" of the different modules?

Yes.


> 2. Pick_FP keeps getting restarted every 10 mins or so.


Make sure the .desc file that has expected heartbeats is double or so
what the sent heartbeats are in the .d file. Make sure there aren't two
modules using pick fp's module name in another .d file
I'm assuming this is in the statmgr.desc file, where I have the following "tsec: 120  page: 0  mail: 0"
In pick_FP.d I have 
"HeartbeatInt            30     # Heartbeat interval, in seconds,
 RestartLength          100"
Seems right, unless I'm looking at the wrong .desc file.

You want to look at the pick_FP.desc file that the statmgr.d refers to.


>        3. I only have about 2 minutes of tank files. There are lots of
> tanks coming in as I have my sensitivity set to very high (for now). Is
> it normal to have such short duration? Approx 30 .tnk files are in the
> directory.

You can increase the megabytes of your tanks. If you do this you'll need
to shut down the wave server, change the config, delete all the old tank
files, and then start the waveserver back up.

Where is this parameter found?

in the channel lines in the wave_serverV.d configuration. They may be in this file, or someone may have configured it with @wsv_channels.d or something, so you'd need to look at the latter file... if you see the "@ include".

#
#           SCNL      Record       Logo                  File Size   Index Size       File Name     New
#          names       size  (TYPE_TRACEBUF2 only)         (megabytes) (max breaks)     (full path)      Tank

Tank ALS   HLZ ST 01  2064  INST_UNKNOWN    MOD_ADSEND_A       1 40  .\tank\\ALS_HLZ_ST_01.tnk
Tank ALS   HSZ ST 01  2064  INST_UNKNOWN    MOD_ADSEND_A       1 40  .\tank\\ALS_HSZ_ST_01.tnk


You'll see in this case 1 is the megabyte size for each line. You'd want to increase the size of that 1 value that's right before the number 40 to increase the size of the tank for this channel.


--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/earthworm_forum/01dc842c-9379-49f7-9ad5-f72d84f155b2n%40googlegroups.com.

To follow ISTI news and updates please subscribe to our newsletter "the isti letter" at  https://www.isti.com/newsletter-sign-up.

S H

unread,
Sep 16, 2022, 5:04:05 PM9/16/22
to Earthworm Community Forum
Ok, I made sure that statmgr.desc expects a heartbeat every 120s heartbeat and pick_FP.d sends one every 30s.
My log file for pick_FP.d is useless as it only lists the stations...

How can I figure out what is going on with this module 150 Pick_FP with it constantly dying?

Maybe the most concerning error:
UTC_Fri Sep 16 20:54:49 2022  x/pick_FP (MOD_PICK_FP) module dead, PID unknown; needs human intervention. Page sent.
UTC_Fri Sep 16 20:55:49 2022  x/statmgr (MOD_STATMGR) module dead


This human intervention must mean it was unrecoverable? Weird thing is it shows up as still being alive...
       startstop 1322389    Alive         ??/ 0 00:00:00  -
         statmgr 1322390    Alive         ??/ 0 00:00:00  <thworm/params/statmgr
        slink2ew 1322391    Alive         ??/ 0 00:00:00  <hworm/params/slink2ew
    wftimefilter 1322392    Alive         ??/ 0 00:00:00  <m/params/wftimefilter
         pick_FP 1322393    Alive         ??/ 0 00:00:00  <thworm/params/pick_FP
       binder_ew 1322394    Alive         ??/ 0 00:00:00  <worm/params/binder_ew
      eqassemble 1322395    Alive         ??/ 0 00:00:00  <orm/params/eqassemble
        localmag 1322396    Alive         ??/ 0 00:00:00  <hworm/params/localmag
    wave_serverV 1322397    Alive         ??/ 0 00:00:00  <m/params/wave_serverV
         ew2file 1322398    Alive         ??/ 0 00:00:00  <thworm/params/ew2file
      copystatus 1322399    Alive         ??/ 0 00:00:00  WAVE_RING HYPO_RING
      copystatus 1322400    Alive         ??/ 0 00:00:00  PICK_RING HYPO_RING



Then the rest of it:

UTC_Tue Sep 13 00:05:06 2022  x/pick_FP (MOD_PICK_FP) module dead
20220913_UTC_00:05:06 Statmgr: sent restart request for pick_FP pid 971973
20220913_UTC_00:18:01 statmgr tport_getmsg missed messages in PICK_RINGUTC_Tue Sep 13 00:18:01 2022  x/statmgr Missed msg(s)  inst:1 mod:150 typ:3  region:1005
UTC_Tue Sep 13 00:18:01 2022  x/pick_FP (MOD_PICK_FP) module alive
UTC_Tue Sep 13 00:18:01 2022  x/binder tport_bufthr: Missed msg(s)  c1 m150 t8  Sequence gap, region:1005.

20220913_UTC_00:18:02 statmgr tport_getmsg missed messages in HYPO_RINGUTC_Tue Sep 13 00:18:02 2022  x/statmgr Missed msg(s)  inst:1 mod:150 typ:3  region:1015
UTC_Tue Sep 13 00:18:01 2022  x/binder tport_bufthr: Missed msg(s)  c1 m150 t8  Sequence gap, region:1005.

UTC_Tue Sep 13 00:18:02 2022  x/binder Missed msg(s)  inst:1 mod:150 typ:8  region:1020.

UTC_Tue Sep 13 00:18:02 2022  x/eqassemble Missed msg(s)  i1 m150 t8  region:1005.

UTC_Tue Sep 13 00:18:02 2022  x/binder Missed msg(s)  inst:1 mod:150 typ:8  region:1020.

UTC_Tue Sep 13 00:19:02 2022  x/pick_FP (MOD_PICK_FP) module dead
20220913_UTC_00:19:02 Statmgr: sent restart request for pick_FP pid 972239
20220913_UTC_00:24:24 statmgr tport_getmsg missed messages in PICK_RINGUTC_Tue Sep 13 00:24:24 2022  x/statmgr Missed msg(s)  inst:1 mod:150 typ:3  region:1005
UTC_Tue Sep 13 00:24:24 2022  x/pick_FP (MOD_PICK_FP) module alive
UTC_Tue Sep 13 00:24:24 2022  x/eqassemble Missed msg(s)  i1 m150 t8  region:1005.

S H

unread,
Oct 3, 2022, 4:51:35 PM10/3/22
to Earthworm Community Forum
Why would my Pick_FP not start with everything else? I ran sniffring HYPO_RING INST_UNKNOWN verbose and receive heartbeats from all the other modules, but regarding Pick_FP...

1664829717 Received INST_UNKNOWN MOD_STATMGR TYPE_PAGE <seq:  3> <Length:   162>
group: larva_test UTC_Mon Oct  3 20:41:57 2022  x/pick_FP (MOD_PICK_FP) module dead, PID unknown; needs human intervention. Page sent.

What is this larva_test btw?

I error above came after replacing the bin file with a different version from CentOS, it seems like Pick_FP never starts as I don't get any heartbeats from it according to sniffring. 
I'm also not sure why the PID is unknown, as in earthworm.d it is defined:  Module   MOD_PICK_FP           150

Thanks

S H

unread,
Oct 3, 2022, 6:09:43 PM10/3/22
to Earthworm Community Forum
Just as an experiment, I'm trying to use pick_ew instead of pick_FP. I recreated my .sta file using the older style. I changed the values in startstop_unix.d removing all references to pick_FP, and inserting the pick_ew references. When I start earthworm I can clearly see that pick_ew is running (opposed to pick_fp) in the status window. However, I'm still getting the "(MOD_PICK_FP) module dead, PID unknown; needs human intervention" error in my HYPO_RING. Why would this be? Also no mention of pick_ew in the HYPO_ring sniffer log.

Larry Baker

unread,
Oct 3, 2022, 7:34:18 PM10/3/22
to 'jona...@ntu.edu.sg' via Earthworm Community Forum, Larry Baker
Samuel,

I doubt you can mix binaries from different build of Earthworm.  At the very least, you should make sure the replacement bin file is 32-bit or 64-bit to match the rest of the bin files.  Use the "file" command on the bin files and make sure they are all the same.  Earthworm cannot tell when you mix 32-bit and 64-bit binaries, and they are definitely NOT compatible.


--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.

S H

unread,
Oct 4, 2022, 11:15:26 AM10/4/22
to Earthworm Community Forum
Thanks Larry, but not only was the CentOS pick_FP 64bit compatible, but the pick_ew is right from the original RHEL installation. Could there be an issue with RHEL 8.5 itself? I tried the earthworm memphis installation (v7.9) and the same thing happened where all the modules apart from pick_FP seem fine. I'm starting to think this is not a configuration issue within my earthworm installation. I think pick_FP (the process) is alive but pick_FP isn't actually functional, neither is pick_ew.

glenda...@gmail.com

unread,
Oct 4, 2022, 3:30:59 PM10/4/22
to Earthworm Community Forum
On the "(MOD_PICK_FP) module dead, PID unknown; needs human intervention" error
if your  pick_fp.d is sending a 30-sec heartbeat, set your "pick.fp.desc" to at least 60 or 120 sec ("tsec" in the desc file)
tsec: 120  page: 0  mail: 100
then restart the pick_fp module
Message has been deleted

S H

unread,
Oct 5, 2022, 12:23:09 PM10/5/22
to Earthworm Community Forum
I don't know why I am getting any errors regarding the pick_FP module as it should be completely disabled.

From before:
Just as an experiment, I'm trying to use pick_ew instead of pick_FP. I recreated my .sta file using the older style. I changed the values in startstop_unix.d removing all references to pick_FP, and inserting the pick_ew references. When I start earthworm I can clearly see that pick_ew is running (opposed to pick_fp) in the status window. However, I'm still getting the "(MOD_PICK_FP) module dead, PID unknown; needs human intervention" error in my HYPO_RING. Why would this be? Also no mention of pick_ew in the HYPO_ring sniffer log.

Stefan Lisowski

unread,
Oct 5, 2022, 1:00:53 PM10/5/22
to earthwo...@googlegroups.com, S H


On 10/5/22 12:23 PM, S H wrote:
> I don't know why I am getting any errors regarding the pick_FP module as
> it should be completely disabled.
>
> From before:
> Just as an experiment, I'm trying to use pick_ew instead of pick_FP. I
> recreated my .sta file using the older style. I changed the values in
> startstop_unix.d removing all references to pick_FP, and inserting the
> pick_ew references. When I start earthworm I can clearly see that
> pick_ew is running (opposed to pick_fp) in the status window. However,
> I'm still getting the "(MOD_PICK_FP) module dead, PID unknown; needs
> human intervention" error in my HYPO_RING. Why would this be? Also no

Because statmgr.d must have a .desc file listed that contains a module
name of MOD_PICK_FP.
>> UTC_Fri Sep 16 20:54:49 2022  x/pick_FP *(MOD_PICK_FP)
>> module dead, PID unknown; needs human intervention.*
>> Missed msg(s)  c1m150 t8  Sequence gap, region:1005.
>>
>> UTC_Tue Sep 13 00:18:02 2022  x/binder Missed msg(s)
>>  inst:1 mod:150 typ:8  region:1020.
>>
>> UTC_Tue Sep 13 00:18:02 2022  x/eqassemble Missed
>> msg(s)  i1m150 t8  region:1005.
>>
>> UTC_Tue Sep 13 00:18:02 2022  x/binder Missed msg(s)
>>  inst:1 mod:150 typ:8  region:1020.
>>
>> UTC_Tue Sep 13 00:19:02 2022  x/pick_FP (MOD_PICK_FP)
>> module dead
>> 20220913_UTC_00:19:02 Statmgr: sent restart request
>> forpick_FP pid 972239
>> https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>
> To post to this group, send an email to earthwo...@googlegroups.com
>
> To unsubscribe from this group, send an email to
> earthworm_for...@googlegroups.com
>
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en
> <http://groups.google.com/group/earthworm_forum?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to earthworm_for...@googlegroups.com
> <mailto:earthworm_for...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--








*To follow ISTI news and updates please subscribe to our newsletter
<https://www.isti.com/newsletter-sign-up>.*





S H

unread,
Oct 5, 2022, 1:16:15 PM10/5/22
to Earthworm Community Forum
Very good, small progress.

UTC_Wed Oct  5 17:05:36 2022  x/pick_ew (MOD_PICK_EW) module dead, PID unknown; needs human intervention.
20221005_UTC_17:14:44 network timeout (600s), reconnecting in 30s

Not sure what the timeout is about... but at least there's no reference to pick_FP anymore. Too bad pick_ew is acting similarly.

S H

unread,
Oct 7, 2022, 11:51:41 AM10/7/22
to Earthworm Community Forum
I am trying to compile from source as I'm thinking there's an issue with the binaries. Is Red Hat 8.6 x64 not supported? 
buildinfo: Unsupported Unix EARTHWORM-64 build of Oct  7 2022 09:34:50

 set PLATFORM env var to LINUX for GNU or Mac OS X systems
make: *** [Makefile:133: CHECK_PLATFORM_UNIX] Error 1


In my bash file I have the following line:   
# Use the Linux makefile.unix options
   export PLATFORM="LINUX"

Stefan Lisowski

unread,
Oct 10, 2022, 8:01:34 PM10/10/22
to earthwo...@googlegroups.com
Earthworm is known to compile on RHEL 8; check your environment
variables, and note you'll probably need to do:
yum install libtirpc libtirpc-devel
> https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com> <https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/earthworm_forum/f7ade983-2a84-47f5-859f-a50ec85cdeafn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >
> > --
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "Earthworm Community Forum" group.
> >
> > To post to this group, send an email to
> earthwo...@googlegroups.com
> >
> > To unsubscribe from this group, send an email to
> > earthworm_for...@googlegroups.com
> >
> > For more options, visit this group at
> > http://groups.google.com/group/earthworm_forum?hl=en
> <http://groups.google.com/group/earthworm_forum?hl=en>
> > <http://groups.google.com/group/earthworm_forum?hl=en
> <http://groups.google.com/group/earthworm_forum?hl=en>>
> >
> > ---
> > You received this message because you are subscribed to the
> Google
> > Groups "Earthworm Community Forum" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send
> > an email to earthworm_for...@googlegroups.com
> > <mailto:earthworm_for...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com> <https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/earthworm_forum/b514d90b-c036-418e-87f7-f1de5c81f8e1n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
>
>
>
>
>
>
>
>
> *To follow ISTI news and updates please subscribe to our newsletter
> "the isti letter" at https://www.isti.com/newsletter-sign-up
> <https://www.isti.com/newsletter-sign-up>
> <https://www.isti.com/newsletter-sign-up
> <https://www.isti.com/newsletter-sign-up>>.*
>
>
>
>
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>
> To post to this group, send an email to earthwo...@googlegroups.com
>
> To unsubscribe from this group, send an email to
> earthworm_for...@googlegroups.com
>
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en
> <http://groups.google.com/group/earthworm_forum?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to earthworm_for...@googlegroups.com
> <mailto:earthworm_for...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/earthworm_forum/701c8232-eecc-40e3-a785-61f781f12d7dn%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/701c8232-eecc-40e3-a785-61f781f12d7dn%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages