Earthworm + NonLinLoc

[johnwe...@gmail.com]

Hi, I'm hoping someone can provide insight into Earthworm's implementation of NonLinLoc. I have a small network (8 stations, although many events only have 4 observations). I'm testing my workflow on 1 day of data with tankplayer. I'm using pick_ew, eqassemble, and nll_mgr in this particular test. In previous iterations I have used pick_FP, phaseworm, eqproc, and hyp2000_mgr.

Tankplayer runs successfully. Picks are made, sent through the eqassemble sausage, and locations are computed by NonLinLoc. All modules stay Alive throughout the process. However, NonLinLoc seems to hang at some point. Multiple NLLoc processes remain on my machine hours after tankplayer is done.

-------------
earthworm@lila:/opt/earthworm$ top

top - 20:26:37 up 77 days, 2:04, 4 users, load average: 2.03, 2.44, 2.61
Tasks: 409 total, 3 running, 405 sleeping, 1 stopped, 0 zombie
%Cpu(s): 25.0 us, 0.1 sy, 0.0 ni, 74.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 48150.3 total, 897.8 free, 2010.7 used, 45241.8 buff/cache
MiB Swap: 8192.0 total, 8150.3 free, 41.7 used. 45432.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2975179 earthwo+ 20 0 461836 170716 2868 R 100.0 0.3 75:26.55 NLLoc
2975878 earthwo+ 20 0 462488 171528 2884 R 99.7 0.3 72:43.93 NLLoc
2996539 earthwo+ 20 0 9772 4392 3540 R 0.3 0.0 0:00.09 top
2953390 earthwo+ 20 0 19428 10208 8328 S 0.0 0.0 0:00.48 systemd
...

2973148 earthwo+ 20 0 8544 5608 3672 S 0.0 0.0 0:00.09 bash
2974219 earthwo+ 20 0 7024 3728 3268 S 0.0 0.0 0:00.00 run_tankplayer.
2974237 earthwo+ 20 0 91504 7496 7064 S 0.0 0.0 0:00.16 startstop
2974238 earthwo+ 20 0 9188 2720 1684 S 0.0 0.0 0:00.14 statmgr
2974239 earthwo+ 20 0 11100 5620 5464 S 0.0 0.0 0:00.13 copystatus
-------------

Furthermore, I have about 38 .arc files in the NonLinLoc output folder (earthworm_temp/nll_mgr0) and only 16 in Earthworm's data/eqk/arc folder. I 

-------------
earthworm@lila:/opt/earthworm$ ls -alh /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/*.arc

-rw-rw-r-- 1 earthworm earthworm 665 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022829.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 663 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022830.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 665 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022831.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022832.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 663 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022833.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 662 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022834.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022835.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022836.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 636 Jan 30 19:10 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022837.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022838.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022839.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022840.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 637 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022841.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022842.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022843.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022844.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 636 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022845.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 662 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022846.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 630 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022847.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022848.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 0 Jan 30 19:11 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022849.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 662 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022920.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022921.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022922.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 662 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022923.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 662 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022924.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022925.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022926.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022927.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022928.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022929.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022930.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 664 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022931.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 665 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022932.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022933.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 629 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022934.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022935.sum.grid0.loc.arc

-rw-rw-r-- 1 earthworm earthworm 0 Jan 30 19:13 /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022936.sum.grid0.loc.arc

earthworm@lila:/opt/earthworm$ ls -alh run/data/eqk/arc

total 56K

drwxr-xr-x 2 earthworm earthworm 4.0K Jan 30 19:13 .

drwxr-xr-x 4 earthworm earthworm 4.0K Jan 21 21:16 ..

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 20190222-043049.qid00022838..arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 20190222-071443.qid00022839..arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:11 20190222-071647.qid00022842..arc

-rw-rw-r-- 1 earthworm earthworm 629 Jan 30 19:13 20190222--09-05.qid00000000..arc

-rw-rw-r-- 1 earthworm earthworm 637 Jan 30 19:11 20190222--10580.qid00000000..arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 20190222-195854.qid00022921..arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 20190222-211640.qid00022927..arc

-rw-rw-r-- 1 earthworm earthworm 626 Jan 30 19:13 20190222-211816.qid00022928..arc

-rw-rw-r-- 1 earthworm earthworm 661 Jan 30 19:13 20190222--21474.qid00000000..arc

-rw-rw-r-- 1 earthworm earthworm 630 Jan 30 19:11 20190222--2873-.qid00000000..arc

-rw-rw-r-- 1 earthworm earthworm 636 Jan 30 19:10 20190222--79989.qid00000000..arc

-rw-rw-r-- 1 earthworm earthworm 636 Jan 30 19:11 20190222--81315.qid00000000..arc 

-------------

Have other people observed this behavior? Could it be related to NonLinLoc not completing a location? E.g., part of the Earthworm terminal output:

-------------

... Reading observation file /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/nllMgrArcIn

Reading next set of observations (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 0) ...

... 4 observations read, 4 will be used for location (/opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022934.20190222.215857).

LOCGAU param CorrLen is zero, will not be used: 0.000000

Locating... (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 16  3DMem: used:0/avail:0/load:0) ...

Applying Octtree search within Grid 0:

INFO: EDT_otime_weight activated, OT_WT exceeds EDT_OT_WT_FLOOR.

OctTree num samples = 10000 / 20000

INFO: Min node size reached, terminating Octree search.

Octree oct_node_value_max= 4.356427e+20 oct_tree_integral= 0.000000e+00

OCTREE nInitial 4400 nEvaluated 10384 smallestNodeSide 0.009766/0.009766/0.009766 oct_tree_integral 0.000000e+00

INFO: EDT_otime_weight: ot_ml_std 92009201297866326016.000000

Finished location: /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022934.20190222.215857.grid0.loc.hyp

Reading next set of observations (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 0) ...

...end of observation file detected.

No more observation files.  1 events read,  1 events located,  0 locations completed.

nll_mgr: system command return <0>

-------------

  
I've attached my params/ dir and log/ dir as well as the NonLinLoc control file (and the INCLUDED gtsrce.in file). I've also attached a text file of the Earthworm terminal output,  a text file of the result of the Linux 'top' command, text from 'ls earthworm_temp/nll_mgr0' (NonLinLoc output), and text from 'ls run/data/eqk/arc'. I can send a link to zipped files of the whole Earthworm and NonLinLoc run directories. It's too big to attach here with the output files.


Thanks,

Jay Wellik
USGS Volcano Disaster Assistance Program

[Jean-Marie SAUREL]

Hello Jay,

I did not find any event that have more than 4 picks according to your terminal output.
With 4 phases only, the location should be pretty straight forward, except if they do not match at all a correct event.

Looking at first at nll_mgr log file, there is something strange.
Most of the lines seem to be about the same event and are quite strange.
I refer to those lines beginning with '20190222 -214 74.83'.
There should not contain a '-' minus in the hour section.

According to me, this points to a problem with NLLoc.
The terminal output also show some 'NLLoc: ERROR' statements from the first location, so indeed, NLLoc failed to locate the event for some reason.
The easiest way to troubleshoot this I guess would be to run manually NLLoc in your temp directory.
If I remember correctly, you just need to update the last.in file to point on the selected .arc input file (default point to nllMgrArcIn) and run "NLLoc last.in" in your terminal.

Maybe you can increase NLLoc verbosity (CONTROL statement in your input file).
Also, you can compare an NLLoc .arc or .hyp output file and try to understand why it places a '-' in the hour.


Last but not least, which version of NLLoc are you running ?

Regards.

Jean-Marie.

Le 30/01/2025 à 20:49, johnwe...@gmail.com a écrit :
> Hi, I'm hoping someone can provide insight into Earthworm's implementation of NonLinLoc. I have a small network (8 stations, although many events only have 4 observations). I'm testing my workflow on 1 day of data with tankplayer. I'm using pick_ew, eqassemble, and nll_mgr in this particular test. In previous iterations I have used pick_FP, phaseworm, eqproc, and hyp2000_mgr.
>
> Tankplayer runs successfully. Picks are made, sent through the eqassemble sausage, and locations are computed by NonLinLoc. All modules stay Alive throughout the process. However, NonLinLoc seems to hang at some point. Multiple NLLoc processes remain on my machine hours after tankplayer is done.
>

> /-------------
> //earthworm@lila:/opt/earthworm$ top/
> /top - 20:26:37 up 77 days, 2:04, 4 users, load average: 2.03, 2.44, 2.61

> Tasks: 409 total, 3 running, 405 sleeping, 1 stopped, 0 zombie
> %Cpu(s): 25.0 us, 0.1 sy, 0.0 ni, 74.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> MiB Mem : 48150.3 total, 897.8 free, 2010.7 used, 45241.8 buff/cache
> MiB Swap: 8192.0 total, 8150.3 free, 41.7 used. 45432.7 avail Mem
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

> *2975179 earthwo+ 20 0 461836 170716 2868 R 100.0 0.3 75:26.55 NLLoc
> 2975878 earthwo+ 20 0 462488 171528 2884 R 99.7 0.3 72:43.93 NLLoc *

> 2996539 earthwo+ 20 0 9772 4392 3540 R 0.3 0.0 0:00.09 top
> 2953390 earthwo+ 20 0 19428 10208 8328 S 0.0 0.0 0:00.48 systemd

> .../
> /2973148 earthwo+ 20 0 8544 5608 3672 S 0.0 0.0 0:00.09 bash

> 2974219 earthwo+ 20 0 7024 3728 3268 S 0.0 0.0 0:00.00 run_tankplayer.
> 2974237 earthwo+ 20 0 91504 7496 7064 S 0.0 0.0 0:00.16 startstop
> 2974238 earthwo+ 20 0 9188 2720 1684 S 0.0 0.0 0:00.14 statmgr
> 2974239 earthwo+ 20 0 11100 5620 5464 S 0.0 0.0 0:00.13 copystatus
> -------------
>

> /Furthermore, I have about 38 .arc files in the NonLinLoc output folder (earthworm_temp/nll_mgr0) and only 16 in Earthworm's data/eqk/arc folder. I 
> /
> //-------------
> //earthworm@lila:/opt/earthworm$ ls -alh /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/*.arc

> -rw-rw-r-- 1 earthworm earthworm 636 Jan 30 19:11 20190222--81315.qid00000000..arc/ //
> /-------------
>
> /

> Have other people observed this behavior? Could it be related to NonLinLoc not completing a location? E.g., part of the Earthworm terminal output:
>

> /-------------/
> /... Reading observation file /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/nllMgrArcIn/
>
> /Reading next set of observations (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 0) .../
>
> /... 4 observations read, 4 will be used for location (/opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022934.20190222.215857)./
>
> /LOCGAU param CorrLen is zero, will not be used: 0.000000/
>
> /Locating... (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 16  3DMem: used:0/avail:0/load:0) .../
>
> /Applying Octtree search within Grid 0:/
>
> /INFO: EDT_otime_weight activated, OT_WT exceeds EDT_OT_WT_FLOOR./
>
> /OctTree num samples = 10000 / 20000/
>
> /INFO: Min node size reached, terminating Octree search./
>
> /Octree oct_node_value_max= 4.356427e+20 oct_tree_integral= 0.000000e+00/
>
> /OCTREE nInitial 4400 nEvaluated 10384 smallestNodeSide 0.009766/0.009766/0.009766 oct_tree_integral 0.000000e+00/
>
> /INFO: EDT_otime_weight: ot_ml_std 92009201297866326016.000000/
>
> /Finished location: /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022934.20190222.215857.grid0.loc.hyp/
>
> /Reading next set of observations (Files open: Tot:6 Buf:0 Hdr:0  Alloc: 0) .../
>
> /...end of observation file detected./
>
> /*No more observation files.  1 events read,  1 events located,  0 locations completed.*/
>
> /nll_mgr: system command return <0>/
>
> /-------------
> /

>   
> I've attached my params/ dir and log/ dir as well as the NonLinLoc control file (and the INCLUDED gtsrce.in file). I've also attached a text file of the Earthworm terminal output,  a text file of the result of the Linux 'top' command, text from 'ls earthworm_temp/nll_mgr0' (NonLinLoc output), and text from 'ls run/data/eqk/arc'. I can send a link to zipped files of the whole Earthworm and NonLinLoc run directories. It's too big to attach here with the output files.
>
>
> Thanks,
>
> Jay Wellik
> USGS Volcano Disaster Assistance Program
>

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>  
> To post to this group, send an email to earthwo...@googlegroups.com
>  
> To unsubscribe from this group, send an email to
> earthworm_for...@googlegroups.com
>  
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en <http://groups.google.com/group/earthworm_forum?hl=en>
>
> ---
> You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com <mailto:earthworm_for...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com?utm_medium=email&utm_source=footer>.


--
--------------------------------------
Institut de Physique du Globe de Paris
Observatoires Volcanologiques et Sismologiques
Responsable technique
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5

[Anthony Lomax]

Hi Jay and all,

Thanks Jean-Marie, your observations and comments are helpful and seem to identify the problems.
I can add a little more on the behavior:

1. The message
   "NLLoc: ERROR: calc_maximum_likelihood_ot: reached end of increasing-time search limit..."
   Indicates that the EDT_OT_WT maximum likelihood OT grid search for optimal origin-time failed - there is apparently no optimum. This might indicate a location with too few and/or highly inconsistent (outlier) data - the location pdf may have no constraint in one or more directions.
2. The above problem gives a garbage origin time (this is kind of a NLL bug, additional logic and code to avoid reporting a garbage origin time is not present). NLL has another quirk that I think needs to be fixed (though it shows up amazingly rarely): if the origin time is just before a (certain?) day boundary and the earliest observation just after, the NLL output origin time will have negative hours, as this was easy to code in C 25 years ago while dealing with calendars and backing up in time "safely". [Does anyone have robust C code to add/subtract decimal seconds from a time and then correctly recover UTC year, month, ... decimal seconds from the resulting time?].

If the above is the cause of the problem, the quick fix might be to replace LOCMETH EDT_OT_WT with LOCMETH EDT_OT_WT in nlloc_stratovolcano.in. This change will skip the EDT_OT_WT maximum likelihood OT grid search. EDT with OT_WT helps improve the chances of getting a useful location when there are many outlier readings (e.g. with early instrumental readings), so *maybe* is not critical for modern pick sets ????????
Another quick fix is to change LOCMETH minimum_number_phases for location to > 4. 4 observations is minimal to constrain x,y,z,t, and one outlier can give a very poorly constrained to garbage solution. However, I usually run with 2 or 4 minimum_number_phases for location and do not recall seeing the calc_maximum_likelihood_ot very often. But when developing station corrections (e.g. SSST) and analyzing location results, I usually filter on ellipsoid semi-major axis, which probably effectively removes events with fewer readings, and I sometimes filter directly on number of readings.

Better fixes would be to 1) correct the negative hour origin time issue, and maybe add logic and error correction code for when the EDT_OT_WT maximum likelihood OT grid search fails, and 2) to make the EDT_OT_WT search more robust to error and in error output.

Fix 1) is becoming a priority for me, since with ML picking the problem should come up more frequently (though still seems rare in my ML relocation experience).

In any case, I hope the above helps, and we can continue further in resolving this issue...

Best regards,

Anthony

[johnwe...@gmail.com]

Hi Jean-Marie, Hi Anthony,

Thanks for your replies.

First, the simple stuff:

...

earthworm@lila:/opt/earthworm$ which NLLoc
/opt/NonLinLoc-dev/src/bin/NLLoc
earthworm@lila:/opt/earthworm$ NLLoc --version
NLLoc (NonLinLoc v7.1.04 05Sep2024)

...

I changed verbosity to level 5.

As Jean-Marie, suggested, I modified last.in and ran it manually from earthworm_temp/nll_mgr0/. Here are the relevant lines from the modified control file:

# Following line added automatically by nll_mgr:
#LOCFILES /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/nllMgrArcIn HYPOINVERSE_Y2000_ARC /opt/NonLinLoc-dev/nlloc_copahue/time/layer /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/0000022936 1

LOCFILES /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/nllMgrArcIn HYPOINVERSE_Y2000_ARC /opt/NonLinLoc-dev/nlloc_copahue/time/layer /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/test

# Following line added automatically by nll_mgr:
INCLUDE /opt/NonLinLoc-dev/nlloc_copahue/run/include/gtsrce.in

# Following line added automatically by nll_mgr:
LOCHYPOUT SAVE_HYPOINVERSE_Y2000_ARC

This ran successfully...
...

END_NLLOC
Finished location: /opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0/test.20190222.221653.grid0.loc.hyp
Reading next set of observations (Files open: Tot:4 Buf:0 Hdr:0 Alloc: 0) ...
Dummy message

...end of observation file detected.

No more observation files. 1 events read, 1 events located, 1 locations completed.

...  


The resulting .arc file is the same as the pre-existing last.arc
...
earthworm@lila:/opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0$ cat test.sum.grid0.loc.arc
201902222216491837S4360070 5948 1266-10004340 14 2000000000000000000 NLL2843 000 0 0003 0 0 0 0 D004X 22936
MAQ VV HHZ P?0201902222216 5308 -1118 0 0 0 0 9 0 0 0 14112600 0 -1211-990 0 934 0
FRO VV HHZ PU0201902222216 5333 0118 0 0 0 0 9 0 0 0 15512300 0 -1228-990 0 935 0
COP VV HHZ PU0201902222216 5389 -0118 0 0 0 0 9 0 0 0 19311500 0 -1230-990 0 935 0
HIT VV HHZ PU1201902222216 5418 9 47 0 0 0 0 9 0 0 0 20811100 0 -1211-990 0 369 0

earthworm@lila:/opt/NonLinLoc-dev/nlloc_copahue/earthworm_temp/nll_mgr0$ cat last.arc
201902222216491837S4360070 5948 1266-10004340 14 2000000000000000000 NLL2843 000 0 0003 0 0 0 0 D004X 22936
MAQ VV HHZ P?0201902222216 5308 -1118 0 0 0 0 9 0 0 0 14112600 0 -1211-990 0 934 0
FRO VV HHZ PU0201902222216 5333 0118 0 0 0 0 9 0 0 0 15512300 0 -1228-990 0 935 0
COP VV HHZ PU0201902222216 5389 -0118 0 0 0 0 9 0 0 0 19311500 0 -1230-990 0 935 0
HIT VV HHZ PU1201902222216 5418 9 47 0 0 0 0 9 0 0 0 20811100 0 -1211-990 0 369 0
...

Anthony, I didn't follow your suggestion for LOCMETH.

My original config was "LOCMETH EDT_OT_WT 9999.0 4 -1 -1 1.68 6 -1.0 1"

You suggest changing it to "LOCMETH EDT 9999.0 4 -1 -1 1.68 6 -1.0 1"?

I just noticed that my VpVs ratio in LOCMETH (1.68) was different than "EQVPVS 1.73". I've updated LOCMETH to also use 1.73. I doubt that could have caused that much of a problem though.


This may be less important, but do you have clarity on the order of processes? Is it correct to say that nll_mgr gets an event and writes earthworm_temp/nll_mgr0/nll_mgr.in (control file copied from the file specified in nll_mgr.d) and nllMgrArcIn (a HYPOINVERSE_Y2000_ARC formatted observation file). nll_mgr.in then gets executed and produces output in earthworm_temp/nll_mgr0 with the quake id as an identifer (e.g., 0000022936.sum.grid0.loc.arc) and last.arc. nll_mgr then reads one of the output files back into HYPO_RING where (in my case) it gets processed by ew2arc to populate run/data/eqk/arc? Or does nll_mgr pass things to HYPO_RING from memory? Some of the data/eqk/*.arc files have empty quakeids, but the files seem to contain good info (other than the date formatting issues):

earthworm@lila:/opt/earthworm/run/data/eqk/arc$ cat 20190222--81315.qid00000000..arc
20190222-813153760-02-330937S3840071 3557 -344-10004356 449999000000000000000000 NLL 1 000 0 0004 0 0 0 0 D004X 22845
COP VV HHZ PU1201902220721 2949 0200 0 0 0 0 9 0 0 0 439 1600 0 -1300-990 01110 0
FRO VV HHZ PU2201902220721 2894 0200 0 0 0 0 9 0 0 0 460 1600 0 -1296-990 01110 0
HIT VV HHZ PD0201902220721 2974 0 0 0 0 0 0 9 0 0 0 504 1600 0 -1303-990 0 0 0
MAQ VV HHZ PD0201902220721 2870 0 0 0 0 0 0 9 0 0 0 507 1600 0 -1115-990 0 0 0     

Thanks,

Jay

[Anthony Lomax]

Hi Jay,

So it looks like, for whatever reason, the NLL negative hour is corrupting the arc file - correct?

Can you send all the NLL output files including *.hyp for the last.in event, and the input observations?

Then I can perhaps confirm the base problem and NLL behavior.

> do you have clarity on the order of processes

I think this is a question for the Earthworm ffolks, or did I have something to do with setting up nll_mgr.in etc many years ago?

Best regards,

Anthony

[johnwe...@gmail.com]

Hi Anthony,

Attached are the files you've asked for (and a few more).

The folder run_A-edt-ot-wt/ includes the output from the run that I originally asked about. It includes
- All the NLL output files for last.in event and the input observations (what you asked for)
- All the NLL output files for quakeID 22829 (this is one of the events that does not have a corresponding arc file in the earthworm output folder)

Then, per your suggestion, I changed LOCMETH EDT_OT_WT to LOCMETH EDT. That allowed the code to run through all events very quickly (within a bathroom break). I get 113 events in the NLL output folder and 113 events in the earthworm output folder. However, the NLL .hyp files are Zero bytes, and I can't make sense of the format in the earthworm arc file (it's close, but I think corrupted).
The folder run_B-edt/ includes:
- ALL NLL output files for last.in event and input observations
- All NLL output files for quakeID 23055 (arbitrarily chosen example) and the corresponding earthworm arc file (earthworm_arc_result*23055.arc)

Here is the body of the earthworm arc file included in the folder:
earthworm@lila:/opt/earthworm/copahue_auto_nll/data/eqk/arc$ cat 20190222-231923.qid00023055..arc
201902222319236237S4287070 4075 -234-10004351 43   5000000000000000000   NLL 151  000   0   0004   0   0  0  0       D004X                   23055             
MAQ  VV  HHZ  PU0201902222319 2285   0  0    0   0   0      0 9  0   0   0 372 1600  0   -1 69-990  0   0   0    
FRO  VV  HHZ  PD0201902222319 2301   0  0    0   0   0      0 9  0   0   0 407 1600  0   -1 74-990  0   0   0    
HIT  VV  HHZ  PU0201902222319 2390   5200    0   0   0      0 9  0   0   0 427 1600  0   -1 63-990  02000   0    
COP  VV  HHZ  PD0201902222319 2357  -5200    0   0   0      0 9  0   0   0 445 1600  0   -1 72-990  02000   0

I'm starting to suspect my issue might have to do with my NLL configuration. On the run with LOCMETH EDT and verbosity level 5, I get several of these errors:

WARNING: oct-tree sample at (44.082031,17.402344,-0.752969) is outside of 2 travel time grids.
travel time grids.
WARNING: oct-tree sample at (44.096680,17.407227,-0.796914) is outside of 1 travel time grids.
WARNING: oct-tree sample at (44.096680,17.416992,-0.806680) is outside of 1 travel time grids.
WARNING: oct-tree sample at (44.096680,17.416992,-0.796914) is outside of 1 travel time grids.

[Jean-Marie SAUREL]

Hello Jay,

> This may be less important, but do you have clarity on the order of processes? Is it correct to say that nll_mgr gets an event and writes earthworm_temp/nll_mgr0/nll_mgr.in (control file copied from the file specified in nll_mgr.d) and nllMgrArcIn (a HYPOINVERSE_Y2000_ARC formatted observation file). nll_mgr.in then gets executed and produces output in earthworm_temp/nll_mgr0 with the quake id as an identifer (e.g., 0000022936.sum.grid0.loc.arc) and last.arc. nll_mgr then reads one of the output files back into HYPO_RING where (in my case) it gets processed by ew2arc to populate run/data/eqk/arc? Or does nll_mgr pass things to HYPO_RING from memory? Some of the data/eqk/*.arc files have empty quakeids, but the files seem to contain good info (other than the date formatting issues):

I think you are correct, that is the process.

The missing quake_id is strange as NLLoc should be transparent on this.
It is the binder that set-up the quake_id which is transported all along up to nll_mgr and that you can find in the nllMgrArcIn input file.
And it should also be in the last.arc output file that is read and sent to the HYPO_RING.

Regards.

Jean-Marie.

>
>
> earthworm@lila:/opt/earthworm/run/data/eqk/arc$ cat 20190222--81315.qid00000000..arc
> 20190222-813153760-02-330937S3840071 3557 -344-10004356 449999000000000000000000 NLL 1 000 0 0004 0 0 0 0 D004X 22845
> COP VV HHZ PU1201902220721 2949 0200 0 0 0 0 9 0 0 0 439 1600 0 -1300-990 01110 0
> FRO VV HHZ PU2201902220721 2894 0200 0 0 0 0 9 0 0 0 460 1600 0 -1296-990 01110 0
> HIT VV HHZ PD0201902220721 2974 0 0 0 0 0 0 9 0 0 0 504 1600 0 -1303-990 0 0 0
> MAQ VV HHZ PD0201902220721 2870 0 0 0 0 0 0 9 0 0 0 507 1600 0 -1115-990 0 0 0     
>
>
>
> Thanks,
>
> Jay
>
> On Monday, February 3, 2025 at 1:27:13 AM UTC-8 Anthony Lomax wrote:
>
> Hi Jay and all,
>
> Thanks Jean-Marie, your observations and comments are helpful and seem to identify the problems.
> I can add a little more on the behavior:
>

> 1. The message

> "NLLoc: ERROR: calc_maximum_likelihood_ot: reached end of increasing-time search limit..."
> Indicates that the EDT_OT_WT maximum likelihood OT grid search for optimal origin-time failed - there is apparently no optimum. This might indicate a location with too few and/or highly inconsistent (outlier) data - the location pdf may have no constraint in one or more directions.

> 2. The above problem gives a garbage origin time (this is kind of a NLL bug, additional logic and code to avoid reporting a garbage origin time is not present). NLL has another quirk that I think needs to be fixed (though it shows up amazingly rarely): if the origin time is just before a (certain?) day boundary and the earliest observation just after, the NLL output origin time will have negative hours, as this was easy to code in C 25 years ago while dealing with calendars and backing up in time "safely". [Does anyone have robust C code to add/subtract decimal seconds from a time and then correctly recover UTC year, month, ... decimal seconds from the resulting time?].
>
> If the above is the cause of the problem, the quick fix might be to replace LOCMETH EDT_OT_WT with LOCMETH EDT_OT_WT in nlloc_stratovolcano.in <http://nlloc_stratovolcano.in>. This change will skip the EDT_OT_WT maximum likelihood OT grid search. EDT with OT_WT helps improve the chances of getting a useful location when there are many outlier readings (e.g. with early instrumental readings), so *maybe* is not critical for modern pick sets ????????

> Another quick fix is to change LOCMETH minimum_number_phases for location to > 4. 4 observations is minimal to constrain x,y,z,t, and one outlier can give a very poorly constrained to garbage solution. However, I usually run with 2 or 4 minimum_number_phases for location and do not recall seeing the calc_maximum_likelihood_ot very often. But when developing station corrections (e.g. SSST) and analyzing location results, I usually filter on ellipsoid semi-major axis, which probably effectively removes events with fewer readings, and I sometimes filter directly on number of readings.
>
> Better fixes would be to 1) correct the negative hour origin time issue, and maybe add logic and error correction code for when the EDT_OT_WT maximum likelihood OT grid search fails, and 2) to make the EDT_OT_WT search more robust to error and in error output.
> Fix 1) is becoming a priority for me, since with ML picking the problem should come up more frequently (though still seems rare in my ML relocation experience).
>
> In any case, I hope the above helps, and we can continue further in resolving this issue...
>
> Best regards,
> Anthony
>
> On Friday, January 31, 2025 at 7:07:46 PM UTC+1 Jean-Marie Saurel wrote:
>
> Hello Jay,
>
> I did not find any event that have more than 4 picks according to your terminal output.
> With 4 phases only, the location should be pretty straight forward, except if they do not match at all a correct event.
>
> Looking at first at nll_mgr log file, there is something strange.
> Most of the lines seem to be about the same event and are quite strange.
> I refer to those lines beginning with '20190222 -214 74.83'.
> There should not contain a '-' minus in the hour section.
>
> According to me, this points to a problem with NLLoc.
> The terminal output also show some 'NLLoc: ERROR' statements from the first location, so indeed, NLLoc failed to locate the event for some reason.
> The easiest way to troubleshoot this I guess would be to run manually NLLoc in your temp directory.

> If I remember correctly, you just need to update the last.in <http://last.in> file to point on the selected .arc input file (default point to nllMgrArcIn) and run "NLLoc last.in <http://last.in>" in your terminal.

> > I've attached my params/ dir and log/ dir as well as the NonLinLoc control file (and the INCLUDED gtsrce.in <http://gtsrce.in> file). I've also attached a text file of the Earthworm terminal output,  a text file of the result of the Linux 'top' command, text from 'ls earthworm_temp/nll_mgr0' (NonLinLoc output), and text from 'ls run/data/eqk/arc'. I can send a link to zipped files of the whole Earthworm and NonLinLoc run directories. It's too big to attach here with the output files.

> >
> >
> > Thanks,
> >
> > Jay Wellik
> > USGS Volcano Disaster Assistance Program
> >
> > --
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Earthworm Community Forum" group.
> >  
> > To post to this group, send an email to earthwo...@googlegroups.com
> >  
> > To unsubscribe from this group, send an email to
> > earthworm_for...@googlegroups.com
> >  
> > For more options, visit this group at

> > http://groups.google.com/group/earthworm_forum?hl=en <http://groups.google.com/group/earthworm_forum?hl=en> <http://groups.google.com/group/earthworm_forum?hl=en <http://groups.google.com/group/earthworm_forum?hl=en>>

> >
> > ---
> > You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com <mailto:earthworm_for...@googlegroups.com>.

> > To view this discussion visit https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com> <https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/earthworm_forum/e7b64ecf-1d14-43c2-8de2-f8e514e3ea49n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> --------------------------------------
> Institut de Physique du Globe de Paris
> Observatoires Volcanologiques et Sismologiques
> Responsable technique

> +33 1 83 95 74 37 <tel:+33%201%2083%2095%2074%2037>
> +33 6 20 35 28 04 <tel:+33%206%2020%2035%2028%2004> (portable)

> 1 rue Jussieu
> 75238 Paris cédex 5
>

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Earthworm Community Forum" group.
>  
> To post to this group, send an email to earthwo...@googlegroups.com
>  
> To unsubscribe from this group, send an email to
> earthworm_for...@googlegroups.com
>  
> For more options, visit this group at
> http://groups.google.com/group/earthworm_forum?hl=en <http://groups.google.com/group/earthworm_forum?hl=en>
>
> ---
> You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com <mailto:earthworm_for...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/earthworm_forum/94dcdac4-887c-44c4-b429-a2d2bdb9f46dn%40googlegroups.com <https://groups.google.com/d/msgid/earthworm_forum/94dcdac4-887c-44c4-b429-a2d2bdb9f46dn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Victor Preatoni]

Hi Anthony,

Regarding "[Does anyone have robust C code to add/subtract decimal seconds from a time and then correctly recover UTC year, month, ... decimal seconds from the resulting time?].":

- Maybe moving from time_t to struct timespec could be a solution for that bug.

-------------

struct timespec represents a simple calendar time, or an elapsed time, with sub-second resolution. It is declared in time.h and has the following members:

time_t tv_sec
The number of whole seconds elapsed since the epoch (for a simple calendar time) or since some other starting point (for an elapsed time).

long int tv_nsec
The number of nanoseconds elapsed since the time given by the tv_sec member.
----------------

There is no difftime function for struct timespec, but I found this simple solution:
struct timespec diff_timespec(const struct timespec *time1, const struct timespec *time0) 

{
  struct timespec diff = {.tv_sec = time1->tv_sec - time0->tv_sec, .tv_nsec = time1->tv_nsec - time0->tv_nsec};
  if (diff.tv_nsec < 0) {
    diff.tv_nsec += 1000000000; // nsec/sec
    diff.tv_sec--;
  }
  return diff;
}

If you direct me into the right line of code, I can start investigating it.

Regards,

Victor

[Anthony Lomax]

Hi Victor,

Thanks for the tip - that got me motivated to investigate.

I think your solution still has the basic problem of not correcting for backing up past day, month or year boundaries, which may happen at:

    diff.tv_sec--;

That is why I did not address the problem initially (laziness 20+ years ago...)

So I worked out a solution with time_t and struct tm:

    if (phypo->sec < 0.0) {

        printf("DEBUG: reset_hypodatetime: phypo time in: %d-%d-%dT%d:%d:%lf\n", phypo->year, phypo->month, phypo->day, phypo->hour, phypo->min, phypo->sec);

        double hypo_sec = -phypo->sec;
        int int_sec_offset = (int) hypo_sec + 1;
        double dec_sec_corr = (double) int_sec_offset - hypo_sec;

struct tm hypo_time;
        hypo_time.tm_year = phypo->year - 1900;
hypo_time.tm_mon = phypo->month - 1;
hypo_time.tm_mday = phypo->day;
hypo_time.tm_hour = phypo->hour;
hypo_time.tm_min = phypo->min;
hypo_time.tm_sec = 0;

time_t time_seconds = mktime(&hypo_time);
        time_seconds += 3600 -int_sec_offset;

        struct tm *new_hypo_time = gmtime(&time_seconds);

        phypo->year = new_hypo_time->tm_year + 1900;
phypo->month = new_hypo_time->tm_mon + 1;
phypo->day = new_hypo_time->tm_mday;
phypo->hour = new_hypo_time->tm_hour;
phypo->min = new_hypo_time->tm_min;
phypo->sec = (double) new_hypo_time->tm_sec + dec_sec_corr;

        printf("DEBUG: reset_hypodatetime: phypo time out: %d-%d-%dT%d:%d:%lf\n", phypo->year, phypo->month, phypo->day, phypo->hour, phypo->min, phypo->sec);

    }

It seems to work on a basic test:

DEBUG: reset_hypodatetime: phypo time in: 2019-1-1T0:0:-3.451263
DEBUG: reset_hypodatetime: phypo time out: 2018-12-31T23:59:56.548737

But if anyone sees a problem...

I will propagate this to https://github.com/ut-beg-texnet/NonLinLoc/tree/dev after a bit of use and triple-checking here...

Thanks,

Ciao,

Anthony

[Victor Preatoni]

Hi Jay and everyone!

Did you have any progress on this? We are working with Julian Olivar, trying to implement an automated EW+NLL.

We just found out a little issue, and we would appreciate your opinions on possible workarounds:

- NLL control file is specific per volcano, so a single nll_mgr.d can only call a single NllCtrlFile, which in fact will only work on a single volcano. With hypo2000 we managed to configure multiple crustal models, and is now working fine. But with NLL we have no clues on how to manage a multi-volcano environment.

Any help appreciated :)

Victor Preatoni