FYI: Strange MacOS failure mode because of Spotlight indexing, and solution

[rus...@ohsu.edu]

Hi everyone,

I just came across some strange behavior on MacOS that was pretty weird to debug, I thought I'd share it so it's documented somewhere in case anyone else runs into the same thing. It's not really a bug, in that WESTPA isn't doing anything wrong, just a weird interaction.

For what it's worth, I suspect this probably exists in some form or other when running any WESTPA simulation on a Mac, and will cause it to use/require way more disk space than the simulation actually needs.

System:

- NaCl association system under OpenMM, from the tutorials

- 10 walkers/bin

- 21 bins

- 150 max iterations

(Nothing wild)

Failure mode:

When leaving it running for many iterations, I noticed I'd often come back to either my Macbook having shut down altogether from a kernel panic, or to failed propagation messages from WESTPA. Inspecting the error logs more closely revealed out-of-disk-space messages.

Debugging/troubleshooting:

Obviously this just suggests I was out of disk space; however, I had plenty of free space relative to what this simulation should make (~200GB free, this whole run shouldn't take more than about 40GB). Monitoring my total drive space with `df` as the WE simulation ran didn't show anything abnormal at first, but I'd notice dozens of GB start to get eaten up over the course of just a few iterations, which was completely inconsistent with the amount of data the simulation was actually generating (maybe 250MB/iteration).

Furthermore, after terminating the simulation, the amount of used disk space would continue increasing, often by 50-100+ GB, for 5-10+ minutes after the end of the simulation.

Inspecting the activity monitor revealed huge amounts of disk IO by the mds_store process..

Diagnosis:

Turns out this is the result of a weird interaction where Spotlight is trying to index the many small files being quickly output by WESTPA. (mds_store is the metadata server for Spotlight.) It seems like it has a really hard time handling the huge amount of files generated by even a quite modest WESTPA run, and for some reason, this appears to blow up the Spotlight cache.

Because it's a result of indexing, and not the actual files written by WESTPA, the disk usage continued to increase after the simulation was terminated, and would eventually mostly free up, but at weird unpredictable times.

Solution:

Spotlight indexing can be disabled with `sudo mdutil -a -i off`.

I did try specifically excluding my WE directory from Spotlight indexing,  but either I did it wrong or Spotlight didn't respect that setting, because the issue persisted.

After disabling Spotlight indexing with the above command, maximum disk usage during the WE simulation went from ~150-200GB to ~40GB.

[Lillian Chong]

Thanks a lot for documenting this, John! Really helpful for our archives. -Lillian

--
You received this message because you are subscribed to the Google Groups "westpa-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/f61a6b34-70a6-486d-b649-3e5937c643e5n%40googlegroups.com.

--

Lillian T. Chong           
Associate Professor    
Department of Chemistry
University of Pittsburgh
219 Parkman Avenue
Pittsburgh, PA 15260
(412) 624-6026