What's happening when there's an apparent time delay in shots being saved?

[eac...@columbia.edu]

I had this problem on Sunday and it took a long time to figure out what was going on, so I wanted to post my symptoms/eventual diagnosis and solution to hopefully save others the headache.

Symptoms: 

• The pipeline visualizations update with each shot, with an image, F(Q), and G(r) that seems consistent with what we expect for our sample.
  
• However, our tiffs and metadata do not immediately appear in the folder we expect them to. 
  
• The terminal running the pipeline displays the message 

"saving file at preamble/samplename/meta/filename.yml 

saving file at preamble/samplename/dark_sub/filename.tiff 

saving file at preamble/samplename/iq/filename.iq"

 

etc, but the folders inside of preamble/samplename/ are empty.

• After a few minutes and a few more shots, a .tiff appears in preamble/samplename/dark_sub (and the corresponding metadata, iq, fq, grs coming off the pipeline appear in their respective folders as well.)
  
• After a while I open up two .tiffs of the sampleA, both in the sampleA folder, but one looks suspiciously like sampleB.

Investigation:

• I looked at the db entries and discovered the following:
• I had called xrun eight times: three times with the number corresponding to sampleA and five times with the number corresponding to sampleB.
• Only six tiffs had saved at this point.
• the db start docs corresponded to my calls to xrun -- e.g. if 8 calls ago, at 4 PM, I called xrun(numberofsampleA, p), db[-8] shows sampleA in the "sample name" entry and 4 PM as the timestamp.
• The metadata returned by db[-1] and db[-2] don't correspond to any existing files.
• I took three shots of sampleA, and then moved on to sampleB. Integrating the three shots of sampleA and comparing, I noticed only the first was actually sampleA, and the other two were sampleB.

Diagnosis:

• Each time I took a shot, the .tiff was immediately saving, but was associated with all of the metadata from 2 shots ago and was being saved in the folder corresponding to the metadata from 2 shots ago.

Hot fix:

• Pretty simple: I restarted the pipeline and this all stopped. But it was really hard to figure out what was actually going on, and I have no idea when this actually started and how many shots ended up mismatched before I noticed this.

[Simon Billinge]

Thanks Liz, good work!

--
You received this message because you are subscribed to the Google Groups "xpd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xpd-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xpd-users/6617e9ba-32b4-4a7b-878f-0e41e1b3a750%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.