runaway cylc suites -- how do I find out what happened [SEC=UNCLASSIFIED]

[Yi Xiao]

Hi, Cylc developers,

We recently have some suites disappeared from their routine running, vanished as if they have never been there. Is there a way to find out what happened to them if logs are also disappeared. cylc db appears to be in place.

Regards,

Xiao

[Hilary Oliver]

Hi Xiao,

Do you mean the suite server programs (suite daemons) disappeared - i.e. no longer running and visible with "ps" or "top"; and the suite (and task job?) logs disappeared too?  I've never seen that happen, so my first guess would be someone accidentally deleted them!  The suite databases, if they still exist, could tell you what the last task events were before this happened, but you need the suite server program log to see any pre-shutdown error messages.

Hilary

--

---
You received this message because you are subscribed to the Google Groups "cylc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cylc+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yi Xiao]

Hi, Hilary

I can ask for the suite db to be copied out, so that we can have a look. Is there a tool to exam the db file? 

Suites were manually restarted once the problem was discovered. Unfortunately no one checked at the time if any task from the suite was still running (very unlikely though, judging by how much time has passed). 

Regards,

Xiao

---

From: cy...@googlegroups.com <cy...@googlegroups.com> on behalf of Hilary Oliver <hilary....@gmail.com>
Sent: Monday, 10 December 2018 9:00 PM
To: cy...@googlegroups.com
Subject: Re: [cylc-dev] runaway cylc suites -- how do I find out what happened [SEC=UNCLASSIFIED]

 

[Hilary Oliver]

Suite DBs are sqlite serverless databases.  A great tool for viewing their contents is "sqlitebrowser": https://sqlitebrowser.org/   Otherwise try the sqlite3 command line client.

(However, I don't expect the DB content will be very informative in the scenario you describe - you really need the suite logs).

Hilary