Log file issues when resuming from checkpoint

Kenta Renard

unread,

Dec 31, 2024, 6:14:28 PM12/31/24

to raxml

Dear All,

I noticed something in my log files when restarting from checkpoint files. I am inferring 30 ML trees and all have roughly the same log-likelihood (around -80000 or so). When I re-start from the checkpoint for an unfinished job, one tree finished almost immediately with a log-likelihood much worse than the already inferred tree set (around -93000). Is this a bug just with the log file or is a tree with the log-likelihood actually being inferred as part of the tree set?

I am using the latest version of RAxML-NG.

Best wishes,

Kenta

Oleksiy Kozlov

unread,

Jan 6, 2025, 6:37:34 PMJan 6

to ra...@googlegroups.com

Dear Kenta,

please attach your log file.

Thanks,
Oleksiy

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/raxml/
> a0035b82-24f6-4761-889a-09361d2ed54fn%40googlegroups.com <https://groups.google.com/d/msgid/raxml/
> a0035b82-24f6-4761-889a-09361d2ed54fn%40googlegroups.com?utm_medium=email&utm_source=footer>.

Message has been deleted

Oleksiy Kozlov

unread,

Feb 18, 2025, 12:09:19 PMFeb 18

to ra...@googlegroups.com

Dear Kenta,

sorry for the late response, and thanks for sending the files!

This is indeed a bug in checkpointing which happens under very specific conditions:

If you restart with fewer --workers than in original run AND some workers has inferred all trees
assigned to them before interruption, then after a restart the first worker will have a stale
"finished" state, and hence will skip the topology optimization. This will result in a much lower
likelihood, exactly as you observed.

I have now fixed this bug in the dev branch of raxml-ng. Luckily, few users would ever be affected
by this, and even then it would likely not compromise the final result since only a single tree is
affected, and one of the remaining trees will be selected as the best ML tree.

Nevertheless, thanks for reporting!

Best,
Oleksiy

On 25.01.25 00:51, Kenta Renard wrote:
> Dear Oleksiy,
>
> Please find attached an example log file (not from the same dataset I described, but the issue is
> the same). When I checked the pairwise RF distances in the tree set, it was clear that the tree with
> the very bad likelihood is included as part of the .mlTrees file.
>
> Best wishes,
> Kenta

>
> On Mon, 6 Jan 2025 at 23:37, Oleksiy Kozlov <alexei...@gmail.com
> <mailto:alexei...@gmail.com>> wrote:
>
> Dear Kenta,
>
> please attach your log file.
>
> Thanks,
> Oleksiy
>
> On 01.01.25 00:14, Kenta Renard wrote:
> > Dear All,
> >
> > I noticed something in my log files when restarting from checkpoint files. I am inferring 30 ML
> > trees and all have roughly the same log-likelihood (around -80000 or so). When I re-start
> from the
> > checkpoint for an unfinished job, one tree finished almost immediately with a log-likelihood
> much
> > worse than the already inferred tree set (around -93000). Is this a bug just with the log
> file or is
> > a tree with the log-likelihood actually being inferred as part of the tree set?
> >
> > I am using the latest version of RAxML-NG.
> >
> > Best wishes,
> > Kenta
> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/raxml/

> CAAii4hmknZrLR2Fu1MmXWEvxMg0KvaXUEgCjug3q70exny7Sww%40mail.gmail.com <https://groups.google.com/d/
> msgid/raxml/CAAii4hmknZrLR2Fu1MmXWEvxMg0KvaXUEgCjug3q70exny7Sww%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

Reply all

Reply to author

Forward