Problem with mOTU version 2.5 and genus counts

142 views
Skip to first unread message

mette Joergensen

unread,
Sep 30, 2020, 4:56:34 AM9/30/20
to NGLess
NGLess - Google Grupper

Now I got mOTU version 2.5 up and running and it works fine except for when I'm trying to get genus level count tables (mOTU, kingdom, class etc works fine)

I get the following error:
--
Exiting after fatal error:
ESC[31mAn unhandled erorr occurred (this should not happen)!

        If you can reproduce this issue, please run your script
        with the --trace flag and report a bug (including the script and the trace) at
                https://github.com/ngless-toolkit/ngless/issues

The error message was: `Logic error in StandardModules/parallel//complete
CallStack (from HasCallStack):
  error, called at NGLess/StandardModules/Parallel.hs:403:32 in NGLess-1.1.1-Fsa6MI5xiN249ZJVS9rmPI:StandardModules.Parallel`

--
I have appended my script and the log file using trace.
I just tried with a couple of other fastq files and it failed in 2 out of ten files. Unfortunately I can't append the datafiles as it is patient data. Hopefully you can read something out of the log.
Any suggestions for how to fix the error will be appreciated.
Best,
Mette

sample_4.log

mette Joergensen

unread,
Sep 30, 2020, 5:00:07 AM9/30/20
to NGLess
I could't send the message when the ngl script was appended, so you get it as a text file here instead.
gut_profiler.txt

Luis Pedro Coelho

unread,
Oct 7, 2020, 3:33:48 PM10/7/20
to NGLess List
After a bit of digging:

1) The underlying issue is that the motus tool does gives a broken (or at least very unexpected) result:


2) NGLess should be a bit more robust. If the motus main developers do not reply quickly, we may also just add a workaround on our wrapper (we already do a bit of normalization of the format)

Best
Luis

Luis Pedro Coelho | Fudan University | http://luispedro.org
--
You received this message because you are subscribed to the Google Groups "NGLess" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.


Attachments:
  • gut_profiler.txt

Message has been deleted

mette Joergensen

unread,
Oct 9, 2020, 7:40:21 AM10/9/20
to NGLess
Hi Luis
I tried to run mOTUv2.5 outside of NGLess and Ileibacterium was twice in the output for both the sample that failed and the sample that didn't failed in NGLess-. So I'm not sure that is the problem, but it is still an unexpected result from mOTU2.
.


Luis Pedro Coelho

unread,
Oct 10, 2020, 5:30:48 AM10/10/20
to NGLess List
I think NGLess will run fine if you run each sample independently, but when you try the collect() function, it errors out.

The error message is not good, but I will defend that erroring out is the right behaviour. Generally, in NGLess, we prefer to error loudly than to quietly give wrong results.

If you'll allow me a long email, here's the explanation:

Here's what collect() does

Example 1 (good)

File 1

A 1
B 2
C 2

File 2

A 0
B 3
C 1

collect() will put these together as

A 1 0
B 2 3
C 2 1

Example 2 (bad)

But when you have

File 1

A 1
B 2
B 3
C 0

File 2

A 0
B 2
B 1
C 2

collect() should fail because both of these are equally good solutions:

A 1 0
B 2 2
B 3 1
C 0 2

or

A 1 0
B 2 1
B 3 2
C 0 2

Maybe the first solution is more natural because it follows the file order, but it's very brittle as we resort the files internally. In effect, these count files are non-conformant and erroring out is correct (although, the particular error message is very awful in this case). Also, collect() will fill-in zeros when a key is missing, which is helpful in many context.

HTH
Luis


Luis Pedro Coelho | Fudan University | http://luispedro.org


mette Joergensen

unread,
Oct 10, 2020, 6:08:44 AM10/10/20
to NGLess
Make sense, it also seems to be failing on the largest sample which is were the collect is most likely to happen.

mette Joergensen

unread,
Oct 20, 2020, 3:19:48 AM10/20/20
to NGLess
Thanks for digging into the issue. Now I have corrected my own mOTU database (modifying the wrong Ileibacterium to Alterileibacterium) while waiting for the new
version and now the collect works fine.

ullo...@googlemail.com

unread,
May 13, 2021, 11:06:21 AM5/13/21
to NGLess
Hi Mette,
how exactly did you fix the problem? I would much appreciate your advice since I'm running into the same issue with mOTUs v2 6
Best,
Ulrike

mette Joergensen

unread,
May 18, 2021, 10:22:45 AM5/18/21
to NGLess
Hi Ulrike 
As far as I remember all I did was to change 

specI_v3_10239 ref_mOTU_v25_10239 2 Bacteria 1239 Firmicutes186801 Clostridia 186802 Clostridiales 543314 Clostridiales Family XIII. Incertae Sedis 1980680 Ileibacterium 1870997 Ileibacterium massiliense
To
specI_v3_10239 ref_mOTU_v25_10239 2 Bacteria 1239 Firmicutes186801 Clostridia 186802 Clostridiales 543314 Clostridiales Family XIII. Incertae Sedis 1980680 Alterileibacterium 1870997 Ileibacterium massiliense

In the file motus.ngm/2.5/mOTUs_v2/db_mOTU/db_mOTU_taxonomy_ref-mOTUs.tsv

ullo...@googlemail.com

unread,
May 21, 2021, 1:14:04 AM5/21/21
to NGLess
Hi Mette,
Thank you very much! Unfortunately, it didn't work for me (v2.6). How did you find out which mOTU was problematic?

Xianghui Liu

unread,
Jul 8, 2021, 5:10:41 AM7/8/21
to NGLess
I want to ask, motu can run by itself vs run by ngless,  are they giving the same results? ... ngless have a better  summarization of results?  

Renato Alves

unread,
Jul 8, 2021, 5:28:30 AM7/8/21
to ngl...@googlegroups.com
On 7/8/21 11:10 AM, Xianghui Liu wrote:
> I want to ask, motu can run by itself vs run by ngless, are they giving the same results?

If using motus 2.x, yes. If using motus 1.x no.
motus 1.x is no longer recommended due to being rather old and outdated by now.

> ngless have a better summarization of results?

No.
NGLess does not alter results from motus 2.x. When using collect(), results from individual samples are aggregated by NGLess.

Cheers,
Renato

On 7/8/21 11:10 AM, Xianghui Liu wrote:
> I want to ask, motu can run by itself vs run by ngless,  are they giving the same results? ... ngless have a better  summarization of results?
>
> 在2021年5月21日星期五 UTC+8 下午1:14:04<ullo...@googlemail.com> 写道:
>
> Hi Mette,
> Thank you very much! Unfortunately, it didn't work for me (v2.6). How did you find out which mOTU was problematic?
>
> mette Joergensen schrieb am Dienstag, 18. Mai 2021 um 16:22:45 UTC+2:
>
> Hi Ulrike
> As far as I remember all I did was to change
>
> specI_v3_10239ref_mOTU_v25_102392 Bacteria1239 Firmicutes186801 Clostridia186802 Clostridiales543314 Clostridiales Family XIII. Incertae Sedis1980680 Ileibacterium1870997 Ileibacterium massiliense
> To
> specI_v3_10239ref_mOTU_v25_102392 Bacteria1239 Firmicutes186801 Clostridia186802 Clostridiales543314 Clostridiales Family XIII. Incertae Sedis1980680 Alterileibacterium1870997 Ileibacterium massiliense
>
> In the file motus.ngm/2.5/mOTUs_v2/db_mOTU/db_mOTU_taxonomy_ref-mOTUs.tsv
>
> torsdag den 13. maj 2021 kl. 17.06.21 UTC+2 skrev ullo...@googlemail.com:
>
> Hi Mette,
> how exactly did you fix the problem? I would much appreciate your advice since I'm running into the same issue with mOTUs v2 6
> Best,
> Ulrike
>
> mette Joergensen schrieb am Dienstag, 20. Oktober 2020 um 09:19:48 UTC+2:
>
> Thanks for digging into the issue. Now I have corrected my own mOTU database (modifying the wrong Ileibacterium to Alterileibacterium) while waiting for the new
> version and now the collect works fine.
>
> lørdag den 10. oktober 2020 kl. 12.08.44 UTC+2 skrev mette Joergensen:
>
> Make sense, it also seems to be failing on the largest sample which is were the collect is most likely to happen.
>
> lørdag den 10. oktober 2020 kl. 11.30.48 UTC+2 skrev lu...@luispedro.org:
>
> __
> I think NGLess will run fine/ if you run each sample independently/, but when you try the *collect()* function, it errors out.
>
> The error message is not good, but I will defend that erroring out is the right behaviour. Generally, in NGLess, we prefer to error loudly than to quietly give wrong results.
>
> If you'll allow me a long email, here's the explanation:
>
> Here's what /collect()/ does
>
> *Example 1 (good)*
>
> File 1
>
> A 1
> B 2
> C 2
>
> File 2
>
> A 0
> B 3
> C 1
>
> /collect()/ will put these together as
>
> A 1 0
> B 2 3
> C 2 1
>
> *Example 2 (bad)*
>
> But when you have
>
> File 1
>
> A 1
> B 2
> B 3
> C 0
>
> File 2
>
> A 0
> B 2
> B 1
> C 2
>
> /collect() should fail because both of these are equally good solutions:/
>
> A 1 0
> B 2 2
> B 3 1
> C 0 2
>
> or
>
> A 1 0
> B 2 1
> B 3 2
> C 0 2
>
> Maybe the first solution is more natural because it follows the file order, but it's very brittle as we resort the files internally. In effect, these count files are non-conformant and erroring out is correct (although, the particular error message is very awful in this case). Also, collect() will fill-in zeros when a key is missing, which is helpful in many context.
>
> HTH
> Luis
>
>
> Luis Pedro Coelho | Fudan University | http://luispedro.org <http://luispedro.org>
> https://orcid.org/0000-0002-9280-7885 <https://orcid.org/0000-0002-9280-7885>
>
>
> On Fri, 9 Oct 2020, at 1:40 PM, mette Joergensen wrote:
>> Hi Luis
>> I tried to run mOTUv2.5 outside of NGLess and Ileibacterium was twice in the output for both the sample that failed and the sample that didn't failed in NGLess-. So I'm not sure that is the problem, but it is still an unexpected result from mOTU2.
>> .
>>
>> onsdag den 7. oktober 2020 kl. 21.33.48 UTC+2 skrev lu...@luispedro.org:
>>
>> __
>> After a bit of digging:
>>
>> 1) The underlying issue is that the motus tool does gives a broken (or at least very unexpected) result:
>>
>> https://github.com/motu-tool/mOTUs_v2/issues/61 <https://github.com/motu-tool/mOTUs_v2/issues/61>
>>
>> 2) NGLess should be a bit more robust. If the motus main developers do not reply quickly, we may also just add a workaround on our wrapper (we already do a bit of normalization of the format)
>>
>> Best
>> Luis
>>
>> Luis Pedro Coelho | Fudan University | http://luispedro.org <http://luispedro.org>
>> https://orcid.org/0000-0002-9280-7885 <https://orcid.org/0000-0002-9280-7885>
>>
>>
>> On Wed, 30 Sep 2020, at 11:00 AM, mette Joergensen wrote:
>>> I could't send the message when the ngl script was appended, so you get it as a text file here instead.
>>>
>>> onsdag den 30. september 2020 kl. 10.56.34 UTC+2 skrev mette Joergensen:
>>>
>>> NGLess - Google Grupper
>>>
>>>
>>> Now I got mOTU version 2.5 up and running and it works fine except for when I'm trying to get genus level count tables (mOTU, kingdom, class etc works fine)
>>>
>>> I get the following error:
>>> --
>>> Exiting after fatal error:
>>> ESC[31mAn unhandled erorr occurred (this should not happen)!
>>>
>>>         If you can reproduce this issue, please run your script
>>>         with the --trace flag and report a bug (including the script and the trace) at
>>> https://github.com/ngless-toolkit/ngless/issues <https://github.com/ngless-toolkit/ngless/issues>
>>>
>>> The error message was: `Logic error in StandardModules/parallel//complete
>>> CallStack (from HasCallStack):
>>>   error, called at NGLess/StandardModules/Parallel.hs:403:32 in NGLess-1.1.1-Fsa6MI5xiN249ZJVS9rmPI:StandardModules.Parallel`
>>> --
>>> I have appended my script and the log file using trace.
>>> I just tried with a couple of other fastq files and it failed in 2 out of ten files. Unfortunately I can't append the datafiles as it is patient data. Hopefully you can read something out of the log.
>>> Any suggestions for how to fix the error will be appreciated.
>>> Best,
>>> Mette
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/3a6dd2b8-6ef0-477a-9d9a-aaf6d29a3248n%40googlegroups.com <https://groups.google.com/d/msgid/ngless/3a6dd2b8-6ef0-477a-9d9a-aaf6d29a3248n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>
>>>
>>> *Attachments:*
>>>
>>> * gut_profiler.txt
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/87870509-fc5b-4fb8-a281-bd6889670aeen%40googlegroups.com <https://groups.google.com/d/msgid/ngless/87870509-fc5b-4fb8-a281-bd6889670aeen%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups "NGLess" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com <mailto:ngless+un...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/b08b4f1d-e6d9-45e5-9f77-36830cf949a2n%40googlegroups.com <https://groups.google.com/d/msgid/ngless/b08b4f1d-e6d9-45e5-9f77-36830cf949a2n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Renato Alves, Dr. rer. nat.
EMBL Bio-IT Community manager - https://bio-it.embl.de
HD-HuB (de.NBI, ELIXIR-DE), EMBL Heidelberg, Germany

Email: renato...@embl.de
Github/Gitlab: @unode - Twitter: @renato_alvs

ullo...@googlemail.com

unread,
Oct 22, 2022, 5:30:52 PM10/22/22
to NGLess
Hi Luis, hi Renato,
I have tried to setup mOTUs v3 recently following the instructions on https://ngless.readthedocs.io/en/latest/modules.html, but I failed. Would it be easily manageable for you to create the module? I guess this might help a lot tackling the problems with mOTUs 2.5 and 2.6 as well as the classification might have improved a lot as well, right? Sorry to bother and maybe I have requested this already.
Best wishes,
Ulrike 

Reply all
Reply to author
Forward
0 new messages