Problem with mOTU version 2.5 and genus counts
142 views
Skip to first unread message
mette Joergensen
Sep 30, 2020, 4:56:34 AM9/30/20
to NGLess
NGLess - Google Grupper
Now I got mOTU version 2.5 up and running and it works fine except for when I'm trying to get genus level count tables (mOTU, kingdom, class etc works fine)
I get the following error:
--
Exiting after fatal error:
ESC[31mAn unhandled erorr occurred (this should not happen)!
If you can reproduce this issue, please run your script
with the --trace flag and report a bug (including the script and the trace) at
https://github.com/ngless-toolkit/ngless/issues
The error message was: `Logic error in StandardModules/parallel//complete
CallStack (from HasCallStack):
error, called at NGLess/StandardModules/Parallel.hs:403:32 in NGLess-1.1.1-Fsa6MI5xiN249ZJVS9rmPI:StandardModules.Parallel`
--
I have appended my script and the log file using trace.
I just tried with a couple of other fastq files and it failed in 2 out of ten files. Unfortunately I can't append the datafiles as it is patient data. Hopefully you can read something out of the log.
Any suggestions for how to fix the error will be appreciated.
Best,
Mette
Luis Pedro Coelho
Oct 7, 2020, 3:33:48 PM10/7/20
to NGLess List
After a bit of digging:
1) The underlying issue is that the motus tool does gives a broken (or at least very unexpected) result:
2) NGLess should be a bit more robust. If the motus main developers do not reply quickly, we may also just add a workaround on our wrapper (we already do a bit of normalization of the format)
Best
Luis
Luis Pedro Coelho | Fudan University | http://luispedro.org
--You received this message because you are subscribed to the Google Groups "NGLess" group.To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/3a6dd2b8-6ef0-477a-9d9a-aaf6d29a3248n%40googlegroups.com.Attachments:
- gut_profiler.txt
Message has been deleted
mette Joergensen
Oct 9, 2020, 7:40:21 AM10/9/20
to NGLess
Hi Luis
I tried to run mOTUv2.5 outside of NGLess and Ileibacterium was twice in the output for both the sample that failed and the sample that didn't failed in NGLess-. So I'm not sure that is the problem, but it is still an unexpected result from mOTU2.
.
Luis Pedro Coelho
Oct 10, 2020, 5:30:48 AM10/10/20
to NGLess List
I think NGLess will run fine if you run each sample independently, but when you try the collect() function, it errors out.
The error message is not good, but I will defend that erroring out is the right behaviour. Generally, in NGLess, we prefer to error loudly than to quietly give wrong results.
If you'll allow me a long email, here's the explanation:
Here's what collect() does
Example 1 (good)
File 1
A 1
B 2
C 2
File 2
A 0
B 3
C 1
collect() will put these together as
A 1 0
B 2 3
C 2 1
Example 2 (bad)
But when you have
File 1
A 1
B 2
B 3
C 0
File 2
A 0
B 2
B 1
C 2
collect() should fail because both of these are equally good solutions:
A 1 0
B 2 2
B 3 1
C 0 2
or
A 1 0
B 2 1
B 3 2
C 0 2
Maybe the first solution is more natural because it follows the file order, but it's very brittle as we resort the files internally. In effect, these count files are non-conformant and erroring out is correct (although, the particular error message is very awful in this case). Also, collect() will fill-in zeros when a key is missing, which is helpful in many context.
HTH
To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/87870509-fc5b-4fb8-a281-bd6889670aeen%40googlegroups.com.
mette Joergensen
Oct 10, 2020, 6:08:44 AM10/10/20
to NGLess
Make sense, it also seems to be failing on the largest sample which is were the collect is most likely to happen.
mette Joergensen
Oct 20, 2020, 3:19:48 AM10/20/20
to NGLess
Thanks for digging into the issue. Now I have corrected my own mOTU database (modifying the wrong Ileibacterium to Alterileibacterium) while waiting for the new
version and now the collect works fine.
version and now the collect works fine.
ullo...@googlemail.com
May 13, 2021, 11:06:21 AM5/13/21
to NGLess
Hi Mette,
how exactly did you fix the problem? I would much appreciate your advice since I'm running into the same issue with mOTUs v2 6
Best,
Ulrike
mette Joergensen
May 18, 2021, 10:22:45 AM5/18/21
to NGLess
Hi Ulrike
As far as I remember all I did was to change
specI_v3_10239 ref_mOTU_v25_10239 2 Bacteria 1239 Firmicutes186801 Clostridia 186802 Clostridiales 543314 Clostridiales Family XIII. Incertae Sedis 1980680 Ileibacterium 1870997 Ileibacterium massiliense
To
specI_v3_10239 ref_mOTU_v25_10239 2 Bacteria 1239 Firmicutes186801 Clostridia 186802 Clostridiales 543314 Clostridiales Family XIII. Incertae Sedis 1980680 Alterileibacterium 1870997 Ileibacterium massiliense
In the file motus.ngm/2.5/mOTUs_v2/db_mOTU/db_mOTU_taxonomy_ref-mOTUs.tsv
ullo...@googlemail.com
May 21, 2021, 1:14:04 AM5/21/21
to NGLess
Hi Mette,
Thank you very much! Unfortunately, it didn't work for me (v2.6). How did you find out which mOTU was problematic?
Xianghui Liu
Jul 8, 2021, 5:10:41 AM7/8/21
to NGLess
I want to ask, motu can run by itself vs run by ngless, are they giving the same results? ... ngless have a better summarization of results?
Renato Alves
Jul 8, 2021, 5:28:30 AM7/8/21
to ngl...@googlegroups.com
On 7/8/21 11:10 AM, Xianghui Liu wrote:
> I want to ask, motu can run by itself vs run by ngless, are they giving the same results?
If using motus 2.x, yes. If using motus 1.x no.
> I want to ask, motu can run by itself vs run by ngless, are they giving the same results?
motus 1.x is no longer recommended due to being rather old and outdated by now.
> ngless have a better summarization of results?
NGLess does not alter results from motus 2.x. When using collect(), results from individual samples are aggregated by NGLess.
Cheers,
Renato
On 7/8/21 11:10 AM, Xianghui Liu wrote:
> I want to ask, motu can run by itself vs run by ngless, are they giving the same results? ... ngless have a better summarization of results?
>
> 在2021年5月21日星期五 UTC+8 下午1:14:04<ullo...@googlemail.com> 写道：
>
> Hi Mette,
> Thank you very much! Unfortunately, it didn't work for me (v2.6). How did you find out which mOTU was problematic?
>
> mette Joergensen schrieb am Dienstag, 18. Mai 2021 um 16:22:45 UTC+2:
>
> Hi Ulrike
> As far as I remember all I did was to change
>
> specI_v3_10239ref_mOTU_v25_102392 Bacteria1239 Firmicutes186801 Clostridia186802 Clostridiales543314 Clostridiales Family XIII. Incertae Sedis1980680 Ileibacterium1870997 Ileibacterium massiliense
> To
>
> In the file motus.ngm/2.5/mOTUs_v2/db_mOTU/db_mOTU_taxonomy_ref-mOTUs.tsv
>
> torsdag den 13. maj 2021 kl. 17.06.21 UTC+2 skrev ullo...@googlemail.com:
>
> Hi Mette,
> how exactly did you fix the problem? I would much appreciate your advice since I'm running into the same issue with mOTUs v2 6
> Best,
> Ulrike
>
> mette Joergensen schrieb am Dienstag, 20. Oktober 2020 um 09:19:48 UTC+2:
>
> Thanks for digging into the issue. Now I have corrected my own mOTU database (modifying the wrong Ileibacterium to Alterileibacterium) while waiting for the new
> version and now the collect works fine.
>
> lørdag den 10. oktober 2020 kl. 12.08.44 UTC+2 skrev mette Joergensen:
>
> Make sense, it also seems to be failing on the largest sample which is were the collect is most likely to happen.
>
> lørdag den 10. oktober 2020 kl. 11.30.48 UTC+2 skrev lu...@luispedro.org:
>
> __
> In the file motus.ngm/2.5/mOTUs_v2/db_mOTU/db_mOTU_taxonomy_ref-mOTUs.tsv
>
> torsdag den 13. maj 2021 kl. 17.06.21 UTC+2 skrev ullo...@googlemail.com:
>
> Hi Mette,
> how exactly did you fix the problem? I would much appreciate your advice since I'm running into the same issue with mOTUs v2 6
> Best,
> Ulrike
>
> mette Joergensen schrieb am Dienstag, 20. Oktober 2020 um 09:19:48 UTC+2:
>
> Thanks for digging into the issue. Now I have corrected my own mOTU database (modifying the wrong Ileibacterium to Alterileibacterium) while waiting for the new
> version and now the collect works fine.
>
> lørdag den 10. oktober 2020 kl. 12.08.44 UTC+2 skrev mette Joergensen:
>
> Make sense, it also seems to be failing on the largest sample which is were the collect is most likely to happen.
>
> lørdag den 10. oktober 2020 kl. 11.30.48 UTC+2 skrev lu...@luispedro.org:
>
> I think NGLess will run fine/ if you run each sample independently/, but when you try the *collect()* function, it errors out.
>
> The error message is not good, but I will defend that erroring out is the right behaviour. Generally, in NGLess, we prefer to error loudly than to quietly give wrong results.
>
> If you'll allow me a long email, here's the explanation:
>
> Here's what /collect()/ does
> The error message is not good, but I will defend that erroring out is the right behaviour. Generally, in NGLess, we prefer to error loudly than to quietly give wrong results.
>
> If you'll allow me a long email, here's the explanation:
>
>
> *Example 1 (good)*
>
> File 1
>
> A 1
> B 2
> C 2
>
> File 2
>
> A 0
> B 3
> C 1
>
> /collect()/ will put these together as
> File 1
>
> A 1
> B 2
> C 2
>
> File 2
>
> A 0
> B 3
> C 1
>
>
> A 1 0
> B 2 3
> C 2 1
>
> *Example 2 (bad)*
> A 1 0
> B 2 3
> C 2 1
>
>
> But when you have
>
> File 1
>
> A 1
> B 2
> B 3
> C 0
>
> File 2
>
> A 0
> B 2
> B 1
> C 2
>
> /collect() should fail because both of these are equally good solutions:/
> But when you have
>
> File 1
>
> A 1
> B 2
> B 3
> C 0
>
> File 2
>
> A 0
> B 2
> B 1
> C 2
>
>
> A 1 0
> B 2 2
> B 3 1
> C 0 2
>
> or
>
> A 1 0
> B 2 1
> B 3 2
> C 0 2
>
> Maybe the first solution is more natural because it follows the file order, but it's very brittle as we resort the files internally. In effect, these count files are non-conformant and erroring out is correct (although, the particular error message is very awful in this case). Also, collect() will fill-in zeros when a key is missing, which is helpful in many context.
>
> HTH
> Luis
>
>
> Luis Pedro Coelho | Fudan University | http://luispedro.org <http://luispedro.org>
> A 1 0
> B 2 2
> B 3 1
> C 0 2
>
> or
>
> A 1 0
> B 2 1
> B 3 2
> C 0 2
>
> Maybe the first solution is more natural because it follows the file order, but it's very brittle as we resort the files internally. In effect, these count files are non-conformant and erroring out is correct (although, the particular error message is very awful in this case). Also, collect() will fill-in zeros when a key is missing, which is helpful in many context.
>
> HTH
> Luis
>
>
> https://orcid.org/0000-0002-9280-7885 <https://orcid.org/0000-0002-9280-7885>
>
>
> On Fri, 9 Oct 2020, at 1:40 PM, mette Joergensen wrote:
>> Hi Luis
>> I tried to run mOTUv2.5 outside of NGLess and Ileibacterium was twice in the output for both the sample that failed and the sample that didn't failed in NGLess-. So I'm not sure that is the problem, but it is still an unexpected result from mOTU2.
>> .
>>
>> onsdag den 7. oktober 2020 kl. 21.33.48 UTC+2 skrev lu...@luispedro.org:
>>
>> __
>
> On Fri, 9 Oct 2020, at 1:40 PM, mette Joergensen wrote:
>> Hi Luis
>> I tried to run mOTUv2.5 outside of NGLess and Ileibacterium was twice in the output for both the sample that failed and the sample that didn't failed in NGLess-. So I'm not sure that is the problem, but it is still an unexpected result from mOTU2.
>> .
>>
>> onsdag den 7. oktober 2020 kl. 21.33.48 UTC+2 skrev lu...@luispedro.org:
>>
>> After a bit of digging:
>>
>> 1) The underlying issue is that the motus tool does gives a broken (or at least very unexpected) result:
>>
>> https://github.com/motu-tool/mOTUs_v2/issues/61 <https://github.com/motu-tool/mOTUs_v2/issues/61>
>>
>> 1) The underlying issue is that the motus tool does gives a broken (or at least very unexpected) result:
>>
>>
>> 2) NGLess should be a bit more robust. If the motus main developers do not reply quickly, we may also just add a workaround on our wrapper (we already do a bit of normalization of the format)
>>
>> Best
>> Luis
>>
>> Luis Pedro Coelho | Fudan University | http://luispedro.org <http://luispedro.org>
>> 2) NGLess should be a bit more robust. If the motus main developers do not reply quickly, we may also just add a workaround on our wrapper (we already do a bit of normalization of the format)
>>
>> Best
>> Luis
>>
>> https://orcid.org/0000-0002-9280-7885 <https://orcid.org/0000-0002-9280-7885>
>>
>>
>> On Wed, 30 Sep 2020, at 11:00 AM, mette Joergensen wrote:
>>> I could't send the message when the ngl script was appended, so you get it as a text file here instead.
>>>
>>> onsdag den 30. september 2020 kl. 10.56.34 UTC+2 skrev mette Joergensen:
>>>
>>> NGLess - Google Grupper
>>>
>>>
>>> Now I got mOTU version 2.5 up and running and it works fine except for when I'm trying to get genus level count tables (mOTU, kingdom, class etc works fine)
>>>
>>> I get the following error:
>>> --
>>> Exiting after fatal error:
>>> ESC[31mAn unhandled erorr occurred (this should not happen)!
>>>
>>> If you can reproduce this issue, please run your script
>>> with the --trace flag and report a bug (including the script and the trace) at
>>> https://github.com/ngless-toolkit/ngless/issues <https://github.com/ngless-toolkit/ngless/issues>
>>
>> On Wed, 30 Sep 2020, at 11:00 AM, mette Joergensen wrote:
>>> I could't send the message when the ngl script was appended, so you get it as a text file here instead.
>>>
>>> onsdag den 30. september 2020 kl. 10.56.34 UTC+2 skrev mette Joergensen:
>>>
>>> NGLess - Google Grupper
>>>
>>>
>>> Now I got mOTU version 2.5 up and running and it works fine except for when I'm trying to get genus level count tables (mOTU, kingdom, class etc works fine)
>>>
>>> I get the following error:
>>> --
>>> Exiting after fatal error:
>>> ESC[31mAn unhandled erorr occurred (this should not happen)!
>>>
>>> If you can reproduce this issue, please run your script
>>> with the --trace flag and report a bug (including the script and the trace) at
>>>
>>> The error message was: `Logic error in StandardModules/parallel//complete
>>> CallStack (from HasCallStack):
>>> error, called at NGLess/StandardModules/Parallel.hs:403:32 in NGLess-1.1.1-Fsa6MI5xiN249ZJVS9rmPI:StandardModules.Parallel`
>>> --
>>> I have appended my script and the log file using trace.
>>> I just tried with a couple of other fastq files and it failed in 2 out of ten files. Unfortunately I can't append the datafiles as it is patient data. Hopefully you can read something out of the log.
>>> Any suggestions for how to fix the error will be appreciated.
>>> Best,
>>> Mette
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/3a6dd2b8-6ef0-477a-9d9a-aaf6d29a3248n%40googlegroups.com <https://groups.google.com/d/msgid/ngless/3a6dd2b8-6ef0-477a-9d9a-aaf6d29a3248n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>> The error message was: `Logic error in StandardModules/parallel//complete
>>> CallStack (from HasCallStack):
>>> error, called at NGLess/StandardModules/Parallel.hs:403:32 in NGLess-1.1.1-Fsa6MI5xiN249ZJVS9rmPI:StandardModules.Parallel`
>>> --
>>> I have appended my script and the log file using trace.
>>> I just tried with a couple of other fastq files and it failed in 2 out of ten files. Unfortunately I can't append the datafiles as it is patient data. Hopefully you can read something out of the log.
>>> Any suggestions for how to fix the error will be appreciated.
>>> Best,
>>> Mette
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>>>
>>>
>>> *Attachments:*
>>>
>>> * gut_profiler.txt
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/87870509-fc5b-4fb8-a281-bd6889670aeen%40googlegroups.com <https://groups.google.com/d/msgid/ngless/87870509-fc5b-4fb8-a281-bd6889670aeen%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "NGLess" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "NGLess" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ngless+un...@googlegroups.com <mailto:ngless+un...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google Groups "NGLess" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ngless/b08b4f1d-e6d9-45e5-9f77-36830cf949a2n%40googlegroups.com <https://groups.google.com/d/msgid/ngless/b08b4f1d-e6d9-45e5-9f77-36830cf949a2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Renato Alves, Dr. rer. nat.
EMBL Bio-IT Community manager - https://bio-it.embl.de
HD-HuB (de.NBI, ELIXIR-DE), EMBL Heidelberg, Germany
Email: renato...@embl.de
Github/Gitlab: @unode - Twitter: @renato_alvs
ullo...@googlemail.com
Oct 22, 2022, 5:30:52 PM10/22/22
to NGLess
Hi Luis, hi Renato,
I have tried to setup mOTUs v3 recently following the instructions on https://ngless.readthedocs.io/en/latest/modules.html, but I failed. Would it be easily manageable for you to create the module? I guess this might help a lot tackling the problems with mOTUs 2.5 and 2.6 as well as the classification might have improved a lot as well, right? Sorry to bother and maybe I have requested this already.
Best wishes,
Ulrike
Reply all
Reply to author
Forward
0 new messages