Python API compatible?

[Marsel Murzabaev]

Hello,

I like qiime2 python api and want to do whole workflow within one jupyter notebook.

Is it possible to run plugins like picrust in python API? If yes, how to call for it?

Best regards,

Marsel

[Gavin Douglas]

Hey Marsel,

There isn’t a tutorial on that, but yes there are just a few key functions which could be called using custom code. There is a single master function, which you can see called and interacted with in this script:

https://github.com/gavinmdouglas/q2-picrust2/blob/master/q2_picrust2/_full_pipeline.py

To run individual steps you can see how that is done in these scripts:

https://github.com/picrust/picrust2/tree/master/scripts

Cheers,

Gavin

--
You received this message because you are subscribed to the Google Groups "picrust-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to picrust-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/picrust-users/ea88a896-5d87-434f-a7d7-3c77e4872b56n%40googlegroups.com.

[Marsel Murzabaev]

Thank you Douglas,

I am trying different ways to process data.

Now I have qiime2-2021.11 environment with picrust2 installed.

In main jupyter notebook, i call:

rep_seqs.export_data('test')

table.export_data('test')

from picrust2 import pipeline

from picrust2.default import (default_ref_dir, default_tables,
                              default_regroup_map, default_pathway_map)

pipeline_results = pipeline.full_pipeline(
    input_table='test/feature-table.biom',
    study_fasta='test/dna-sequences.fasta',
    processes=128,
    output_folder='jupyter_picrust',
    placement_tool='epa-ng',
    ref_dir=default_ref_dir,
    in_traits='EC',
    custom_trait_tables=None,
    marker_gene_table=default_tables['16S'],
    pathway_map=default_pathway_map,
    rxn_func='EC',
    no_pathways=False,
    regroup_map=default_regroup_map,
    no_regroup=False,
    stratified=False,
    max_nsti=2.0,
    min_reads=1,
    min_samples=1,
    hsp_method='mp',
    edge_exponent=0.5,
    min_align=0.8,
    skip_nsti=False,
    skip_minpath=False,
    no_gap_fill=False,
    coverage=False,
    per_sequence_contrib=False,
    wide_table=False,
    skip_norm=False,
    remove_intermediate=False,
    verbose='highly_verbose'
)

I feel there should be a way to run it with all default setting and feeding feature_table and rep_seqs directly without exporting but haven't figured out yet. Also not sure what is the right way to unpack results of function.

[Gavin Douglas]

Glad that’s working for you!

That function automatically writes out the output tables, but the path to the output files is returned by the function, as at the bottom of this script where the outputs are re-loaded in as BIOM tables: https://github.com/gavinmdouglas/q2-picrust2/blob/master/q2_picrust2/_full_pipeline.py

func_outputs is a dictionary with ids for each gene family category (e.g., KO and EC). The value is a list with the path to the unstratified table as the first index and (if specificed) the path to the stratified as the second index.

The pathway_outputs object is also a dictionary, but with ids as “unstrat_abun” and “strat_abun” for the paths to the unstratified and stratified pathway tables, respectively.

Hopefully that helps!

Cheers,

Gavin

To view this discussion on the web visit https://groups.google.com/d/msgid/picrust-users/e307ebdc-e5e4-40a9-b2d0-5a266b6a0445n%40googlegroups.com.

[Marsel Murzabaev]

I got a warning while running pipeline, did it finish properly?

Maybe I should try with smaller set first.

Warning message:

Inferring pathways from predicted EC

554 of 111323 ASVs were above the max NSTI cut-off of 2.0 and were removed from the downstream analyses. pathway_pipeline.py --input picrust_stratified/EC_metagenome_out/pred_metagenome_contrib.tsv.gz --out_dir picrust_stratified/pathways_out --map /home/asm/anaconda3/envs/qiime2-2021.11/lib/python3.8/site-packages/picrust2/default_files/pathway_mapfiles/metacyc_path2rxn_struc_filt_pro.txt --intermediate picrust_stratified/intermediate/pathways --proc 128 --regroup_map /home/asm/anaconda3/envs/qiime2-2021.11/lib/python3.8/site-packages/picrust2/default_files/pathway_mapfiles/ec_level4_to_metacyc_rxn.tsv --verbose /home/asm/anaconda3/envs/qiime2-2021.11/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py:702: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. warnings.warn( /home/asm/anaconda3/envs/qiime2-2021.11/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py:702: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. warnings.warn( Wrote predicted pathway abundances and coverages to picrust_stratified/pathways_out

[Gavin Douglas]

That is not a typical warning and it would definitely have me worried that an error occurred. You may be running into memory usage limitations with so many ASVs, which could be causing this problem.

Gavin

To view this discussion on the web visit https://groups.google.com/d/msgid/picrust-users/32a840f5-7827-488c-b144-3af19d8a9e48n%40googlegroups.com.

[Marsel Murzabaev]

This is definitely strange because I'm using server with 1TB ram. My guess is some processes do not scale well on this number of ASV. If it continues I will try next week with a smaller subset. 

You received this message because you are subscribed to a topic in the Google Groups "picrust-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/picrust-users/w8Kq5iz9Cck/unsubscribe.
To unsubscribe from this group and all its topics, send an email to picrust-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/picrust-users/BE649254-CB38-496C-8480-CD62BEE5D39E%40gmail.com.