[QIIME 1.9.0] biom add-metadata

[Marco Severgnini]

Hello everybody, I'm trying to add metadata to a JSON-formatted BIOM file in QIIME 1.9.0 release and I incurred in an error message which I could not understand completely.

The command I'm using to add metadata is:
biom add-metadata -i t.biom -o OUT.biom --observation-metadata-fp t.metadata --observation-header "KO,annotation" --sc-separated "annotation" --sc-pipe-separated "annotation"

The same command produced a correctly formatted BIOM file (with metadata) in QIIME 1.8.0, thus I'm guessing that something changed in the latest QIIME 1 release

t.biom and t.metadata are attached.

The error traceback is as follows:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/qiime_1_9_0/bin/biom", line 11, in <module>
    sys.exit(cli())
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/cli/metadata_adder.py", line 114, in add_metadata
    write_biom_table(result, fmt, output_fp)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/cli/util.py", line 35, in write_biom_table
    table.to_hdf5(f, biom.parse.generatedby())
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/table.py", line 4123, in to_hdf5
    self.group_metadata(axis='observation'), 'csr', compression)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/table.py", line 4095, in axis_dump
    formatter[category](grp, category, md, compression)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/table.py", line 277, in general_formatter
    vlen_list_of_str_formatter(grp, header, md, compression)
  File "/opt/miniconda3/envs/qiime_1_9_0/lib/python2.7/site-packages/biom/table.py", line 341, in vlen_list_of_str_formatter
    data[i, :len(value)] = [v.encode('utf8') for v in value]
AttributeError: 'numpy.ndarray' object has no attribute 'encode'

Thank you very much for your help,

Marco Severgnini

[Colin Brislawn]

Hello Marco,

Thanks for posting these files. I took at look at the metadata, and I think it found a clue to this issue: the metadata doesn't match up with the command you posted: 

#KO annotation

K13993 HSP20 family protein

K01505 1-aminocyclopropane-1-carboxylate deaminase [EC:3.5.99.7]

K00459 nitronate monooxygenase [EC:1.13.12.16]

K01960 pyruvate carboxylase subunit B [EC:6.4.1.1]

K01959 pyruvate carboxylase subunit A [EC:6.4.1.1]

--sc-separated "annotation" --sc-pipe-separated "annotation"

As far as I can see, these columns are separated by a tab (\t) not a semicolon, and I don't see any |pipes| in the annotation column.

I'm not sure the best way forward here, but my first thought is to reformat the metadata so that it matches the command you provided. Like this 

#KO annotation

K13993; HSP20 family protein

K01505; 1-aminocyclopropane-1-carboxylate deaminase [EC:3.5.99.7]

K00459; nitronate monooxygenase [EC:1.13.12.16]

K01960; pyruvate carboxylase subunit B [EC:6.4.1.1]

K01959; pyruvate carboxylase subunit A [EC:6.4.1.1]

I hope that helps!

Colin

[Marco Severgnini]

Dear Colin,
thank you for your answer.

However, if I understood correctly how "--sc-separated" parameter works, it refers only to the annotation column, dividing it in multiple levels (L2,L3,L4,...) separated by a semicolumn (";"). This is how, for example, QIIME recognizes objects belonging to the same taxa when summarizing OTU counts at genus, family,etc levels.

Anyway, I tried substituting the TABS with ";" in my t.metadata file and re-do the add-metadata step. This time, I incurred in no error, but the formatting of the annotation is far from the one I desired, since, after converting back to txt, I obtained:

# Constructed from biom file
#OTU ID ERR589531       ERR589534       ERR589543       ERR589554       ERR589557       ERR589785       ERR589802       ERR589805       ERR589826       ERR589864
K13993  861.0   1486.0  1356.0  2775.0  1728.0  3090.0  704.0   956.0   1535.0  2063.0  H; S; P; 2; 0;  ; f; a; m; i; l; y;  ; p; r; o; t; e; i; n
K01505  40.0    74.0    92.0    76.0    56.0    56.0    54.0    50.0    120.0   49.0    1; -; a; m; i; n; o; c; y; c; l; o; p; r; o; p; a; n; e; -; 1; -; c; a; r
K00459  797.0   1027.0  612.0   459.0   627.0   635.0   985.0   482.0   1014.0  578.0   n; i; t; r; o; n; a; t; e;  ; m; o; n; o; o; x; y; g; e; n; a; s; e;  ; [
K01960  1866.0  2202.0  6804.0  8605.0  9667.0  8697.0  3546.0  4893.0  3201.0  6101.0  p; y; r; u; v; a; t; e;  ; c; a; r; b; o; x; y; l; a; s; e;  ; s; u; b; u
K01959  238.0   267.0   829.0   253.0   186.0   172.0   343.0   135.0   394.0   152.0   p; y; r; u; v; a; t; e;  ; c; a; r; b; o; x; y; l; a; s; e;

Thank you anyway, other ideas?
Marco

[Marco Severgnini]

I have found a workaround for my files.

Using directly the "-m" option in "biom convert" did the trick. The full command is like this:

biom convert -i INPUT.txt -o OUTPUT.biom -m t.metadata --to-json --table-type "Function table" --process-obs-metadata "sc_separated"

Now, the annotations are correctly formatted and usable in QIIME starting from the .biom table.

But this does not resolve the issue with add-metadata command per se. Thus, any comment on the issue is still welcome

[Colin Brislawn]

hello Marco,

Oh, you are totally right about the tab and semicolon separation. Sorry to cause confusion!

In the documentation, this is the example command:
biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt

I wonder if omitting either the --observation-header "KO,annotation" or --sc-separated "annotation" flags would help out here... 

EDIT: I'm glad you found this workaround! While Qiime 1 is not supported, the biom-format is, so we should try to resolve this issue!

Thanks for all your help, Marco,

Colin