parsing beast summary trees with traits

[Simon Harris]

Hi,

BEAST allows users to reconstruct ancestral traits across a tree. These traits can be anything from geographical locations to bacterial host, but lead to annotations in the MCC tree that cause dendropy to fail to read the summary tree.

These new features in BEAST can add a lot of annotation to the summary tree. For example, one node may have the following extra annotations:
State.set.prob={0.6947407407407408,0.3052592592592593}
State.count=2.2813987504973414
State.count_range={0.09201042400407272,12.475354628956747}
State.prob=0.6947407407407408
State.rate=0.16574556862715406
State.count_median=2.0117797165791265
State.count_95%_HPD={0.37755235967981626,4.716265451551478}
State="Human"
State.set={"Human","Animal"}
State.rate_95%_HPD={0.09729594260327873,0.2382717343012908}
State.rate_median=0.16060014510428278
State.rate_range={0.07496096410509989,0.5467001120774253}

It would be really useful if dendropy was less strict on what information it can read from a beast tree (there are many new features like this being added to BEAST all the time), but more importantly, these extra annotations mean that dendropy cannot even read the tree.

The first problem is that some of the annotations in this type of output do not match the regex in dendropy.dataio.nexustokenizer.FIGTREE_COMMENT_FIELD_PATTERN, as they can contain only a single value between the parenthases. e.g. When one state has a probability of 1 on a node, only that state is shown in the State.set and State.set.prob, meaning that, for example you could have State.set.prob={1.0}.

This leads to:
File "build/bdist.linux-x86_64/egg/dendropy/dataio/beast.py", line 125, in parse_beast_tree_node_info
ValueError: invalid literal for float(): 1.0}

To try to get around this I have tried editing dendropy.dataio.nexustokenizer.FIGTREE_COMMENT_FIELD_PATTERN to re.compile(r'(.+?)=({.+?}|{.+?,.+?}|.+?)(,|$)'). This works, but I have now hit another problem, which is that the State and State.set annotates nodes with values that are not floats. e.g. State="Human" and State.set={"Human","Animal"}, so I now get the following error:
 File "build/bdist.linux-x86_64/egg/dendropy/dataio/beast.py", line 128, in parse_beast_tree_node_info
ValueError: could not convert string to float: "Animal"

Is there a way to work around this, or perhaps allow dendropy to read this kind of data?

Thanks,

Simon

[Jeet Sukumaran]

The perils of using an non-standardized ever-changing idiosyncratic
format that re-purposes a vaguely defined feature of a vaguely-defined
variant of format for which dozens of idiosyncratic, conflicting, and
non-documented variants exist to try and express rich information
rigorously.

You can submit a feature request on GitHub, and I will look into it.

My plan is to deprecate the beast summary tree at some point, mainly
because (a) DendroPy's metadata infrastructure handles this and other
cases in a more general, abstract way [albeit, with some of the work
delegated to client code] and (b) I do not want to keep chasing ever-
changing formats and format variants all over hill and dale.

In the mean time, (and, in fact, the recommended way to handle this
now and the future), you can make use of the fact that: (a) DendroPy
parses and makes available all comments associated with trees and
nodes and (b) DendroPy can optionally extract `[&...]` comments as
metadata (see below).

Simply read in the tree as a normal dendropy.Tree object, and then
examine the comments and metadata directly.

If you ask DendroPy to parse out the metadata, by, for e.g.:

TreeList.get_from_path("foo.tre",
...
extract_comment_metadata=True)

the metadata will be available in DendroPy's metadata infrastructure,
as described here:

http://pythonhosted.org/DendroPy/tutorial/working_with_metadata_annotations.html

This type of parsing is less strict than the BeastSummaryTree one, so
may get all you need.

Alternatively, you can as DendroPy *not* to parse the comments into
metadata ('extract_comment_metadata=False'), and access the comments
directly by the ``comments`` attribute of node (and tree) objects.
This will simply be a list representing the comment strings associated
with each object.

In either case, you will have to do some work to parse the data,
coerce to proper type, and promote to attributes (if needed).

[Simon Harris]

Hi Jeet,

Thanks very much for your speedy reply. That sounds perfect (much better than using the beast-summary-tree). I'm very happy to handle the annotation parsing myself.

Thanks again,

Simon

[Simon Harris]

Hi Jeet,

I tried asking dendropy to parse the annotation data using 'extract_comment_metadata=True' as you suggested, but unfortunately it didn't parse the new annotations properly. Here's what I get for one node in the tree when I print the name and value for the stored annotations:


allScot_trait_reconsAug20.rate_range ['1.6122098866561642E-4', '0.010083112321905948']
height_95%_HPD ['19.999999999994998', '20.000000000004']
State.set.prob ['1.0}', 'length_range={1.519834658539', '70.65314751068']
State.rate 0.16574556862715406
length_95%_HPD ['2.652916625839', '30.80144338872']
State.count_median 2.511426137550946
height_range ['19.99999999998999', '20.000000000010004']
State.count_95%_HPD ['0.15986248266169087', '7.64960401586123']
allScot_trait_reconsAug20.rate_95%_HPD ['3.5032460164412195E-4', '0.004349335805937519']
State.count 3.1406094128116524
allScot_trait_reconsAug20.rate_median 0.0018541858912397525
State.count_range ['0.0674509546375468', '25.349133790506468']
height_median 20.0
height 19.999999999999993
State "Animal"
State.rate_95%_HPD ['0.09729594260327873', '0.2382717343012908']
State.set ['"Animal"}', 'allScot_trait_reconsAug20.rate=0.002079527016480585', 'State.rate_median=0.16060014510428278', 'length=13.77309548815188', 'State.prob=1.0', 'length_median=11.533482940655', 'State.rate_range={0.07496096410509989', '0.5467001120774253']

Notice, in particular, State.set.prob ['1.0}', 'length_range={1.519834658539', '70.65314751068'], where it has included a second annotation inside the first. I assume this is because the State.set.prob annotation only contains one value. Is there an easy way to fix this, or do I need to reparse each annotation manually?

Thanks,

Simon

On Monday, April 22, 2013 4:57:41 PM UTC+1, Jeet Sukumaran wrote:

[Jeet Sukumaran]

Hi Simon,

The relevant code is in dendropy.dataio.nexustokenizer.py, starting at
line 68. Lines 68 and 70 list the reg-ex pattern to extract the data,
while the main ``parse_comment_metadata()`` function iterates over the
reg-ex matches and does the job of inserting the appropriate key/value
pairs into the annotations database. If you want, you might look into
fiddling with the reg-ex if some simple tweaks will get everything
working again.

> >http://pythonhosted.org/DendroPy/tutorial/working_with_metadata_annot...