Network scoring questions
Jeff Klann
pseudo-counts)? I suppose the easiest way to set the pseudocount to 1
is to modify MultinomialCPD_Py like I did for pebl2dne? I assume this
would improve structure learning?
Also, how does one penalize for network complexity with this K2/BDe
score? Is that somehow already in the score, or is this best done with
a weighted uniform prior? If the latter, what is a good way to
determine the optimal weight?
Third, is it reasonable to compare scores (given the same prior)? I've
read that K2 is not score-equivalent, but I'm not sure what that says
about comparing scores of different networks. Perhaps if
score(A)>score(B), then A is at least as good as B?
Finally, I noticed I was getting a cyclic graph from
consensus_network(). Is that expected? I think I don't understand
consensus networks well.
Thanks for all your help,
Jeff Klann
p.s. I am still unable to find a fast conditional independence test in
Python. I've updated mine and will post it soon, but it is still too
slow. Any ideas where I should look or whom I should ask?
Charles R. Twardy
> score? Is that somehow already in the score, or is this best done with
I haven't looked at the pebl code here. Generally, BDe does penalize
for complexity, but without using an explicit structure prior.
> Third, is it reasonable to compare scores (given the same prior)?
I'm not sure what you're asking. Clearly if you couldn't compare
scores, it wouldn't work as a metric. If I recall correctly, most
metrics approximate a transform of P(Model & Data), the numerator of the
posterior probability. Better scores mean higher estimated posterior
probability. The different metrics make different assumptions. But I
wouldn't go so far as to believe the probabilities themselves, if that's
what you are asking.
And apologies, I haven't found time to look into fast conditional
independence tests. Or anything pebl-related. :-(
-crt
--
Charles R. Twardy
Research Assistant Professor
George Mason University C4I Center
703 993 1846 voice
571 212 0674 mobile
ctw...@gmu.edu email
Abhik Shah
finally cleaning.. :)
On Wed, Dec 1, 2010 at 11:00 PM, Jeff Klann <jkl...@gmail.com> wrote:
> Is pebl actually using the K2 score (the Bd score with no
> pseudo-counts)? I suppose the easiest way to set the pseudocount to 1
> is to modify MultinomialCPD_Py like I did for pebl2dne? I assume this
> would improve structure learning?
Yes, pebl uses the BD (aka K2) score. The metric can be derived from
Heckerman's BDe by setting all N'_{ijk} to 1 (pseudocount). So, I
don't think there's a need to set any pseudocounts in the CPT because
it's built in to the scoring metric.
>
> Also, how does one penalize for network complexity with this K2/BDe
> score? Is that somehow already in the score, or is this best done with
> a weighted uniform prior? If the latter, what is a good way to
> determine the optimal weight?
>
I believe BD does implicitly penalize for network complexity although
not as explicitly or heavily as BIC.
> Third, is it reasonable to compare scores (given the same prior)? I've
> read that K2 is not score-equivalent, but I'm not sure what that says
> about comparing scores of different networks. Perhaps if
> score(A)>score(B), then A is at least as good as B?
>
BD (as opposed to BDe) is not likelihood equivalent. This means that
two networks that induce the same set of conditional independencies
could have different scores.
For example, A-->B and A<--B both state that there are no conditional
independencies. If the data are purely observational, they shouldn't
discriminate between these two models.
With BDe, both models would have the same score... but BDe requires a
user-supplied prior structure that is difficult in practice.
With BD, the two models could have different scores. All N'_{ijk} are
set to 1 instead of being calculated from a prior structure.
I've found that if all nodes have the same self-entropy, the BD score
is likelihood-equivalent. The discretization scheme (equal frequency
binning) used by pebl ensures this. So, pebl uses BD but is*
likelihood-equivalent. *Note that this is an empirical result but I
haven't proven it analytically.
So, with pebl's discretization, you can compare scores between
networks. With a different discretization, you can still compare
scores but two very similar networks could have slightly different
scores.
> Finally, I noticed I was getting a cyclic graph from
> consensus_network(). Is that expected? I think I don't understand
> consensus networks well.
>
A consensus network is the set of all edges with posterior probability
higher than threshold. The posterior prob for an edge (or any network
feature) is calculated by marginalizing over the network posterior:
P(edge|data) =~ \Sum P(model|data)P(edge|model). Note that
P(edge|model) is just an indicator (0 or 1).
So, the consensus network is a weighted average over the network
posterior. It's not a Bayesian network but an average over them (hence
cycles). This is based on N. Friedman's "Being Bayesian about Network
Structure" paper.
Thanks,
Abhik.