Recommendations for Inference?

[Gordon]

I've seen some cross posting of pebl on other bayes net projects but what is the recommended way to use networks learned with pebl to perform inference? For example, continuous valued variables are discretized for structure learning but are the parameters maintained so that inference can be done even if approximately?

--
Gordon Rios -- http://www.linkedin.com/in/gordonrios
Cork Constraint Computation Centre -- http://4c.ucc.ie

[Abhik]

Hi Gordon,
There really isn't currently any recommended way to use pebl
networks elsewhere.

We don't explicitly learn the parameters but rather use the BDe metric
to calculate the marginal likelihood. We could learn point estimates
or posteriors for parameters, but for structure learning, there's no
use..

To do inference following structure learning with pebl, you would just
need to write an adapter function to convert a pebl network
(network.py:Network) to the class/datastructure required by another
library.

Thanks,
Abhik.

[Charles R. Twardy]

Gordon,

So, one possibility is to give the structure to Netica, and then let
Netica learn the parameters via counting, EM, or gradient descent. I'd
be interested in helping. I like that PEBL handles missing data, and
would like to use the learned structure for inference.

NB: Some structure learning (MML) *does* make use of parameter estimates.

Charles

On 1/7/2010 9:29 PM, Abhik wrote:
> Hi Gordon,
> There really isn't currently any recommended way to use pebl
> networks elsewhere.
>
> We don't explicitly learn the parameters but rather use the BDe metric
> to calculate the marginal likelihood. We could learn point estimates
> or posteriors for parameters, but for structure learning, there's no
> use..
>
> To do inference following structure learning with pebl, you would just
> need to write an adapter function to convert a pebl network
> (network.py:Network) to the class/datastructure required by another
> library.
>
> Thanks,
> Abhik.

--
Charles R. Twardy
Research Assistant Professor
George Mason University C4I Center
703 993 1846 voice
571 212 0674 mobile (10c/min)
ctw...@gmu.edu email
ctwar...@jawp.js.smil.mil SIPRNet

[Gordon]

Abhik, that makes perfect sense now. Is there an open source BN package that you think is a particularly good fit for an adapter like this? Ideally, if I write an adapter I'd like to contribute it back and I'll likely work in python.

Thanks,
Gordon

[Gordon]

Hi Charles, sounds good, my earlier reply got missed because I sent it from another gmail address. Hopefully, I've rectified it now. I'll take a look at Netica ...

Thanks,
Gordon

[Abhik]

I don't have any experience with inference using a python library.. A
year or so back, I tested a couple out (openbayes.org and pybnt, I
believe) but had trouble getting them to work.

Kevin Murphy's BNT (in matlab) seems to be the best one that I know of..

Thanks,
Abhik.

[Charles R. Twardy]

Hi,

I have used Netica a lot. Not GPL, but it has a well-documented API and
bindings in many languages. Once upon a time I had wrapped some for
Python, but it might be easier to use Jython and just call the Java API.
Free download (with limits) and reasonable prices esp. education.

Also, my university participates in UnBBayes, which is all Java. I'm
looking into that -- it is GPL so they could be distributed together. I
don't know of python options.

-Charles

> I don't have any experience with inference using a python library.. A
> year or so back, I tested a couple out (openbayes.org and pybnt, I
> believe) but had trouble getting them to work.
>
> Kevin Murphy's BNT (in matlab) seems to be the best one that I know of..

--

[gparker]

This is a good recommendation ... though I have a pretty strong
preference for open source. I'll look at Netica further ...

I know about BNT but it requires Matlab ... what about some of the C /
C++ projects?

> ctwa...@gmu.edu                email

[Charles R. Twardy]

> I know about BNT but it requires Matlab ... what about some of the C /
> C++ projects?

The one that comes to mind in Intel's PNL. It's the C++ implementation
of (much of) BNT. Last I knew, it wasn't being maintained.

I'll talk to Rommel about UnBBayes on Tuesday.

--
Charles R. Twardy
Research Assistant Professor
George Mason University C4I Center
703 993 1846 voice
571 212 0674 mobile (10c/min)

ctw...@gmu.edu email
ctwar...@jawp.js.smil.mil SIPRNet

[gparker]

Hi Abhik,

Poked around in the code a bit and I don't see direct references to
the CPD's in the network class. Also, what's the current state of
interchange formats like BIF, BNIF, etc? The material I can find on
the web seems a bit dated. Any ideas? My preference would be to
convert to an open standard.

> > Gordon Rios --http://www.linkedin.com/in/gordonrios

[Abhik]

Hi Gordon,
The network simply specifies the set of nodes and edges. The
evaluator class (any of the classes in evaluator.py) maintain a list
of CPDs, one for each node. This decoupling makes it easier to write
code to evaluate a network against multiple datasets..

I also had a difficult time finding a format that was under active
development and widely used.

Thanks,
Abhik.

[Gordon]

Hi Abhik,

Okay, then I'll need a bit more information to adapt the pebl network / evaluator to a bayes network that can be used for inference with other packages. The LearnerResult instance returned from run() contains the posterior and the consensus_network() but I also need to get the evaluator which I don't see in the LearnerResult.

For mine or future purposes can you add a bit of documentation for how to extract the necessary pieces from the system to enable an adapter? It would be a useful addition to the doc and someone might just pick up the task of the adapter and contribute it back :)

Thanks,
Gordon

--
Gordon Rios -- http://www.linkedin.com/in/gordonrios

[Charles R. Twardy]

Abhik,

Sometimes I am slow to realize things. It only just occurred to me
today that you can't be doing standard EM, because PEBL doesn't have
inference.

Looking at your code and Heckerman, you Gibbs sample over all possible
completions for all the missing data, and score the result. If I
understand right, you do this by:
for each row i (a case with missing observations):
for each col j (a variable whose value is unknown for case i):
1. Score every possible completion of variable X_j.
2. Convert the scores to a prob. distribution P_j.
3. Sample a value for X_j from P_j.
Repeat that loop until max_iters.

Therefore, on average, the network is scored based the most favorable
way to fill in the missing data.

To do inference, either I need to extract the resulting sampled CPD, or
pass the structure to a learner with EM or similar. (Netica has this).
Are the sampled CPDs available?

If I understand Heckerman, this is appropriate for data missing at
random, but not if missingness depends on the value of a variable.

Do I understand correctly?

Charles

[Abhik]

On Tue, Jan 19, 2010 at 2:17 PM, Charles R. Twardy <ctw...@gmu.edu> wrote:
> Abhik,
>
> Sometimes I am slow to realize things.  It only just occurred to me today
> that you can't be doing standard EM, because PEBL doesn't have inference.
>
> Looking at your code and Heckerman, you Gibbs sample over all possible
> completions for all the missing data, and score the result.  If I understand
> right, you do this by:
>   for each row i (a case with missing observations):
>      for each col j (a variable whose value is unknown for case i):
>         1. Score every possible completion of variable X_j.
>         2. Convert the scores to a prob. distribution P_j.
>         3. Sample a value for X_j from P_j.
> Repeat that loop until max_iters.
>
> Therefore, on average, the network is scored based the most favorable way to
> fill in the missing data.
>

This is correct.

> To do inference, either I need to extract the resulting sampled CPD, or pass
> the structure to a learner with EM or similar. (Netica has this). Are the
> sampled CPDs available?
>

No, they're not. For efficiency, we use the same CPD object and just
update it every iteration. You could modify the Gibbs sampler to
write each sampled CPD to file but you'd take a big I/O hit in doing
that.

> If I understand Heckerman, this is appropriate for data missing at random,
> but not if missingness depends on the value of a variable.
>

Correct..

Thanks,
Abhik.

[gparker]

Hi Charles,

I think I'm reaching a similar finding on this ... my goal is to put
together a Mac/Linux based tool chain for prototyping a modeling and
scoring process. Given a dataset (two cases, one fully observed and
one partially).

1. Learn structure from dataset (PEBL or BANJO)
2. Learn CPD's from dataset Tabular CPT or model (learn parameters)
BNT is a bit clunky since it can't load or save a network, JAGS/BUGS
use a modeling language.
3. Score an entire new dataset on a daily basis by inference over the
unobserved variables with the Bayes Net learned in steps 1 and 2.

With many other modeling approaches this could be done in R with
python, PERL, or Ruby for moving data around, extracting from a
database, etc. The packages, even commercial ones, seem to be oriented
around building a bayes network (using a gui) and then performing
inference over a single new case. The closest was BNet but the engine
component (explicitly allows reading from three different formats of
Bayes Net) but is a cool $5,000 :)

Netica is probably the best bet but the API looks very much oriented
to exploratory work rather than production oriented batch processing.
Any thoughts or refinements? Am I missing the boat on Netica?

Thanks,
Gordon

On Jan 20, 4:11 pm, Abhik <abhiks...@gmail.com> wrote:

> > ctwa...@gmu.edu                email

[Gordon]

A refinement on Netica is that it looks like it has API support for batches of cases (i.e. "casesets", which they say are used only for learning) but for production there needs to be a way to write code that does inference over a sequence of cases and saves off estimates for the unobserved variables of interest.

Also, not sure how much support Netica has for learning non-Tabular (e.g. gaussian, etc.) parameters from Casesets.

Other than that it looks like Netica could be the best overall complement for PEBL or BANJO given requirements for non-Windows platforms, open source or at least economically sensitive pricing, industry grade software quality(?), good documentation, graphical capability, etc.

--
Gordon Rios -- http://www.linkedin.com/in/gordonrios

[Charles R. Twardy]

Hi Gordon,

Right, Netica does batch learning, and you can either use their default
algorithm or EM (or gradient descent). You can do incremental learning
as well because each table has an "experience number" that tracks the
number of observations. I haven't tried using that with their EM
algorithm though, so it would take some testing.

Right now, Netica's built-in learning is tabular only. They have a very
expressive language for writing arbitrary functions -- decision trees,
gaussians, etc. -- but you would have to manage that learning yourself.
Even inference is done by "equation to table" and discretizing, though
they say they are looking into taking advantage of decision trees during
inference.

I should point out that CaMML from Monash university has a Weka wrapper
that is already integrated with Netica for inference. Having been at
Monash, I'm biased, but I'd say the learner is very good -- it learns
decision trees or logit models and has a very good metric and search --
but it doesn't yet handle missing data very well. Also, while decision
trees and logit help it learn more connections, it currently flattens
them to a table before sending out to Netica for inference, which
sometimes makes for ridiculously big files.

-Charles

On 1/23/2010 12:57 PM, Gordon wrote:
> A refinement on Netica is that it looks like it has API support for
> batches of cases (i.e. "casesets", which they say are used only for
> learning) but for production there needs to be a way to write code that
> does inference over a sequence of cases and saves off estimates for the
> unobserved variables of interest.
>
> Also, not sure how much support Netica has for learning non-Tabular
> (e.g. gaussian, etc.) parameters from Casesets.
>

--

Charles R. Twardy
Research Assistant Professor
George Mason University C4I Center
703 993 1846 voice
571 212 0674 mobile (10c/min)

ctw...@gmu.edu email

[Gordon]

Hi Charles,

This helpful -- but I had a stumbling block with Netica as it happens that their Java version won't run on Leopard (10.5.8) only on Snow Leopard and Tiger :(

They use JNI rather than pure java and since it's closed source one can't simply rebuild for the right platform -- too bad that so many people throw away the benefits of Java but I guess it's easier than writing the core in Java.

Part of my reason for having a strong open source preference is simply practical (as in this case) since my only option now is to upgrade my system to Snow Leopard or run another OS like Linux or Windows using Bootcamp or VMWare.

I'm looking at Weka now ... I've used it for classification tasks in the past using Naive Bayes. Never thought of it as a BN package but it *seems* to have some of the necessary elements now and reads BIF. Any thoughts on that?

I'd like to pass it the network structure from PEBL or BANJO, provide data for the variables, and say "learn!" :)

[Charles R. Twardy]

Hi Gordon,

First, Netica. I find them to be very responsive to requests for
particular platforms. I once gave them an account on my iBook running
linux, and overnight they logged in and compiled a version for me.
Granted, that depends on whether someone there has some free time, but I
was impressed. Not open source, but very responsive.

Weka's Naive Bayes is different from the BN package that Remco Boucert
(?sp) wrote. The BN package does full structure learning with either
metrics or CI tests. It doesn't do EM yet. However, it should be
possible to use it for inference and hopefully for parameter learning. I
asked about this about a week ago and Remco said to look at the BN GUI
classes for that. I haven't done so yet.

Let me know what you plan to use -- it would be good to coordinate.

-Charles

> ctw...@gmu.edu <mailto:ctw...@gmu.edu> email

[Charles R. Twardy]

Another thought.

If we're going to use Weka for inference, it might actually be easier to
port the Gibbs sampling algorithm to Weka, or to implement EM? I'd hate
to leave behind the beauty of Python, but it should be considered.

Charles