How to do exact inference when features are attached a two-variable-factor (using a factor3)?

[Keenon]

First of all, excellent work with Factorie!

I'm trying to attach features (invariant under inference) to a factor that connects two binary variables that can change during inference. I'm getting incorrect results when I run exact inference. I must be doing something wrong. What's the correct way to attach features to a multi-node variable and get exact inference?

I've attached a simple example, that implements the features with factor3 created from a 

DotFamilyWithStatistics3[BooleanVariable,BooleanVariable,FeatureVectorVariable[String]]

Gibbs output works fine, but exact inference yields strange results that are invariant to changes in the feature values.

Here's the code (also attached as a .scala file), and at the end I list the results when I run it.

import cc.factorie.Parameters
import cc.factorie.model.{DotFamilyWithStatistics3, Model}
import cc.factorie.variable.{FeatureVectorVariable, CategoricalVectorDomain}
import cc.factorie.infer._
import cc.factorie.la           // Linear algebra: tensors, dot-products, etc.
import cc.factorie.variable._
import cc.factorie.model._

/**
 * Created by keenon on 4/25/15.
 *
 * Testing putting log-linear feature weights on a binary feature in FACTORIE
 */
object LogLinearFeaturesForMultiFactors extends App {

  // Create the variables we'll do inference over

  val firstVar = new BooleanVariable()
  val secondVar = new BooleanVariable()

  // Create a feature variable (constant during inference), with string features

  val featureDomain: CategoricalDomain[String] = new CategoricalDomain[String]
  val featuresVar = new FeatureVectorVariable[String]() {
    override def domain: CategoricalVectorDomain[String] = new CategoricalVectorDomain[String] {
      override def dimensionDomain = featureDomain
    }
  }

  // put a single feature in the variable
  // "feat1" -> 1.0

  featuresVar.update(featureDomain.index("feat1"), 1.0)(null)

  // Create a model to associate the 3 variables:
  // Ascii diagram:
  //
  //          featureVariable <-- (constant)
  //                 |
  //     firstVar ---+--- secondVar

  val model = new Model with Parameters {
    val errorModel = new DotFamilyWithStatistics3[BooleanVariable,BooleanVariable,FeatureVectorVariable[String]] {
      val weights = Weights(new la.DenseTensor3(2, 2, 1))

      // HERE'S MY QUESTION: It seems this corresponds to weights for assignments
      // of firstVar and secondVar, to be multiplied by the weight of featuresVar["feat1"] = 1.0
      //
      // Is that the case?
      //
      // that would imply an interpretation like the following:
      //
      // (firstVar = true, secondVar = true) = featuresVar["feat1"]*3.0
      // (firstVar = true, secondVar = false) = featuresVar["feat1"]*2.0
      // (firstVar = false, secondVar = true) = featuresVar["feat1"]*1.0
      // (firstVar = false, secondVar = false) = featuresVar["feat1"]*1.0

      weights.value := Array(3.0, 2.0, 1.0, 1.0)
    }

    override def factors(variables: Iterable[Var]): Iterable[Factor] = {
      List(errorModel.Factor(firstVar, secondVar, featuresVar))
    }
  }

  //
  // This is the exact inference, which seems to not work
  //

  // Do the inference over firstVar and secondVar using BP on a Tree
  val sumExactBeliefs : Summary = BP.inferTreeSum(List(firstVar, secondVar), model)
  // Get the marginals
  val m1 : DiscreteMarginal1[BooleanVariable] = sumExactBeliefs.getMarginal(firstVar).get.asInstanceOf[DiscreteMarginal1[BooleanVariable]]
  println("firstVar marginal: "+m1.proportions)
  val m2 : DiscreteMarginal1[BooleanVariable] = sumExactBeliefs.getMarginal(secondVar).get.asInstanceOf[DiscreteMarginal1[BooleanVariable]]
  println("secondVar marginal: "+m2.proportions)

  //
  // This is the gibbs inference, which seems to work fine, but is a bit slow compared to correctly working exact inference
  //

  // Do the inference over firstVar and secondVar using Gibbs
  val r = new scala.util.Random(42)
  // randomly initialize to valid values
  firstVar.set(false)(null)
  secondVar.set(true)(null)
  // run gibbs sampler
  val gibbsBeliefs : Summary = new InferByGibbsSampling(samplesToCollect = 10000, 10, r).infer(List(firstVar, secondVar), model)
  // Get the marginals
  val g1 : DiscreteMarginal1[BooleanVariable] = gibbsBeliefs.getMarginal(firstVar).get.asInstanceOf[DiscreteMarginal1[BooleanVariable]]
  println("firstVar gibbs: "+g1.proportions)
  val g2 : DiscreteMarginal1[BooleanVariable] = gibbsBeliefs.getMarginal(secondVar).get.asInstanceOf[DiscreteMarginal1[BooleanVariable]]
  println("secondVar gibbs: "+g2.proportions)
}

Here's the results:

firstVar marginal: Proportions(0.33333333333333337,0.6666666666666667)

secondVar marginal: Proportions(0.33333333333333337,0.6666666666666667)

Warning: SparseIndexedTensor slow += with type cc.factorie.la.Singleton2BinaryLayeredTensor3

firstVar gibbs: Proportions(0.8357,0.1643)

secondVar gibbs: Proportions(0.6937,0.3063)

Thanks in advance!

- K

[Keenon]

I've figured out what is going wrong. For exact inference, there are undocumented sparsity control tensors for situations like this (limitedDiscreteValues1, limitedDiscreteValues12, limitedDiscreteValues123), and they defaults to full sparsity (no non-zero entries), so BP skips every entry in the factor. You need to override them and fill them with ones in order to get correct behavior.

...

[Keenon]

For posterity, in case anyone stumbles across this thread having the same problem, here's how that works in code:

val errorModel = new DotFamilyWithStatistics3[BooleanVariable,BooleanVariable,FeatureVectorVariable[String]] {
  val weights = Weights(new la.DenseTensor3(2, 2, 1))

  weights.value := Array(3.0, 2.0, 1.0, 1.0)

  // This is the key:
  // you need to initialize the tensors to the correct size for the marginal
  // you're dealing with, and then fill it with 1.0s

  // Initialize the case this thread is about
  limitedDiscreteValues12 = new SparseBinaryTensor2(firstVar.domain.dimensionSize, secondVar.domain.dimensionSize)
  // Set all entries to "true"
  for (i <- 0 to limitedDiscreteValues12.size-1) limitedDiscreteValues12.+=(i, 1.0)
}

On Saturday, April 25, 2015 at 4:13:05 PM UTC-7, Keenon wrote:

...

[Emma Strubell]

Great, thanks for sharing your solution!

Emma

--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu
To unsubscribe, email: discuss+u...@factorie.cs.umass.edu

[Keenon]

I create a Pull request on GitHub with a change for the default behavior here. Hopefully this will keep future users from having the same trouble:

https://github.com/factorie/factorie/pull/281

Keenon

[Keenon]

Ok, a fix has been merged into the main Factorie. This shouldn't be a problem in the future. https://github.com/factorie/factorie/pull/282

Keenon