Weights and biases in probabilistic regression example

[Malachy Campbell]

Hi All, 

I'm trying to understand the weights and biases output in the probabilistic regression example, specifically case 3 (epistemic uncertainty). Two arrays are returned by model.weights: the first has four elements and the second has two. I'm pretty sure the second is the weight and bias for the location parameter. Is the first array the weights and biases for the scale and location of the posterior?

Here is the code:

# Specify the surrogate posterior over `keras.layers.Dense` `kernel` and `bias`.

def posterior_mean_field(kernel_size, bias_size=0, dtype=None):

  n = kernel_size + bias_size

  c = np.log(np.expm1(1.))

  return tf.keras.Sequential([

      tfp.layers.VariableLayer(2 * n, dtype=dtype),

      tfp.layers.DistributionLambda(lambda t: tfd.Independent(

          tfd.Normal(loc=t[..., :n],

                     scale=1e-5 + tf.nn.softplus(c + t[..., n:])),

          reinterpreted_batch_ndims=1)),

  ])

  

# Specify the prior over `keras.layers.Dense` `kernel` and `bias`.

def prior_trainable(kernel_size, bias_size=0, dtype=None):

  n = kernel_size + bias_size

  return tf.keras.Sequential([

      tfp.layers.VariableLayer(n, dtype=dtype),

      tfp.layers.DistributionLambda(lambda t: tfd.Independent(

          tfd.Normal(loc=t, scale=1),

          reinterpreted_batch_ndims=1)),

  ])

# Build model.

model = tf.keras.Sequential([

  tfp.layers.DenseVariational(1, posterior_mean_field, prior_trainable, kl_weight=1/x.shape[0]),

  tfp.layers.DistributionLambda(lambda t: tfd.Normal(loc=t, scale=1)),

])

# Do inference.

model.compile(optimizer=tf.optimizers.Adam(learning_rate=0.01), loss=negloglik)

model.fit(x, y, epochs=1000, verbose=False);

# Profit.

[print(np.squeeze(w.numpy())) for w in model.weights];

yhat = model(x_tst)

assert isinstance(yhat, tfd.Distribution)

Thanks,

Malachy

[Pietro Portolani]

Hi,

I'm trying to figure this out as well: I think the first list contains locations and scales of the posteriors of weights and biases, instead the second list holds locations of the priors. I still don't understand (and can't find anywhere) what is the actual sequence of locations and scales and their relation to the output neurons, in order to visualise them.

Anyone knows it or has any idea?

Thanks,

Pietro