TFP Feature Selection

[Sarthak Kala]

Hi, I am new to TFP, trying to build a Bayesian regression model with it. Is there a method for feature selection in TFP for Bayesian Linear Regression since I have a lot of features.

[Krzysztof Rusek]

Hi, Try using Laplace distribution as  prior, this will give you lasso.

Also in tfp.experimental.nn you can find implementation of spike and lab prior 

Wiadomość napisana przez Sarthak Kala <sarthak...@gmail.com> w dniu 04.04.2022, o godz. 21:25:

Hi, I am new to TFP, trying to build a Bayesian regression model with it. Is there a method for feature selection in TFP for Bayesian Linear Regression since I have a lot of features.

--
You received this message because you are subscribed to the Google Groups "TensorFlow Probability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfprobabilit...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfprobability/4f371d58-3c5a-4774-9758-f1e5f0c97b16n%40tensorflow.org.

[Sarthak Kala]

Hi, Can anyone share a working code patch for tensorflow probability spike and slab for feature selection ?

[Mike Lawrence]

Pretty sure the consensus in the Bayes/stats world in recent years (decade or so?) is that spike and slab is not a very good way to achieve feature selection. Take a look at the Finnish horseshoe as a more modern alternative. 

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfprobability/666a9b5f-8dc3-4dad-91c8-26fddbd65346n%40tensorflow.org.

--

--
Mike Lawrence, PhD

Co-founder & Research Scientist

Axem Neurotechnology

axemneuro.com

~ Certainty is (usually) folly ~

[Mike Lawrence]

See the horseshoe here: 

https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Horseshoe

And the Finnish Horseshoe (Vehtari & colleagues are in Finland) here: 

https://www.tensorflow.org/probability/api_docs/python/tfp/sts/SparseLinearRegression

[Brian Patton 🚀]

There's some support for spike and slab Gibbs sampling for STS in here, if it's interesting. 

https://github.com/tensorflow/probability/blob/v0.16.0/tensorflow_probability/python/experimental/sts_gibbs/gibbs_sampler.py

Might be hard to divine usage since we haven't formally exposed an API or explicitly supported its use with STS yet.

Brian Patton | Software Engineer | b...@google.com

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfprobability/CAB%2BQPJBS%2BhF1NQKP_FKeXSK7xg4nZOzFNanu3iq77qSc2kzFBw%40mail.gmail.com.

[Colin Carroll]

Echoing and clarifying some things: the MAP estimate with a Laplace prior on weights will recover Lasso regression, but if you sample or compute an expectation, your features will not actually be sparse. 

The spike and slab Gibbs sampler is quite experimental, so if you use it keep an eye on release notes (or pin a nightly), but would love to hear anyone's experience using it (successes, failures, frustrations, wishes)!

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfprobability/CAC%3D%2Bdz7u71Vi_%2BhPLOpSowJApDyuVKzHWk5jbsLK728Y8FnBEA%40mail.gmail.com.