calibration and "precision"

[Bob Carpenter]

I'm trying to follow Michael's advice and discuss why we care
about posterior predictions for new data.

I want to say that

1. models need to be calibrated in the posterior interval coverage
sense, and

2. given two calibrated models, the one with narrower intervals
makes better predictions.

Is there a name for (2) other than "precision" (which is overused)?
I didn't see a direct discussion of (2) in BDA, either.

I think (2) is closely related to

3. given two models, the one with higher posterior density for
new data is better.

Usually because to get higher density for the new data, you also
get narrower intervals.

- Bob

[Andrew Gelman]

I’m cc-ing Aki who’s thought a lot about these issues. With regard to Bob’s points 1, 2, and 3 below, I’m not sure if you’re talking about fake-data checking (i.e., checking that Stan is fitting the model it says it’s fitting) or posterior predictive checking.
A

[Bob Carpenter]

I'm talking about posterior predictive checking.
But really just talking about models in general.
Michael said I should explain more about why we're
doing posterior p-values and why good log density
on held out data is a good thing.

- Bob

P.S. Aki's on the stan-dev list, so no need to CC.

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

[Dustin Tran]

Oh cool, I’ve been thinking a lot about ppc’s recently as well. I think this is a finicky and somewhat unclear issue. To me, why we care about posterior predictions for new data is both for model checking (via ppc’s) and model comparison (via information criteria).

Model checking is ideal because it helps us during the iterative cycle of data analysis (e.g., by box, rubin, and andrew’s BDA introduction). Model comparison, selection, and averaging seems stranger, because technically you can do these with ppc’s, but it seems like with recent research Aki and Andrew focus on these tasks based on comparing log densities of held out data. It was never clear to me why we never use ppc’s for this; however, for certain things such as choosing the # of mixture components, it seems like they're okay with using it? (I’ve been meaning to ask you all about this for a while.)

Dustin

[Bob Carpenter]

For me, the reason to care about posterior predictions is that
you often want to make decisions based on a model's estimates.
The better it is predictively, the better those decisions should be.

- Bob

[Michael Betancourt]

Right. Ultimately we want to make decisions that will perform well under new measurements, and the best way to ensure this is to have the inferred predictive distribution as close as possible to the true distribution over data. Of course we don't know the true distribution so we can only quantify our predictive performance so well, in particular this is why we are limited to relative performance measures. PPCs, cross validation, information criteria -- they're all just ways of approximating this relative performance.

Gory details here: http://arxiv.org/abs/1506.02273. Sorry for all the theory -- I'm planning on doing a lower level description in my copious free time.

[Andrew Gelman]

At the end of the day there’s gonna be some model you’re going with. That model might well be a choice, averaging, or expansion of earlier models. But, whatever it is, I want to check its fit to the data.

There are different sorts of ppc’s. One is the targeted check, where there’s some particular aspect of the data that we really want to model accurately. Another is the exploratory plot, where we’d like the opportunity to discover something new. Chapter 6 of BDA has examples of both. But in general I don’t like simple histograms or density plots of data: on one hand, usually the marginal distribution of some variable is not what we care so much about, and on the other hand, simple histograms or density plots don’t show enough to be good exploratory plots. I _did_ use histograms for the speed-of-light and 8-schools examples in BDA, but now I wish I hadn’t, if this has caused the mistaken impression that this is generally a good thing to check.

I agree with Aki that the 1-d ppc graphs are not so helpful in Bob’s baseball example.

> On Jan 24, 2016, at 6:49 PM, Dustin Tran <dustinv...@gmail.com> wrote:
>

[Dustin Tran]

Targeted checks and exploratory plots (graphical checks in general) are all pretty clearly written about ppc’s in Chapter 6 in my opinion. The use case of ppc’s for model expansion (or reduction) and averaging are also clear. However, I didn’t get a good grasp of how to do model comparison: whether we should do so through ppc’s, as in the psychology example (Gelman, Meng, Stern 1996) which assess the # components of a mixture model, or by evaluating information criteria.

To be specific, my confusion is—barring any computational concerns—when do we use ppc’s with omnibus measures versus leave-one-out cross validation. When I dug through your and Aki and Jonah and Jessica’s papers on information criteria, I learned a lot about technique and relationships of various criteria. but at a higher level I never understood when it’s applicable over ppc’s (are they somehow used in concert with omnibus measures in ppc’s)?

Dustin

[Bob Carpenter]

OK, then what's the current recommendation? Just don't plot the
posterior p-values? I think it's helpful to understand what they're
actually doing.

As to the 1D PPC graphs, should I just tell people not to do PPC
of this kind in simple models or should I plot it some other way?

Sorry --- I know you didn't like what I did, but I don't understand
what you're suggesting as an alternative.

- Bob

[Krzysztof Sakrejda]

On Sunday, January 24, 2016 at 11:31:10 PM UTC-5, Dustin Tran wrote:
> Targeted checks and exploratory plots (graphical checks in general) are all pretty clearly written about ppc’s in Chapter 6 in my opinion. The use case of ppc’s for model expansion (or reduction) and averaging are also clear. However, I didn’t get a good grasp of how to do model comparison: whether we should do so through ppc’s, as in the psychology example (Gelman, Meng, Stern 1996) which assess the # components of a mixture model, or by evaluating information criteria.

I think one place where posterior predictive checks make a lot of sense is when model performance (in the practical sense) is somewhat divorced from goodness of fit as measured by the posterior density.

Lately I've been working with a model where I need to: 1) estimate the shape of the CDF for some data; and 2) use the CDF to scale a count (so true_count ~ observed_count/CDF(x) ), where the original measurements and model were on x. Turns out there's a huge right tail on x, but it's for a small proportion of x's so ultimately it doesn't do much to the scaling but it's awful to model properly. I'm not sure how my model would play out in the context of Bob's 1/2/3 rubric (it might do ok), but in real predictions against held-out data it does very nicely (moves the horizon where we can make good predictions forward another 1.5 months).

[Jonah Gabry]

On Monday, January 25, 2016 at 11:19:44 AM UTC-5, Bob Carpenter wrote:
>
> As to the 1D PPC graphs, should I just tell people not to do PPC
> of this kind in simple models or should I plot it some other way?
>

Maybe, I'm not thinking about the same 1D PPC graphs as you are, but I actually think they do have at least a bit of value. For me that value is actually as a substitute for computing the posterior predictive p-values. Rarely do I actually care what the precise value of one of the p-values is, so instead of computing them I'll just make a quick plot (e.g. T(y) as a vertical bar on a histogram of T(yrep)) and eyeball it. So I guess what I'm saying is that the 1D graphs and p-values aren't particularly useful, but if I'm going to pick one I'd prefer the graph.

[Aki Vehtari]

On Sunday, January 24, 2016 at 11:45:39 PM UTC+2, Bob Carpenter wrote:
> I'm trying to follow Michael's advice and discuss why we care
> about posterior predictions for new data.

The word "predictions" is important here.

> I want to say that
>
> 1. models need to be calibrated in the posterior interval coverage
> sense, and

I would write "posterior predictive" instead of just "posterior",
because most of the time we can check the calibration only with
respect to observations.

> 2. given two calibrated models, the one with narrower intervals
> makes better predictions.
>
> Is there a name for (2) other than "precision" (which is overused)?

Sharpness. See "Probabilistic forecasts, calibration and sharpness"
https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jrssb.pdf

> I didn't see a direct discussion of (2) in BDA, either.

We refer to this paper, but don't mention the term sharpness. Maybe in
the next edition?

> I think (2) is closely related to
>
> 3. given two models, the one with higher posterior density for
> new data is better.
>
> Usually because to get higher density for the new data, you also
> get narrower intervals.

Again it would better to have "higher posterior *predictive* density"
instead of just "higher posterior density". I think it's better to be
explicit here.

Yes, you need sharpness, but you need to be also somewhat calibrated
to get high density for new data.

In baseball example we have separate independent test data, but often
we don't have that. Chapter 6 in BDA3 assumes mostly that we don't
have independent test data.

Short story of Posterior predictive checking vs. Cross-validation
- posterior predictive checking is good for joint predictive distribution checks, but there is no calibration in test statistics
- in cross-validation checking the test statistics are calibrated (given usual assumptions), but it can be used only for marginal predictive distributions (univariate or m-dimensional where m is the block size)

Longer explanation

Posterior predictive checking (as defined in BDA) compares posterior
predictive distribution conditioned on data to the same data and this
causes the problem that the posterior predictive p-values are not
calibrated (overfitting, double use of data, etc). This is seen
especially when the chosen test statistic is related to one of the
parameters in the model. For example, in the baseball example, the
mean test statistic is bad since the model has a parameter related to
that. In BDA light speed example, variance is also bad since the model
has variance term, too. Posterior predictive checking is good for
examining properties of joint predictive distribution, such as the
minimum or maximum observed value in the baseball and light speed
example, or number state changes in hot hand examples. In weather
prediction, we might be interested in knowing the probability that the
temperature drops below zero in any part of a specific road, and thus
we need again the joint predictive distribution. Another good
example, is population dynamic models, model generated population
sizes in time are plotted and the resulting curves are compared to
observed curves and they should have similar look. It is not so easy
to make test statistic for summarizing with a single value features of
connected prey-predator population dynamics.

Predictive checking with new data avoids the double use of data, but
before we have that data we need to do something else and
cross-validation is useful option for that (most information criteria
are close to some version of leave-one-out cross-validation, but loo
is more robust http://arxiv.org/abs/1507.04544,
http://arxiv.org/abs/1412.7461). In cross-validation the double use
of data is avoided and test statistic can be calibrated. In
leave-one-out cross-validation we examine (usually) univariate
posterior predictive distributions, and thus we can't examine
properties of the joint predictive distribution. To examine
specifically whether the model predictions are calibrated, the usual
test is to look at the leave-one-out cross-validation predictive
cumulative density function values, which are uniformly distributed if
the model is calibrated (BDA3, Ch 6, Section "Marginal predictive
checks"). If the model is calibrated based on cross-validation
predictive distributed tail area probabilities, then you could look at
the sharpness of the cross-validation predictive
distributions. Looking at the cross-validation predictive densities
directly combines the calibration and sharpness and "given two models,
the one with higher cross-validation predictive density is
better". This what the 'loo' package is doing. Furthermore, in
leave-one-block-out we can examine joint posterior predictive for
blocks, so that we could look for example joint predictions for
several patients in one hospital.

One problem in reporting log predictive densities is that people tend
to have difficulties on interpreting the values and differences
between the values (leading to attaching magical properties to
information criteria and model complexity). If possible, it is good to
report the predictive performance also using utility (cost) function
which is easier to interpret by the target audience (e.g. medical
doctor). In epidemiology these could be, e.g., classification accuracy
or area under curve (AUC). In the baseball example, when estimating the accuracy of the batting rate estimates, you could also report RMSE or MAE with the independent test data used as the ground truth. You could also consider betting scenarios such as
- guess who has the highest batting rate at the end of the season
- which three players (in any order) have the highest batting rates
and calculate which of the models would have given the best chance of
winning. (or you could add also loo, to show how you can estimate the
predictive performance even before you see the end of the season).

Aki

[Andrew Gelman]

Feel free to blog this!

[Dustin Tran]

What Andrew said. I found that very insightful and I’m still trying to digest it!

Dustin

[Matt Hoffman]

+1. One of the more thoughtful and lucid things I've read recently.

[Michael Betancourt]

> I would write "posterior predictive" instead of just "posterior",
> because most of the time we can check the calibration only with
> respect to observations.

+1

> Sharpness. See "Probabilistic forecasts, calibration and sharpness"
> https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jrssb.pdf

I dislike this term — the problem is that it’s not about getting the smallest
predictive interval as possible, but rather the predictive interval that best
matches the variance of the true data generating process. It’s not about
precision but rather calibration, albeit calibration to a distribution that we
don’t know.

> Again it would better to have "higher posterior *predictive* density"
> instead of just "higher posterior density". I think it's better to be
> explicit here.

I agree that posterior _predictive_ is an important distinction. But higher
density alone is not all that useful out of context.

> Short story of Posterior predictive checking vs. Cross-validation
> - posterior predictive checking is good for joint predictive distribution checks, but there is no calibration in test statistics
> - in cross-validation checking the test statistics are calibrated (given usual assumptions), but it can be used only for marginal predictive distributions (univariate or m-dimensional where m is the block size)

I very much disagree with this summary. Both are of these approaches
are estimators for the true relative predictive performance scores (again
see my paper that I referenced earlier) which are subject to specific
biases and variances. Neither is calibrated. Ever. EVAAAAAAAR.

Cross validation tends to be more robust to overfitting, but the same
robustness can be manifested in PPCs by splitting the data and fitting
one one partition and comparing the predictive distribution to the
held-out data set instead of using the same data set for both.

[Andrew Gelman]

Right, I wouldn’t plot p-values. I’d do exploratory plots of some sort. I’m not sure what to do in this example. It’s good to have some PPC just to demonstrate the point, but it’s hard to know what to do in such a simple case without there being a particular concern about the model.
A

[Aki Vehtari]

On Tuesday, January 26, 2016 at 5:38:09 AM UTC+2, Michael Betancourt wrote:
> > Sharpness. See "Probabilistic forecasts, calibration and sharpness"
> > https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jrssb.pdf
>
> I dislike this term —

I'm not big fan of the term either, but it's useful as it has a specific definition and avoids the confusion compared to precision.

> the problem is that it’s not about getting the smallest
> predictive interval as possible, but rather the predictive interval that best
> matches the variance of the true data generating process.

I guess that you don't mean what I think your sentence is saying (sorry if the problem is my poor English). Most of the time it's unlikely that we could get "the predictive interval that best matches the variance of the true data generating process". Even if "the true model" is included, the predictive distribution includes the uncertainty on other parameters. For example normal model with unknown variance has predictive variance (1+1/n)sigma, which is not matching the "the variance of the true data generating process" sigma.

> I very much disagree with this summary. Both are of these approaches
> are estimators for the true relative predictive performance scores (again
> see my paper that I referenced earlier) which are subject to specific
> biases and variances. Neither is calibrated. Ever. EVAAAAAAAR.

It seems we are discussing different definitions of calibration (or I've written some nonsense due to my poor English writing skills). I didn't write about calibration of predictive performance scores here (I've written about that in several of my papers). I was writing about checking whether the predictive distributions are calibrated in the sense that cross-validation predictive distribution tail area probabilities have uniform distribution. Even a bad predictor can be calibrated in this sense, e.g. just predict with the marginal distribution of the data.

> Cross validation tends to be more robust to overfitting, but the same
> robustness can be manifested in PPCs by splitting the data and fitting
> one one partition and comparing the predictive distribution to the
> held-out data set instead of using the same data set for both.

If you split the data it's not "posterior predictive checking" as defined in BDA anymore (don't overload the terms). I call this kind of splitting "partial predictive" (which has been used in this sens before me, too). The problem with splitting is that you are not then getting the full joint predictive distribution. Consider the road weather example. You split the data and then you are able to test the performance only on half of the road.


Aki

[Alp Kucukelbir]

this is an amazing thread; would love to see it become a blog post!

alp

[Michael Betancourt]

>> I dislike this term —
>
> I'm not big fan of the term either, but it's useful as it has a specific definition and avoids the confusion compared to precision.

Yeah, but that specific definition isn’t very intuitive which I would
argue leads to more confusing than overloading precision! Especially
if we’re explicit and differentiate between to which distribution’s precision
we’re referring.

>> the problem is that it’s not about getting the smallest
>> predictive interval as possible, but rather the predictive interval that best
>> matches the variance of the true data generating process.
>
> I guess that you don't mean what I think your sentence is saying (sorry if the problem is my poor English). Most of the time it's unlikely that we could get "the predictive interval that best matches the variance of the true data generating process". Even if "the true model" is included, the predictive distribution includes the uncertainty on other parameters. For example normal model with unknown variance has predictive variance (1+1/n)sigma, which is not matching the "the variance of the true data generating process" sigma.

Yes, I agree that this is a relatively common behavior and not an unexpected
one. Because the posterior predictive distribution is a convolution it will in
general be more diffuse then the true data generating process as it incorporates
_both_ parameter uncertainty and measurement variation.

But if the true data generating process is in the likelihood then asymptotically
Berstein-von Mises applies and the posterior predictive distribution will converge
to the true data generating process. What I’m saying is that this is the _ideal_
case to which we strive: the ideal prediction will never be a point mass because
of the inherent variability in the true data generating process.

So we want our predictive intervals to be as close as possible to those of the
true data generating process.

> It seems we are discussing different definitions of calibration (or I've written some nonsense due to my poor English writing skills). I didn't write about calibration of predictive performance scores here (I've written about that in several of my papers). I was writing about checking whether the predictive distributions are calibrated in the sense that cross-validation predictive distribution tail area probabilities have uniform distribution. Even a bad predictor can be calibrated in this sense, e.g. just predict with the marginal distribution of the data.

Okay — I use calibration to refer to the _power_ of the test, not the significance.
So, as you note, you can argue that cross-validation can be made uniform under
the null (the posterior predictive) but you can’t argue that the power for any
cross-validation estimator can be computed, because we don’t have an alternative
model!

Because we can’t compute anything like a power, we can’t differentiate between
bad fits and unlikely data (the power essentially constraints the probability of
unlikely data). Which leaves us to make relative comparisons in which the
sensitivity to unlikely data cancels.

> If you split the data it's not "posterior predictive checking" as defined in BDA anymore (don't overload the terms). I call this kind of splitting "partial predictive" (which has been used in this sens before me, too). The problem with splitting is that you are not then getting the full joint predictive distribution. Consider the road weather example. You split the data and then you are able to test the performance only on half of the road.

Sorry, and I’m happy to argue with people here, but taking something as generic
as the concept of “posterior predictive checking” and shoving into an narrow definition
like that is an unwelcome idea. It makes naming anything else a nightmare. Ultimately
it’s not _overloading_ , it’s just not using a general name for something so particular.

Yes, splitting data induces a bias because you have to estimate the performance of
the full predictive distribution with that from only part of the data (appropriately scaled
up). And you need a product structure on the data to do this anyways. But cross validation
requires the same exact structure and suffers from the same exact biases!

[Bob Carpenter]

> On Jan 26, 2016, at 10:57 AM, Michael Betancourt <betan...@gmail.com> wrote:
>
> ...

> So we want our predictive intervals to be as close as possible to those of the
> true data generating process.

In this repeated binary trials case, we assume there's
a fixed chance-of-success and the trials are exchangeable.
So there's no data-generating uncertainty for that chance
of success (despite it coming from a population distribution
in the generative model), it's just a number. When it's fixed,
the binomial nature of the data induces uncertainty in the
number of successes in a given number of trials, but that's
different. That gives us the upper limit of what our predictions
could be (all, of course, conditioned on the model being "true").

The posterior around the value of the chance of success is all just
epistemic uncertainty on our part --- not an uncertainty of a
data-generating process. As we get more data, our estimate of it
will approach a delta function (which we can call sharp or pointy
or peaky or entering the Bernstein-von Mises zone, or whatever
you guys decide on).

> Sorry, and I’m happy to argue with people here, but taking something as generic
> as the concept of “posterior predictive checking” and shoving into an narrow definition
> like that is an unwelcome idea. It makes naming anything else a nightmare. Ultimately
> it’s not _overloading_ , it’s just not using a general name for something so particular.

It's just semantics, like whether we call something concentrated
or peaky or precise or sharp or pointy. Or like deciding whether
Pluto is a planet. It's not something we discover, it's something
we define by convention for purposes of communication. So there's
no right or wrong answer, just a decision as to which definitions
will be most useful.

- Bob

[Michael Betancourt]

> In this repeated binary trials case, we assume there's
> a fixed chance-of-success and the trials are exchangeable.
> So there's no data-generating uncertainty for that chance
> of success (despite it coming from a population distribution
> in the generative model), it's just a number. When it's fixed,
> the binomial nature of the data induces uncertainty in the
> number of successes in a given number of trials, but that's
> different. That gives us the upper limit of what our predictions
> could be (all, of course, conditioned on the model being "true").
>
> The posterior around the value of the chance of success is all just
> epistemic uncertainty on our part --- not an uncertainty of a
> data-generating process. As we get more data, our estimate of it
> will approach a delta function (which we can call sharp or pointy
> or peaky or entering the Bernstein-von Mises zone, or whatever
> you guys decide on).

Not quite. The chance of success is a parameter that indices
the space of data generating processes in the likelihood, and
any given value uniquely identifies one such data generating
process. But then when we make predictions we have to
consider some random variable of the data, which will have
some distribution on it (like number of successes in 100 trials).
That is the inherent data variability.

Our posterior analyses assigns uncertainty to the “correct”
chance of success parameter, which really assigns uncertainty
over the space of data generating processes.

> It's just semantics, like whether we call something concentrated
> or peaky or precise or sharp or pointy. Or like deciding whether
> Pluto is a planet. It's not something we discover, it's something
> we define by convention for purposes of communication. So there's
> no right or wrong answer, just a decision as to which definitions
> will be most useful.

Yeah, of course, but there’s this nasty “branding” habit in machine
learning that’s leaking into statistics where people assign very
generic names to very specific algorithms/concepts. Which then
requires us to force new names for the real generic algorithms/
concepts.

There’s a reason that math makes great strides when someone
redefines all of the notation and nomenclature into something
reasonable and self-consistent. Look at differential geometry
texts in the 1930s compared to now! It’s a world of difference.

[Dustin Tran]

Michael brings up a great point about held-out data and posteror predictive checks. I tried thinking of LOO-CV as a special case of the partial posterior predictive check. This is similar to in BDA (2ed, p181) where it states that the expected deviance under the posterior (evaluated on all the data) can be seen as the special case of a posterior predictive check. However, how does this extend to CV settings? For example, how does it make sense to evaluate prediction by evaluating the leave-out-out posterior predictive density on generated data from another posterior predictive distribution? Is the most general procedure of LOO-CV+(partial) PPC’s as follows: generate data from the posterior predictive p( yrep | y_i), evaluate log p( yrep | y_{-i} ), and do this for each data point; then compare it to the realized discrepancy, sum log p( y_i | y_{-i} )?

Dustin

[Michael Betancourt]

Again, see http://arxiv.org/abs/1506.02273, especially Section 3.
Ultimately the only thing we can reasonable even approximate
is Equation 3, which is amenable to Monte Carlo approximation
if we had lots of repeated measurements (first equations in 3.2).
We don’t typically have lots of repeated measurements, so we
have to approximate further. We can use the single measurement
we have (Section 3.2.1), we could split the data and assume that
we can simply scale them up to approximate the unsplit measurement
(Section 3.2.2) or we could permute over many possible splittings
(Section 3.2.3). All of these additional approximations induce
new biases and variances in the estimator that can be tricky if
not impossible to quantify in practice.

[Bob Carpenter]

> On Jan 26, 2016, at 3:49 PM, Michael Betancourt <betan...@gmail.com> wrote:
>
>> In this repeated binary trials case, we assume there's
>> a fixed chance-of-success and the trials are exchangeable.
>> So there's no data-generating uncertainty for that chance
>> of success (despite it coming from a population distribution
>> in the generative model), it's just a number. When it's fixed,
>> the binomial nature of the data induces uncertainty in the
>> number of successes in a given number of trials, but that's
>> different. That gives us the upper limit of what our predictions
>> could be (all, of course, conditioned on the model being "true").
>>
>> The posterior around the value of the chance of success is all just
>> epistemic uncertainty on our part --- not an uncertainty of a
>> data-generating process. As we get more data, our estimate of it
>> will approach a delta function (which we can call sharp or pointy
>> or peaky or entering the Bernstein-von Mises zone, or whatever
>> you guys decide on).
>
>
> Not quite. The chance of success is a parameter that indices
> the space of data generating processes in the likelihood, and
> any given value uniquely identifies one such data generating
> process. But then when we make predictions we have to
> consider some random variable of the data, which will have
> some distribution on it (like number of successes in 100 trials).
> That is the inherent data variability.

That's what I meant by the binomial uncertainty. I understand
that's irreducible here.

> Our posterior analyses assigns uncertainty to the “correct”
> chance of success parameter, which really assigns uncertainty
> over the space of data generating processes.

Right --- in the simplest case, we get a beta-binomial prior.
But in practice, the bigger the parameters to the beta part of
it, the better, assuming we have the right mean. And the sky's
the limit here (actually, the binomial's the limit).

>> It's just semantics, like whether we call something concentrated
>> or peaky or precise or sharp or pointy. Or like deciding whether
>> Pluto is a planet. It's not something we discover, it's something
>> we define by convention for purposes of communication. So there's
>> no right or wrong answer, just a decision as to which definitions
>> will be most useful.
>
> Yeah, of course, but there’s this nasty “branding” habit in machine
> learning that’s leaking into statistics where people assign very
> generic names to very specific algorithms/concepts. Which then
> requires us to force new names for the real generic algorithms/
> concepts.

That's not from machine learning or computer science, it's just
how language works. You invent something and give it a name.
How general or specific depends on the application. For instance,
consider "football" or "gravity". The former was originally a misnomer
(you can use your head) and then got overloaded in different countries (for
games where you can use every part of your body); the latter got
reinterpreted from a noun of manner to an abstract concept in physics.

> There’s a reason that math makes great strides when someone
> redefines all of the notation and nomenclature into something
> reasonable and self-consistent. Look at differential geometry
> texts in the 1930s compared to now! It’s a world of difference.

To some extent, words are just placeholders. They may have
associations, but meanings drift over time. Humpty Dumpty was
wrong (words don't mean whatever he wanted them to mean), but headed
in the right direction (words mean whatever the community wants them
to mean). But of course, it's not that organized unless you have
a terminology board that can make decrees about what counts as a planet.

- Bob