what is false positive?

[RichD]

A schoolboy question:

Thinking about the COVID issue, I haven't seen any data
on false positive/negative rates on the tests.

But first, a definition of 5% false pos.:

i) Given 100 positive results, it means 5 are erroneous?
or,
ii) Given 100 subjects non-infected, it means 5 will return positive?

You could translate these to the predictions for an individual.
Odd, that I never thought about this before -

And why is it, in medical testing generally, they focus
on the false positive rate, but discount false negatives?

--
Rich

[Rich Ulrich]

On Tue, 18 Aug 2020 15:44:34 -0700 (PDT), RichD
<r_dela...@yahoo.com> wrote:

>A schoolboy question:
>
>Thinking about the COVID issue, I haven't seen any data
>on false positive/negative rates on the tests.
>

Test + Test -
True + A B
True - C D

A = True Positive
B = False Negative
C = False Positive
D = True Negative

>But first, a definition of 5% false pos.:
>
>i) Given 100 positive results, it means 5 are erroneous?

C / (C+A) ? no

>or,
>ii) Given 100 subjects non-infected, it means 5 will return positive?

C / (C+D) ? yes

A quick Google gives the definition,
"The false positive rate is the proportion of all negatives that still
yield positive test outcomes"

"Sensitivity" and (negative) "specificty" are the terms used
more often in epidemiology papers, to comprise the errors
in both directions, Together, they define the reliability of
the test. Technical papers will always discuss BOTH since
most of the tests have a quantitative cutoff that can be
adjusted, whereby you increase either at the expense of
the other.

The cutoff that is used will sometimes take into acount the
base-rate of the disease, since a change in base rate will
change the actual number and rate of errors in a particular
sample.

>
>You could translate these to the predictions for an individual.
>Odd, that I never thought about this before -
>
>And why is it, in medical testing generally, they focus
>on the false positive rate, but discount false negatives?

Your phrase - "in medical testing generally" - does not describe
the reality of medical testing. It may describe what you have
been reading, lately, in newspapers about covid-19.

Below are links to some sources of articles that include some
on sensitivity and specificity concerns. This content was originally
posted here by David Jones on June 1, 2020, under Subj:
"Analyzing Covid Data"



<<
I have not tried to follow any of the above. But, anyone with a
statistical interest in this epidemic should probably know about the
following list of articles associtaed with the Significance magazine:

https://www.significancemagazine.com/business/647

The list features the UK rather heavily and relates to the RSS
Covid-19
Task Force, which is outlined here:

https://rss.org.uk/news-publication/news-publications/2020/general-news/rss-launches-new-covid-19-task-force/

and:

https://rss.org.uk/policy-campaigns/policy/covid-19-task-force/

There is some overlap in these lists.
>>

--
Rich Ulrich

[David Jones]

For a quick look at some actual numbers regarding test accuracy, see
https://www.finddx.org/covid-19/sarscov2-eval-molecular/molecular-eval-results/

... this also shows the large number of different tests available and
indicates how well the test accuracies are known

[Bruce Weaver]

Hi Rich. Here is a BMJ article & infographic that might help:

https://www.bmj.com/content/369/bmj.m1808

Notice that it unlike the example Rich Ulrich showed, it puts the Test results in the rows (T+ and T-) and the true disease status in the columns (D+ and D-). Using this approach, Rich's table would be recast as follows:

D+ D-
T+ tp fp
T- fn tn

Or using a-d:

D+ D-
T+ a b
T- c d

With this layout, the usual calculations are:

Sens = TPF = p(T+|D+) = a / (a+c)
Spec = TNF = p(T-|D-) = d / (b+d)
PPV (or PV+) = p(D+|T+) = a / (a+b)
NPV (or PV-) = p(D-|T-) = d / (c+d)
FPF = p(T+|D-) = b / (b+d) = 1 - Spec
FNF = p(T-|D+) = c / (a+c) = 1 - Spec

The final F in TPF, TNF FPF and FNF stands for "fraction". Sometimes you see "rate" in place of fraction, but it really is a fraction (or proportion).

Novices (and I'm not suggesting you are one!) often find it easier if this is just expressed in terms of row and column percentages. For the layout I have shown above:

Column % for a = Sens (or TPF)
Column % for d = Spec (or TNF)
Column % for b = 1-Spec = FPF
Column % for c = 1-Sens = FNF

Row % for a = PV+ (or PPV) -- predictive value of a positive test
Row % for d = PV- (or NPV) -- predictive value of a negative test
Row % for b = 1-PV+
Row % for c = 1-PV-


Getting back to your question:

> i) Given 100 positive results, it means 5 are erroneous?

This is describing the row percentage for cell b in the table above, so is equal to 1 - PV+.

> or,
> ii) Given 100 subjects non-infected, it means 5 will return positive?

This is describing the column percentage for cell b, which is equal to 1-Spec or the FNF.

As I said, this is the *conventional* terminology regarding false positive and false negative fractions (or rates). But who knows for sure whether journalists are following these conventions!

HTH.

PS- In case it is not obvious, Sens is short for sensitivity and Spec for specificity.

[Rich Ulrich]

On Wed, 19 Aug 2020 06:43:31 +0000 (UTC), "David Jones"
<dajh...@nowherel.com> wrote:

>
>
>For a quick look at some actual numbers regarding test accuracy, see
>https://www.finddx.org/covid-19/sarscov2-eval-molecular/molecular-eval-results/
>
>... this also shows the large number of different tests available and
>indicates how well the test accuracies are known

Thanks -
A few comments -

That says it is updated on July 3, so there are more tests
available by now, I'm sure.

Those are high values for both sensitivity and specificity,
many of them measured for their samples at 100%.

First, I am surprised and impressed. The numbers I
saw (probably in June) for a couple of tests were lower.
Those high numbers give me more reason to trust that
recent re-analysis of older data (mostly from April) where
they concluded that the "actual number" of cases ranged
from twice to 13 times (varying by place) the number of
"confirmed diagnoses".

Second, those were for "clean" samples. Collection in
the field introduces errors and inaccuracy. For instance,
the "deep nasal swab" is uncomfortably deep, and can
be done wrong, failing to get a good sample (ergo, lower
sensitivity).

One of the developments I have seen hyped in the last
week or so is the potential for a cheap saliva-test that may
be administered at home, with quick results. The lack of
ideal sensitivity is offset by cheap-and-quick. I think I read
that the US government is paying about $100 per test, though
I don't know how much of that goes to the labs.

--
Rich Ulrich

[David Jones]

Yes, this earlier article
https://www.significancemagazine.com/science/667-comparing-and-assessing-covid-19-tests
quotes some rather lower success rates for tests.

The high succces rates quoted seem to relate to this statemen:t
"On 19 February 2020, FIND launched an expression of interest (EOI) for
test developers of in vitro diagnostics (IVDs) that detect SARS-CoV-2
nucleic acid (molecular tests). The EOI closed on 9 March 2020. Over
200 submissions were received and applications were reviewed ..." Out
of which the 21 "best" were selected for independent evaluation and
inclusion in the tables:
https://www.finddx.org/covid-19/sarscov2-eval-molecular/
... so the tables only include tests available early on.

A wider view might be found on this page:
https://finddx.shinyapps.io/COVID19DxData/
which eventually provides a graphical scatter plot of Sensitivity
against Specificity, subject to various options ... but it does take
some time for the graph to appear, so some patience is needed. I don't
really unbderstand what is going on here, but you might want to move
the top button from "Antibody" to "Molecular".

[Rich Ulrich]

I think - antibody is whether you might have long-term immunity;
molecular is the virus by PCR; antigen is the virus otherwise (?).

I thoroughly did not understand "heat map."

The page was last updated today, so this one may be worth
checking for changes across the weeks.

I read today that Penn State is considering using a scratch-and-
sniff test -- The article said that casual reports get 50% of cases
reporting on total loss of sense of smell as an /early/ symptom,
and it may be 75% that admit to it on being pressed. That is
good sensitivity for a very simple, immediate, and cheap test.
Reusable, too, for everyone nearby?

The usual coronavirus case does /not/ have nasal congestion -
common cold could be the source of most false positives, but
the loss of smell is more severe for covid-19.

--
Rich Ulrich

[David Jones]

Rich Ulrich wrote:

>
> One of the developments I have seen hyped in the last
> week or so is the potential for a cheap saliva-test that may
> be administered at home, with quick results. The lack of
> ideal sensitivity is offset by cheap-and-quick. I think I read
> that the US government is paying about $100 per test, though
> I don't know how much of that goes to the labs.

This article ...

https://news.sky.com/story/coronavirus-what-are-the-different-types-of-covid-19-tests-12068081

outlines some of the different tests that might be available. Once
again, it is UK-centric.

[David Jones]

THis letter from the RSS to a newspaper (September 11)
https://rss.org.uk/RSS/media/File-library/News/Press%20release/Letter-to-the-Times-on-government-moonshot-testing-plans-110920.pdf
suggests that the tests currently being used for those who might have
the infection are rather poor for certain intended uses.

[Rich Ulrich]

Thanks. I suspect that the same holds for most testing in the US,
but it is worth "discussion" - as the author says - about what
everyone is talking about as "the current tests" and the alternatives.
Do positive test resuts (somewhere) get re-tested with more precision?
I believe they were doing that at the NFL training camps, because
I did hear of a couple of false positives from their testing -- they
could not have decided that if they did not do the retest.

article -
<< Tests cause harm when they miss or wrongly diagnose cases.
Our current tests have 1 and 2% false positive rates – which, when
millions are being tested every day, risks causing personal and
economic harm to tens of thousands of people. This problem is
exacerbated if the new tests, as is likely, are less accurate than the
ones used currently.
If mass-testing can give people confidence that they are
disease-free, tests need to detect nearly all cases. Our current tests
miss around a fifth of those with the disease – if the new tests are
even less sensitive, they may not be accurate enough for the safe
running of events but could be useful for complementing social
distancing measures. >>

--
Rich Ulrich

[David Jones]

Back to pool testing ... a recent BBC radio programme on pool testing
can be accessed here (9 minutes long):
https://www.bbc.co.uk/sounds/play/p08rwy4n

[Rich Ulrich]

Stephen Colbert (late night comic) passed along the news
item that the Finns have trained dogs to detect the
coronavirus. I see several news stories available. The dogs
are being tried at an airport terminal, for incoming passengers.

This one repeats what Colbert said, that the dogs can
detect the virus potentially a day or two earlier than
PCP tests. It also says "94.5% accuracy" - which
I hope means sensitivity.

https://www.newsweek.com/dog-smell-coronavirus-covid-testing-pcr-nose-swab-finland-helsinki-1533905

This one also says "94% accuracy", also without being clear
on the criterion. It adds comments about the logistics of dogs
being used for this. To work in the airport, they have to
tolerate the noisy environment.

https://www.nytimes.com/2020/09/23/world/europe/finland-dogs-airport-coronavirus.html

--
Rich Ulrich