recommended statistics package? Requirements: ANOVA, Brown-Forsythe

[王富民awaw]

Hi follow Gophers

I wonder is there a canonical, verifiably correct Go package for statistics?

In particular, Go code that does the Brown-Forsythe test of equal variance.

Ideally in pure Go, but linking with CGo is OK.

A search on Google and pkg.go.dev does not return helpful results.

I wonder is there anything that the community could share?

[Jan]

hi, I did a quick search and I didn't find anything in Go. But looking at the definition and at one implementation in JS, it sounds something relatively easy to write and share :)  You can use the R implementation to create some test datasets. Maybe gonum/stat would be a potential home for such a function ? What do you think ?

cheers 

[Martin Schnabel]

Hi,

I attempted to translate the linked JS implementation for fun. Maybe
someone can use it as a starting point and correct or verify its
correctness.

https://go.dev/play/p/Wrw2yDRof0z

Have fun!

On 10/23/23 07:38, Jan wrote:
> hi, I did a quick search and I didn't find anything in Go. But looking
> at the definition and at one implementation in JS

> <https://github.com/lukem512/brown-forsythe-test/blob/master/src/brown-forsythe.js>, it sounds something relatively easy to write and share :)  You can use the R implementation to create some test datasets. Maybe gonum/stat <https://godocs.io/gonum.org/v1/gonum/stat> would be a potential home for such a function ? What do you think ?

>
> cheers
> On Friday, October 20, 2023 at 10:54:55 AM UTC+2 王富民awaw wrote:
>
> Hi follow Gophers
>
> I wonder is there a canonical, verifiably correct Go package for
> statistics?
> In particular, Go code that does the Brown-Forsythe test of equal
> variance.
> Ideally in pure Go, but linking with CGo is OK.
>

> A search on Google and pkg.go.dev <http://pkg.go.dev> does not

> return helpful results.
> I wonder is there anything that the community could share?
>

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/7ee10c0f-8af6-4b31-baaf-ce2ccb9c0211n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/7ee10c0f-8af6-4b31-baaf-ce2ccb9c0211n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Jan]

So cool!

[Jason E. Aten]

For the ANOVA, I usually just call from Go into R for such things -- at least until I 

can validate if its the right thing to do/ meets the sensitivity/power needs of the analysis.

https://statsandr.com/blog/anova-in-r/

https://github.com/glycerine/rmq#and-the-reverse-embedding-r-inside-your-golang-program

[王富民awaw]

Hi Jan,  Martin, and Jason

Thanks for your tips and encouragement for rolling our sleeves.

I have put together a small statistics library:

stat package - github.com/fumin/stat - Go Packages

For my own needs, in addition to Brown-Forsythe, I also needed the Welch t-test and tools for multiple testing, so these are what's in the above package.

[Jason E. Aten]

王富民awaw: Very nice.

The Levene function, though, is confusingly named. The Levene test is distinct from the Brown-Forsthe test, so mixing up the names

seems odd.[1]

[1] quoting https://www.statisticshowto.com/brown-forsythe-test/#:~:text=The%20Levene%20test%20uses%20deviations,a%20test%20that's%20more%20robust.  below: (see the underlined sentence, underline and bold there is mine)

How the Test Works

Both the Levene and B-F tests transform dependent variables for use in an ANOVA test. The only difference between the two tests is in how those transformed variables are constructed. The Levene test uses deviations from group means, which usually results in a highly-skewed set of data; This violates the assumption of normality. The Brown-Forsythe test attempts to correct for this skewness by using deviations from group medians. The result is a test that’s more robust. In other words, the B-F test is less likely than the Levene test to incorrectly declare that the assumption of equal variances has been violated.

The test works as follows:

• The median is calculated for each factor level group.
• The median value is subtracted from each dependent variable in the group.
• An ANOVA is run with the transformed variables. If a factor’s p-value is less than the significance level (usually 5%), the population variances are not equal.

W or F statistic?

The test statistic used in a regular ANOVA is an F-statistic. The statistic used in an ANOVA with transformed variables is sometimes called a W-Statistic — but it’s really just an F-Statistic with a different name. It should not be confused with the coefficient of concordance W-statistic, which is used to assess agreement between raters.

Cautions

For the most part, the B-F test is thought to perform as well as or better than other available tests for equal variances. However, Glass and Hopkins (1996 p. 436) state that the Levene and B-F tests are “fatally flawed”; It isn’t clear how robust they are when there is significant differences in variances and unequal sample sizes. Hill et. al (2006) advise repeating the test using a non-parametric method.

[王富民awaw]

Hi Jason

Yes, I concur that the function "Levene" named in my package should really be called Brown-Forsthe.

I was vacillating between this correct naming versus following the nomenclature of scipy from which my implementation is numerically tested against.

I looked at a couple of popular statistics software, and it seems that most differentiate between Levene and Brown-Forsythe

* JMP

* SPSS

* R

Therefore, perhaps I should not be following scipy and rename my function from Levene to BrownForsythe?

I admit I am no statistician and would appreciate some professional guidance.

[Jason E. Aten]

On Monday, October 30, 2023 at 3:14:29 AM UTC 王富民awaw wrote:

Therefore, perhaps I should not be following scipy and rename my function from Levene to BrownForsythe?

 

Since you have made it available to the public Go community, yes, it would be a good idea to name it BrownForsythe,

as it is using the deviations from the group medians.

Also the function makes the assumption, as does Median, that each group's samples[j] is already sorted. You

should document that assumption, or sort.Slice() each group's sample before calling Median.

[王富民awaw]

Hi Jason

Thanks for your suggestion.

I have renamed the function Levene to BrownForsythe, as well as added a comment to note that it assumes its input to be sorted ascendingly.