bug

[Nikita Khromov-Borisov]

Dear colleagues,

It seems there is a bug in your estimation stats.

I have an attached data set of paired observations.

The results are incorrect and contradictory, see attachment.

Correct results are given by JASP or any other software.

Moreover, I suggest in addition providing the estimation of nonparametric effect size as it realized in JASP https://jasp-stats.org/

Otherwise a lot of contardictions are obtained: CI for the ES covers indifferent value 0, bit p-value based on the Wiclcoxon test is extremely small

Best regards,

Dr. Nikita Khromov-Borisov

Commission on Pseudoscience 

Russian Academy of Sciences (RAS)

RUSSIA

http://independent.academia.edu/NikitaKhromovBorisov

https://www.researchgate.net/profile/Nikita_Khromov-Borisov

[Joses Ho]

Hi Nikita,

As far as I understand, JASP does not use bootstrap resampling to compute the 95% CIs; instead it uses the textbook equations as far as I can tell.

The estimationstats.com webapp uses bootstrap resampling and thus the CIs will not match perfectly with JASP's output.

We do provide nonparametric effect sizes on the webapp (Cliff's Delta), but this is only applicable to unpaired datasets.

Please feel free to contribute to dabestr or DABEST-Python with modifications and improvements of your own!

Hope this clarifies,

Joses

[Nikita Khromov-Borisov]

Dear colleagues,

once again. Here I have attached another example of incorrect calculations.

Best regards,

Dr. Nikita Khromov-Borisov

Commission on Pseudoscience 

of Russian Academy of Sciences (RAS)

RUSSIA

http://independent.academia.edu/NikitaKhromovBorisov

https://www.researchgate.net/profile/Nikita_Khromov-Borisov

чт, 22 авг. 2019 г. в 16:53, Nikita Khromov-Borisov <nikita.khr...@gmail.com>:

[Joses Ho]

Hi Nikita,

Thank you for pointing this out again. There was indeed a bug in certain cases; this should be rectified. You can try again on the webapp, and let us know if you spot any other inconsistencies.

Best,

Joses

On Saturday, August 31, 2019 at 8:17:43 PM UTC+8, Nikita Khromov-Borisov wrote:

Dear colleagues,

once again. Here I have attached another example of incorrect calculations.

Best regards,

Dr. Nikita Khromov-Borisov

Commission on Pseudoscience 

of Russian Academy of Sciences (RAS)

RUSSIA

http://independent.academia.edu/NikitaKhromovBorisov

https://www.researchgate.net/profile/Nikita_Khromov-Borisov

чт, 22 авг. 2019 г. в 16:53, Nikita Khromov-Borisov:

[Nikita Khromov-Borisov]

Dear colleagues,

I have the attached data set, which your program is not able to analyze.

Is it possible to solve the problem?

Best regards,

Dr. Nikita Khromov-Borisov

чт, 22 авг. 2019 г. в 16:53, Nikita Khromov-Borisov <nikita.khr...@gmail.com>:

[Joses Ho]

Hi,

Your dataset has 64414 datapoints. Notably, the Norm group has N=64379, while the Pathol group has N=35.  Aalyzing such a large (and very imbalanced) dataset, the webapp is likely to timeout before the computation is completed. It is hosted and provided for free, so this is perfectly understandable.

If you need to analyze a large dataset, I'd advise install DABEST-Python or dabestr on a server with suitable RAM, and using your custom scripts.

Thanks,
Joses

On Sunday, November 24, 2019 at 1:12:15 AM UTC+8, Nikita Khromov-Borisov wrote:

Dear colleagues,

I have the attached data set, which your program is not able to analyze.

Is it possible to solve the problem?

Best regards,

Dr. Nikita Khromov-Borisov

чт, 22 авг. 2019 г. в 16:53, Nikita Khromov-Borisov <nikita.khromovborisov@gmail.com>:

[Nikita Khromov-Borisov]

Dear colleagues,

please, don't ignore my wails!
Once again, please, check your algorithms. Here I have attached the
data and results of their analysis with three programs. Results and
conclusions observed with your program contradict those obtained with
PAST and JASP as well as many other programs.
BTW the name Hedges should begin with the capital letter.
Otherwise, I will be forced to publish a refutation.

Best regards,

Dr. Nikita Khromov-Borisov
Commission on Pseudoscience
Russian Academy of Sciences (RAS)
RUSSIA

http://independent.academia.edu/NikitaKhromovBorisov
https://www.researchgate.net/profile/Nikita_Khromov-Borisov

сб, 23 нояб. 2019 г. в 20:11, Nikita Khromov-Borisov

[Joses Ho]

Hi Nikita,

In he dataset you provide, do you want to compare the paired difference of BLS_pre - BLS_post, or BLS_post - BLS_pre?

The "diff" column in the dataset suggests you want to compute BLS_pre - BLS_post.

As clearly noted in the paired example dataset on the paired analysis page, the control group (aka BLS_post here) should be in the first column, while the paired test group (BLS_pre) should be pasted in the second column.

I assume you pasted the data as is into estimation stats, hence it would return BLS_post - BLS_pre.

estmationstats.com also uses 5000 resamples to compute the CI of the effect size. If you use 100 resamples, you will get a CI much closer to what JASP et al produce. (Refer to this Jupyter notebook here.)

While I appreciate you comparing our output against other opensource statistics software, please note that we generate our 95% CIs via non-parametric bootstrap resampling. I am not certain how the other software pacakges do it, but I suspect it is via parametrized formulas based on the student's t distribution. So while the output might not match up, this is to be expected because we use bootstrap resampling to obtain the CIs, and also we use permutation testing to obtain the p-values.

If you download the CSV file after performing the analysis on estmationstats.com, you will see we also provide the paired Student's t-test P value there.

If you want to change the number of bootstrap resamples, consider using our Python or R packages, where you can specify this value.

Hope this helps,

Joses