Alert: Two critical bugs found in the stats module of IDTxl

[michael...@web.de]

Dear IDTxl users,

unfortunately we found two bugs in the statistics module of our toolbox. 

These bugs concern the sequential max_statistics_sequential computation and the FDR_correction.

Of these, the part concerning the sequential max-stats is most likely of minor importance in isolation -- in the sense that links that were significant with the old code remain significant after the fix with very very high probability. However the exact p-values for each (source x lag) pair, i.e. the 'accepted candidates' in the code​, are not paired correctly between p-value and significant candidate. This is inconsequential as long as no subsequent *per-candidate* FDR correction is performed, but may remove the wrong candiates in the per-candidate FDR correction. This bug does not affect the per-target hierarchical FDR correction (Novelli, Netw. Neurosci. 2019), as the per target p-values used there come from the omnibus test for each target.

The second bug however invalidates the per-target  FDR correction, as again there is a sorting/assignment problem.

We are working with high intensity on correcting the code. Please wait for *release 1.6* which contains these fixes, before running or rerunning analyses !!

Once release 1.6 is available our advice is to rerun analyses if:

    

    1. mTE or bivariate TE-analyses were performed and FDR correction was applied, no matter whether the FDR correction was performed using IDTxl's functionality or an external FDR correction.

    2. p-values per target or per candidate (source x lag) were interpreted directly in support of some conclusion.

    

Unfortunately, these bugs only had very minor or no consequences when there was a crystal clear pattern of highly significant information flows on some links and none on the others.  As our unit-tests and system-tests were of this kind (to easily check for the correctness of results) we missed these bugs despite our best efforts. We will add better test systems.

We apologize for the inconvenience this may obviously cause.

On a positive note, results may improve now, especially when comparing between groups -- so it may make sense to also rerun older, apparently failed analyses now.

Patricia Wollstadt and Michael Wibral,

on behalf of the IDTxl team