Alpha Wars Hack Tool

[Jannet Nevels]

Ihave strong doubts that someone reading a blog called barely significant in this troubled period might have escaped the saga of the summer. However, as this story relates closely to the name of this blog and to the motivations behind its creation, I could not help myself to write a summary of this saga (or of its beginning, at least), to inform any unfortunate people that might have missed it.

The adequacy of Null-Hypothesis-Significance-Testing (NHST) -and particularly of the dichotomised use of p-values- to scientific progress has long been questioned (e.g., Bakan, 1966; Gigerenzer, Krauss, & Vitouch, 2004; Kline, 2004; Lambdin, 2012). Despite this wealth of critics, the p-value stays one of the favourite statistical tool of the experimentalist. As an optimistic person though, I like to see recent institutional changes such as the ASA statement on the use of the p-values as an encouraging sign (Wassertein & Lazar, 2016). However, statements such as the ASA statement do not obtain a majority of agreements, neither within statisticians nor within applied scientists. Moreover, efforts to distancing p-values from scientific daily practice might be hindered by alternative usages of NHST that have been suggested during the last years (e.g., the New Statistics of Cumming, 20121).

This summer was very prolific in regards to this debate, because many new ideas on NHST were presented and discussed (in pre-prints but also in blogposts). Noteworthy here is the rythm of the debate, that has been insanely increased by the use of social medias and by preprint sharing. In this post I will then try to summarise the highlighits of this exciting saga.

Their second point is that lowering the threshold to 0.05 would lead to a more reasonable rate (lower) of false positive. The following figure shows the false positive rate as a function of power, for different levels of significance and different prior odds that there is a true effect.

This proposal also implies (and this is maybe the most crucial point) that sample sizes should increase (of approximately 70% on average), in order to keep power constant and to not inflates the rate of false negative.

Immediately after the publication of this preprint, the rebellion started to organise online, with Daniel Lakens as its head. Suddenly, a wealth of blogposts explaining why this proposal was a terrible idea appeared (see for instance here, here, here, or here). Three days later, Daniel Lakens launched on Twitter a massive appeal to a generalised rebellion, asking help to whoever wanted to join the fight.

The main argument of this proposal is that p-values should be used in a Neyman-Person spirit, meaning as a way of balancing type-I and type-II errors. Accordingly, they suggest that the significance threshold should be calibrated according to study-specific objectives, and stressing that researchers should transparently report and justify all choices they make when designing a study, including the alpha level. On how to choose this threshold, they add that this decision should be based on statistical decision theory, where costs and benefits are compared against a utility function (Neyman & Pearson, 1933; Skipper, Guenther, & Nass, 1967).

Although we agree with Benjamin et al. (2017) that the relatively high rate of non-replication in the scientific literature is a cause for concern, we do not believe that redefining statistical significance is a desirable solution: (1) there is not enough evidence that a blanket threshold of p

Instead of the current exclusive reliance on p-values to draw inferences from empirical data, they suggest that the p-value be demoted from its threshold screening role and instead, treated continuously, be considered along with the neglected factors as just one among many pieces of evidence. Although this idea was admittedly already present in the proposal of round II (Lakens et al., 2017), their conclusion diverges quite considerably from the conclusion of Lakens et al., as they recommend to entirely abandon NHST. This suggestion is partly based on the well-known criticisms of NHST as based on a non-existing, non-realistic nil hypothesis, that is summarised in these lines:

Another of their concern is the problem of the dichotimisation of evidence into statistically significant and non-significant results, that according to the authors, would lead to biases when reasoning from such statistics, leading to confuse the statistical and the practical significance of a result. They also insist on the urge to consider, along with the p-value, what they call the neglected factors, that is, prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain.

Moreover, in the meantime, a comment has been published in Nature Human Behaviour (Amrhein & Greenland, 2017), also pushing to remove statistical thresholds, and arguing that a lowered threshold for significance would only accentuate the pitfalls that the first proposal wanted to highlight, by unreasonnably increasing our confidence in findings with p

Since the first upload of this post, a new fifty-authors comment has been submitted to Nature Human Behaviour, for which a preprint can be found on PeerJ. While this comment (lead by David Trafimow and Valentin Amrhein) is echoing some of the propositions put forward by McShane et al., it also brings some new arguments into the light. For instance, it highlights the point made by Trafimow & Earp (2017) about type-I and type-II errors, namely, that their relative importance might differ across study, researchers, or disciplines (a point similar to the proposal lead by Lakens), and that the domain to which they apply is often blurry and usually remains to be defined. They also stress that the relative importance of type-I and type-II errors depends on a large variety of factors, rendering undesirable any fixed recommendation concerning the alpha level.

Concerning replicability, they argue that shifting from the usual threshold to the .005 one would increase the importance of the population effect size in obtaining statistical significance. Indeed, if sample sizes remain similar, such a proposal would favour huge population effect sizes and make smaller effect sizes more difficult to obtain. Based on a tasty example from the history of physics, they demonstrate that replicability should not depend on the population effect size. They add:

Finally, they relay the idea presented in Trafimow (2017) & Trafimow & MacDonald (2017), of using a priori inferential statistics, in replacement for NHST. They also stress the importance of considering multiple sources of information for inference (i.e., not relying on a single index, such as a p-value or a Bayes Factor), and suggest that statistical inference should not be based on single studies.

In this blogpost, I tried to summarise the main official contributions that have been published these last months around the alpha-wars. It is highly probable that a lot of new contributions will add to this list, and so this post may be edited / updated regularly.

After moving beyond the phase of paper prototypes, or prototypes built in something like Axure, we'll often build quick and dirty web prototypes in HTML, CSS, and Javascript. So far, we've taken the same approach as with any other prototype, in that these are meant to be tested, used to garner user feedback, and eventually discarded.

However recently we've been approached on multiple occasions by customers to continually add features to these prototypes until they essentially become a minimal viable product. When should the transition typically be made from throw-away prototype to alpha software in the UX process?

The alpha is something you give to actual users. This is the version, if successful, you hope to build upon to become the beta and eventually the product you release to the public. I would hope your alpha is built upon a solid enough foundation that you can built upon.

A hacky prototype usually provides a very poor foundation. If you do have a prototype based off a proper framework (very rare and I question whether you could have gotten away with a more low-fi prototype), then when all features in your MVP is complete and your devs feel they have sufficient stability to support your X number of alpha clients. And you have a plan in place to handle bugs from clients etc... then it's okay to start alpha testing your software.

My experience is that if you leave any stone unturned with your design - ie, any place for interpretation - the developers will come up with all sorts of solutions that can very well be counter-usablity.

In this day and age, where many designs involve an excess of interactions (and animations), functional prototypes become more and more important. Personally, the interactive requirements of some designs mean that spelling these out in a static document makes no sense whatsoever. I find myself sometimes delivering designs with an interactive prototype (an actual website) saying "this is how it should work like" and only spell in the design document things you cannot show (or not obvious) on an interactive prototype (eg, "Sorting by 'active' means by the most recent timestamp of both posts and replies").

Having had my time as a developer, I have recently ditched wireframes altogether and jump straight from excessive sketching to working prototypes using HTML, CSS and AngularJS. The backend is all a stub simulating the real system, but the front end, in quite a few cases at least, can be copied-and-pasted to the actual system.

However, as nightning mentioned, speed is critical for these prototypes, so although the prototype works as expected, its code is fairly dirty in most cases (you don't spend time to code it right, you just need it to work). So some extra refactoring work is often needed.

I think this depends on your software methodology. This somehow reminds me to eXtreme Programming (XP), where you start with the simplest solution and keep adding new functionality through short development cycles.

3a8082e126