Criticism in beta

[Zoya]

Ouya shipped some early units last week to backers of the Kickstarter. Following negative reviews on the verge and engadget, they have issued a statement saying that the console is not yet ready for press reviews because this early release constitutes a testing phase, and as such no criticism is valid until the official launch date. (source)

I can completely sympathise with a developer of any product not wanting to be judged by the media while still testing. But this brings up a difficult problem: with beta testing and final release becoming increasingly fluid to the point that many games and products are never going to leave beta, at what point is criticism legitimate?

As an example, last year I wrote up my impressions of Phantasy Star Online 2 not long after it officially left Beta and launched in Japanese. I was mostly negative about it back then, but when I returned to the game five months later my exerience subtly but noticeably improved. Part of this will be due to bad FTUE design, but it seems reasonable to assume that Sega had iterated on at least some of the game's features. And Sega are not even a shining example of iterative design.

At what point can we start criticising Dota 2? Is it still too early, even though almost everyone on my Steam list seems to have played it already? What about PS Home, which Sega have indicated is unlikely to leave Beta at all?

With British soap operas, you often get a lot of reviews coming out when there is a change of writer or a new set of characters are introduced to the show. The equivalent would be to periodically critique perpetual beta games when a press release announces major updates. But is that just pandering to a company's PR department?

[Alan Williamson]

I think this is a Google problem, where products were in 'beta' with no real completion goals. If I'd put money down for an Ouya, I'd want a finished product. Their definition of 'beta' seems to be 'be a guinea pig for our product', which isn't quite the same thing. You can't patch bugs in hardware- unless you use those magnets on the controller to make some kind of robot Band-Aid I guess.