How about making a beta release before making an official one

[Tarun K]

Hello all,

This is my understanding of how official Selenium releases are made - 

There are major bux fixes or feature additions > Unit test case execution (or may be some thing else) > All look good > Time for new Selenium release.

But despite all efforts at times defects like this slip in - http://code.google.com/p/selenium/issues/detail?id=2513

I was thinking if Selenium could leverage its vast user community before making an official release.

So we still follow the same cycle -

There are major bux fixes or feature additions > Unit test case execution (or may be some thing else) > All look good

But a new phase here -

> And now we ask Selenium users to test new release in a time bound manner (May be by announcing it on Selenium User group, may be having beta testing last for a week or may be less/more time)

This would be as simple as Selenium users replacing existing Selenium library with new release > Executing their tests and hopefully not finding any critical exception due to new library. and if 

All look good > Time for new Selenium release

But if there were a critical error reported by user -

Hot fix > Beta testing by Selenium Users > All look good > Time for new Selenium release

Though I am not sure how long this cycle could go on.

Thoughts/comments?

~tarun

[David Burns]

As we move forward with the project plan of standardisation of the API, issues like this are going to be more of a browser issue.

If we do see issues like this, and the browser vendor owns the implementation like with Google Chrome, should we hold up a release? I for one would be extremely against that. So am -1

At the moment we are roughly one weekly releases so unless it's a total blocker, and as in the build is red, we should carry on as normal. Everyone with the commit bit should run tests as often as they can so we may be aware of issues.

David

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-developers/-/JE28xgLGUgMJ.
To post to this group, send email to selenium-...@googlegroups.com.
To unsubscribe from this group, send email to selenium-develo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/selenium-developers?hl=en.

[Daniel Wagner-Hall]

I'm not sure what this gains us in a world where every older version is available. I believe the current mechanism is as you describe, with one additional step: if users have issues, they down grade until the issue is resolved. This may be annoying for them, but it's also a workable solution, and means that we have more users doing that testing.

As we've moved to ~weekly releases, I'm sure a lot of people are also just skipping some updates, too.

On 10 Nov 2011 05:33, "Tarun K" <tku...@gmail.com> wrote:

[Andreas Tolf Tolfsen]

* Also sprach Daniel Wagner-Hall <dawa...@gmail.com>:

> I'm not sure what this gains us in a world where every older version
> is available. I believe the current mechanism is as you describe, with
> one additional step: if users have issues, they down grade until the
> issue is resolved. This may be annoying for them, but it's also a
> workable solution, and means that we have more users doing that
> testing.
>
> As we've moved to ~weekly releases, I'm sure a lot of people are also
> just skipping some updates, too.

I agree pretty much agree with what's being said here.

Maintaining different release channels for Selenium would be _a huge_
undertaking considering the complexities of this project. Considering
that we're not pushing Selenium to any package management systems for
OSes, this is a workable solution.

But:

I'm unsure what the “@Beta” thing means in all of this. Does it mean we
will continue to maintain and release important fixes for the stable
(previous) version, effectively introducing two release channels? Or
does it mean the users will just have to upgrade to the latest bleeding
edge to get patches for bugs?

This sounds suboptimal to me. I realize that Selenium is not an
ordinary project and cannot offer the same LTS release schemes as other
traditional software (like in the traditions of Debian) without major
effort, and frankly there is no need to do so for the core project
either, but there is a chance we'll be communicating things less clearly
by separating between final- and beta versions.

I'd personally like to see Selenium reach such a stable state that a
user wouldn't have to upgrade in between minor (x.Y.z) releases, but
with things moving so fast within the browser automation world, I
realize that might be a far stretch.

I'd like to hear the reasons for introducing “@Beta”, as it seems to me
that Daniel Wagner-Hall's described approach above (“if things doesn't
work for you, please downgrade and use the previous version until we get
time to fix it and release a newer version”) seems more than adequate to
me. It's true that this isn't optimal in terms of user-friendliness,
but then this is all just really a symptom of our testing environment
not being good enough.

Whenever we catch a bug like above, we should actively be finding ways
to test for that in the future. That said, we also need to make it
easier and more stable to run the full stack of tests. Or, in the
least, ensure that our CI environment is functioning correctly.

Thanks!!

--
Andreas Tolf Tolfsen
Systems Developer, Core Systems
Opera Software ASA

[Simon Stewart]

The @Beta annotation is a marker that essentially says "this is an
experimental feature that is liable to change, but we'd love some
feedback." Our policy for removing APIs that are not marked beta is
that they are deprecated for a release, and then removed in the next.
There is no such promise for @Beta features, allowing us to iterate
through possible APIs until we find that we and our users like.

As a concrete example, we've marked the window control APIs as @Beta
in this release as we're not entirely sure we've got the user-facing
pieces right. Let's see what the reception is and how implementors
feel about it.

Simon

> --
> You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

[Tarun K]

It's definitely dev's call. 

And chrome driver may be one long running exceptional issue. But I hope we want users to upgrade to new release every week (or so). And not keep going back to previous release because of some issues.

[semperos]

To throw another consideration in...

In my opinion, x.x.0 releases of software are expected to be stable. I haven't found Selenium's releases to be buggy for my own use cases, but I think this isn't a point so easily shrugged off, and should be kept on the "back burner" as Selenium goes forward.

It is precisely because of the complexity of this project that creating pre-release builds on specific timetables (which encourage people to focus their testing efforts) are a Good Thing. While these releases aren't in OS package managers, they are posted to places like the central Maven repository, and telling people to go back to a previous version definitely does not instill confidence in the project.

-Daniel

[Paul Hammant]

You don't know it, and won't necessarily agree given I'm pointing it out, but you're making a case for better regression tests before release. Only.

- Paul

--

You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/selenium-developers/-/gsGxSVyylAkJ.

[Tarun K]

No, I would definitely agree to it. It is indeed for better regression testing....