New Check have to pass through experimental project – sevntu.checkstyle

[Roman Ivanov]

Summary from https://sourceforge.net/mailarchive/forum.php?thread_name=1382680537.750410093%40f325.i.mail.ru&forum_name=checkstyle-devel

Please consider to transfer new Check ideas (only) from Checkstyle to Sevntu.Checkstyle project. 

After all that Checks that we done in sevntu.checkstyle project I do not believe that it is possible to finish Check after first implementation and without proper/long testing on big ugly sources.

Lets make new ideas(Checks) to get maturity in sandbox/experimental project – sevntu.checkstyle.

We already able to be used in most famous IDE(Eclipse, Netbeans, Idea) and maven and ant(ivy), and unstable state for sevntu.checkstyle project is OK in comparison to main project Checkstyle.

Right now Checkstyle is in vicious circle  - no body contribute from outside because Admins does not apply patches, admin does not apply patches because afraid of awful code support and testing, Checkstyle project is not developing because Admin have not focused on it full time and nobody contribute from outside.

As said before, to apply new Check or feature from outside Admin need to spend significant amount of time to be sure that Check is make sense and do work well on huge amount of sources - so Admin will never apply new Check even it is reasonable and useful.

In my experience, even I did 10 code reviews of Check, the same amount of testing against almost all sources I have - Check still have Exceptions during execution on non trivial code after release. The only factor helps to mature Check is a time and being alive in sandbox extension that always do code validation whenever you do development during your work, the more code was validated the better.

We need to give community way to make changes, test them-self, apply to their work process, test one time, let use it to others and then do contribution to Checkstyle. 

Yes, we could put in main Checkstyle issue/reminder to port/copy particular idea to main project.  But I do believe that even author of new Check will remind us to move him out of unstable project(our sandbox). 

Sanbox project could make each year graduation voting - where admins decide what Check is mature enough to go to main project.

As result I would rather skip additional issue registering in main project. It will be too much of them. It will not be manageable. But if  all admin vote for double issue registering - I am ok.

[Daniil Yaroslavtsev]

> even I did 10 code reviews of Check, the same amount of testing against almost all sources I have - Check still have Exceptions during execution on non trivial code after release

You are right. There should be at least few months of complete testing on real code to say that check is stable.

> I would rather skip additional issue registering in main project. It will be too much of them. 

Agree. But, maybe, we should create issues such as 'Port checks xxx-yyy from sandbox to main repo' for main Checkstyle project.

[Roman Ivanov]

> Agree. But, maybe, we should create issues such as 'Port checks xxx-yyy from sandbox to main repo' for main Checkstyle project.

that will create massive amount of issues that are on hold and will not be applied soon. Example: NameAbbreviationCheck it was done at Oct 2011 and after that it got 3 bug fixes and kind of final version is Sep 2013 - 2 years!!!! to become stable. I will try to get approval from Admins for it might be in next half of year when I have free time to do it.

it is not required, as: 

if Check is done and still required - smb will demand its porting (it could be even Admins desire to port other developer idea).

if Check is done and nobody is require it to be ported - it means all are satisfied to be hosted in Sandbox, no need to change anything.

Rule of thumb: we make porting only on high demand or activity of Author that help to port and test and .... . If demand if weak it is sign that it is not required or worth doing for now.

[Roman Ivanov]

Moratorium will be stopped or we will start massive Check moving from sandbox project,  after fixing:

https://github.com/checkstyle/checkstyle/issues/555

https://github.com/checkstyle/checkstyle/issues/556

[Roman Ivanov]

Reconsideration of moratorium period:

I would allow any(approved) new Check to appear in Chekcstyle (even without sevntu.chekcstyle stage) if contributor fix 5 functional issues(bugs) in our issue tracker.

Reason:

we still have constantly growing amount of functional bugs in our issue tracker (that is not a regression, just feed back become more intensive).

Our team enforced all famous static analysis tools and 100% coverage for all code, Test Driven Development is enforced. 

In addition to this basic check for Exceptions was added to Travis.

So regression is always possible but unlikely happen.

We still lack of validation of functional regression - some tool that check checkstyle reports over a project before and after fix.

Introduction of new Check always result in few bugs appearance after release, no matter how thoroughly we did a testing.

Usual case contributor come, introduce Check and disappeared (stop contributing). So after release few issues with this new Check become our team responsibility and backlog is growing.

Solution: Contributor pay off debt of previous contributors (fix 5 bugs) and got a approval for new Check introduction. If he stay with us and fix all issues - that is awesome. If contributor disappear (worst case) we will get few more bugs as it was before.

[Roman Ivanov]

moratorium is removed.

We have good tools to test any new Check - https://github.com/checkstyle/contribution/tree/master/checkstyle-tester

Contributors are welcome to submit PRs for new Checks that are approved (got label "approved" in github issue).