[Openbravo-development] Backing out faulty changesets from erp/devel/pi

[Juan Pablo Aroztegi]

I rise a discussion from the IRC channel to this list:

In the process of building a continuous integration system in
erp/devel/pi, there are some decisions to be taken. In this case, what
do we do if a changeset breaks the build or does not pass the tests? I
suggest backing out that changeset. The "hg backout" command applies the
reverse patch of that changeset.

There might be some corner cases where a later changeset may depend on
that faulty changeset. I think that those situations should be handled
manually.

What do you think? Do you see any problems to this?

Juan Pablo

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Openbravo-development mailing list
Openbravo-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openbravo-development

[Ismael Ciordia, Openbravo]

I agree with this approach, I don't see any problem. If a later changeset
depends on the faulty one it would be backed out as well, wouldn't it?

We need to put in place a number of automated tests (Functional testing,
DBSourceManager consistency, stability and upgrade; Modularity rules for
minor versions, etc.) as soon as possible. I suggest that release management
prepares the environment and explain to the rest of the community how to
include new test cases in the nigthly build process. This way Release
management overhead won't be a bottleneck, what do you think?

Ismael

-----Mensaje original-----
De: Juan Pablo Aroztegi [mailto:juanpablo...@openbravo.com]
Enviado el: miercoles, 25 de marzo de 2009 10:44
Para: openbravo-...@lists.sourceforge.net
Asunto: [Openbravo-development] Backing out faulty changesets
fromerp/devel/pi

[Juan Pablo Aroztegi]

> I agree with this approach, I don't see any problem. If a later changeset
> depends on the faulty one it would be backed out as well, wouldn't it?

Yes, you are right.

> We need to put in place a number of automated tests (Functional testing,
> DBSourceManager consistency, stability and upgrade; Modularity rules for
> minor versions, etc.) as soon as possible. I suggest that release management
> prepares the environment and explain to the rest of the community how to
> include new test cases in the nigthly build process. This way Release
> management overhead won't be a bottleneck, what do you think?

This is on the way. Feedback is welcome:
http://wiki.openbravo.com/wiki/ERP/Continuous_Integration

Thanks,

Juan Pablo

[Stefan Huehner]

Juan Pablo Aroztegi wrote:
> I rise a discussion from the IRC channel to this list:
>
> In the process of building a continuous integration system in
> erp/devel/pi, there are some decisions to be taken. In this case, what
> do we do if a changeset breaks the build or does not pass the tests? I
> suggest backing out that changeset. The "hg backout" command applies the
> reverse patch of that changeset.
>
> There might be some corner cases where a later changeset may depend on
> that faulty changeset. I think that those situations should be handled
> manually.
>
> What do you think? Do you see any problems to this?

Main problem i see is that we cannot determine functional dependency
between changesets. Example is when a later changeset needs the
changeset we want to backout for correct function but not for
compilation. It is in general impossible to detect this situation and
backing out the old changeset could go unnoticed and create bigger
problem then not backing out a changeset at all.

Stefan

[Asier Lostalé]

I agree with Stefan. I think it would be easier not to backout but
follow this approach:
1-push to main complete blocks of changesets in pi, this means, push all
the changesets since the last push till the last one that works
2-detect the first changeset not working and reopen the bug which that
one is supposed to fix
3-re-fix that bug with that bug with a new changeset, test it and start
from step 1

[Juan Pablo Aroztegi]

After reading Stefan's and Asier's comments, I agree with that
approach. The risk is lower and it defines the workflow to move
changesets from pi to main.

Isma: what's your opinion on this approach?

Juan Pablo

[Juan Pablo Aroztegi]

We have discussed [1] the details on the process on the IRC channel,
and we have decided that to begin with we are going to keep it as
simple as possible, meaning that:

* Loop: poll for new changesets. Build and test a new changeset.
a) On success, push and merge that changeset into main.
b) On failure, report to issue tracker or reopen referenced issue.

Unless anyone has any objection to this we'll follow this workflow.

Juan Pablo

[1] http://villi.iki.fi/irc/openbravo/index.php?date=2009-03-25

[sathiyan.s...@openbravo.com]

Hi Juan Pablo,
I do agree the process.

However I have observed some scenario in real time which could be
discussed here that could be useful for you.

1) Developer m : He has pushed some solution which is created changeset w.
It's broken by some test cases (for instance import.sample.data). So we
may called it faulty changeset (after poll and build on this changeset).
We have notified by email notification. Assume appropriate developer has
started to look into this issue. Since we have ERP, it doesn't broke all
part of the application. Even some of the frequently used ant target by
developer doesn't touch/test all part of the unit cases (for instance
compile.development or smartbuild).

2) Developer n, Developer o, Developer p has pushed their solution as
changeset x, changeset y and changeset z respectively. chageset x,
changeset y and changeset z are all faulty one, Since those were pushed to
PI after some minutes to changeset w as changeset w is faulty. However
changeset x, changeset y and changeset z those are not necessarily faulty
one, if we consider their push alone. Poll will pick up chageset one by
one, it suggest all changesets are faulty after changeset w. Am I valid?

3) All changeset are faulty after any faulty one, may be act as faulty,
since we have single header concept in Mercurial before push. So our
decision of pushing right changeset to main will be struggled.

4) Assume now we have fixed issue which has created by changeset w.
Solution been pushed which has been created as "changeset aaa". Now we
have to start push from PI to Main. Where to start from? we should not
miss to push changeset x, y and z (whatever has come after chageset w).

I may assume following things that may be consider as solution(from
discipline point of view):

1) All developers should do testing (what our continuous build does -
should be exactly same) before push to repository.

2) If we got any notification for faulty one from build system, we should
not push till the re-fix happened.

waiting queue will be lengthy one, if we get settled fix of faulty changeset.

If the above points are valid(it looks like big story), then take into
consideration.

Regards
Sathiyan.

------------------------------------------------------------------------------

[Juan Pablo Aroztegi]

Hi Sathiyan,

> 2) Developer n, Developer o, Developer p has pushed their solution as
> changeset x, changeset y and changeset z respectively. chageset x,
> changeset y and changeset z are all faulty one, Since those were pushed to
> PI after some minutes to changeset w as changeset w is faulty. However
> changeset x, changeset y and changeset z those are not necessarily faulty
> one, if we consider their push alone. Poll will pick up chageset one by
> one, it suggest all changesets are faulty after changeset w. Am I valid?

This is right. It is tested as a whole, so it will keep trying and
testing till it finds a changeset that passes all the tests.

> 3) All changeset are faulty after any faulty one, may be act as faulty,
> since we have single header concept in Mercurial before push. So our
> decision of pushing right changeset to main will be struggled.

We push the changesets indidually. But we won't push a changeset without
including thre previous ones. So if w is faulty and the later x, y, z
are good changesets, they won't be pushed till a new aaa changeset comes
that solves the problem of x and passes all the tests. When that
happens w,x,y,z and aaa will be pushed to main.

> 4) Assume now we have fixed issue which has created by changeset w.
> Solution been pushed which has been created as "changeset aaa". Now we
> have to start push from PI to Main. Where to start from? we should not
> miss to push changeset x, y and z (whatever has come after chageset w).

Right, as described in the last paragraph.

> 1) All developers should do testing (what our continuous build does -
> should be exactly same) before push to repository.

I completely agree.

> 2) If we got any notification for faulty one from build system, we should
> not push till the re-fix happened.
> waiting queue will be lengthy one, if we get settled fix of faulty changeset.

Yes, we'll wait no matter how long the list is.

Thank your for the comments.

Juan Pablo