OpenXml breaking changes...

[Jim Hargrave]

Going forward, we will need to be more careful about breaking merge when we make changes to filters. The XLIFF difference tests are our main method of detecting these breaking changes. We should avoid these if at all possible. I understand in some cases we have to balance breaking changes with bug fixes or new features , but In the case of the openXML filter we seem to break merge more often than not. This is causing a lot of headaches in production. Because in order to get bug fixes, we must use a snapshot. 

I think we need to be very strict about our release and how we handle dot versions (1.48.2, 1.48.3 etc..). I know we don't do this now, but I think going forward it's going to be important.

cheers,

Jim

[Denis A. Konovalyenko]

Hi Jim,

Thank you for initiating this conversation!

Indeed, incompatible changes have been introduced to the OOXML filter since last release. I agree that merging may require the same filter version as when the extraction was made, which may not always be possible with the latest snapshot version. I also support your idea of releasing dot versions more frequently. Do you think making versions semantic would be even more reasonable? Here is how it can look like.

Semantic versioning principles are used:
1. MAJOR version when incompatible changes are made
2. MINOR version when new functionality in a backward compatible manner
   is added
3. PATCH version when backward compatible bug fixes are made

Additional SNAPSHOT label for pre-release is available as an extension to the MAJOR.MINOR.PATCH format.

One more time, if and only if the MAJOR parts are different,
then versions are incompatible. For instance:

- 2.0.0-SNAPSHOT
- 2.0.0
  These can be linked to the 2nd main branch - Okapi versions with 2.+.+ values are compatible.

And now more about branching.

- main/${MAJOR} should be branched off of the current main/${MAJOR} (the new MAJOR version number must be incremented), integrates various feature and bug branches, serves for deployment of releases and snapshots. E.g.:

git checkout main/1
git checkout -b main/2
git push

- feature/${GITLAB-ISSUE-NUMBER}-${SHORT_DESCRIPTION} branched off of the main/${MAJOR}, incorporates a solution for a feature. Before opening a merge request the MINOR version number has to be incremented in the POM files. E.g.:

git checkout main/2
git checkout -b feature/1429-explicit-cells
# here all related changes are introduced
mvn versions:set "-DnewVersion=2.1.0"
git commit -am "#1429: OpenXML Filter: XLSX: extraction of explicitly specified cells supported."
git push

- bug/${GITLAB-ISSUE-NUMBER}-${SHORT_DESCRIPTION} branched off of the main/${MAJOR}, incorporates a fix for a bug. Before opening a merge request the PATCH version number has to be incremented in the POM files. E.g.:

git checkout main/2
git checkout -b bug/1414-empty-string-item
# here all related changes are introduced
mvn versions:set "-DnewVersion=2.0.1"
git commit -am "#1414: OpenXML Filter: XLSX: crash on empty string item fixed."
git push

If incompatible changes are introduced in the scope of any feature or bug branches, then a new main/${MAJOR} branch must be created with the incremented MAJOR version number and a merge request opened against it. Before opening a merge request MAJOR version number has to be incremented in the POM files (MINOR and PATCH parts have to be 0). E.g.:

git checkout main/2
git checkout -b main/3
git push
git checkout -b bug/1366-ids-generation
# here all related changes are introduced
# the $SNAPSHOTS_BRANCH variable in ci/gitlab/gitlab-ci.yml has to be manually adjusted with a new value: main/3
mvn versions:set "-DnewVersion=3.0.0"
git commit -am "#1366: Core: Code IDs generation reconsidered."
git push

This may require keeping in sync another main${MAJOR+1} branch for changes, which could be made by contributors who are not in the Okapi Framework group on GitLab... Such synchronisation could probably be automated in a CI script after successful deployment.

I believe that adhering to these principles would provide us with greater flexibility during the development process and clarity regarding which version to use in production. Of course, I would be interested to hear everyone's opinion on this.

With best regards,

Denis

--
You received this message because you are subscribed to the Google Groups "okapi-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to okapi-devel...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/okapi-devel/36913804-8f80-4335-8bf6-7e1b078e2141%40gmail.com.

[Jim Hargrave]

Yes, we have discussed semantic versioning before and I do believe this is part of the solution. As far as the branch suggestions below i think this is a good idea. It will help us better track changes and their implications. 

The problem I see here is that since some filters change often we don't want to imply with the version stamp that other filters will introduce backward changes. I think what we really need, unfortunately, is a way to version each module independently. That way we know for sure what filters can be used without issues. So perhaps after the holidays we can have a copy meeting to discuss these approaches.

I also think we need to discuss having different build and release cycles for the applications versus the modules. It's obvious that getting a full release done is difficult updating a module or multiple modules definitely doable more frequently.

I wish we had a dedicated release engineer - i will continue to push on our side to see if we can get more volunteer resources. The only other option I see is to have an okapi/home/jimh/Downloads/LenovoBiosUpdateTool/Rfs/Fw/IdeaPad/ fork which is what most companies do.

jim

To view this discussion visit https://groups.google.com/d/msgid/okapi-devel/3c28190e-492e-3b1a-7ac8-b17559f5d49c%40gmail.com.

[Denis A. Konovalyenko]

Jim,

Thank you for developing this deeper.

It would be nice to have an aligned vision on how to approach the version changes of Okapi components independently. I think we should bump up version of every dependant component as well. And this would require us to know them all. For instance, if we increment the OpenXML filter version from 1.48.0 to 2.0.0, then it would be reasonable to adjust versions of everything that include it as a dependency. We can even get versions showing all versions of sub-components. E.g.: 1.48.0+...filter-idml-1.48.0-filter-openxml-2.0.0... With all that said I have a gut feeling that it would bring a sophisticated branching and CI... :)

If by saying "modules", you mean git modules which implies splitting Okapi into a bunch of projects, then this would simplify the CI to some extent. But what about usage and maintenance? I think this would definitely complicate them... Maybe this is another item for discussion.

I do support frequent releases. And if I am reading our CI script right, we just need to create a tag to release... Is there anything else you are thinking about when mentioning a release engineer?

Best regards,

Denis

[Jim Hargrave]

By modules, I currently mean Maven modules. I think we currently have those organized pretty well, Though I do wonder about moving to Java 9 based modules at some point. Some have been opposed to versioning each module independently as it is More difficult and complex. But there are many advantages to treating.each filter or library almost like an independent product. 

Keep in mind I'm currently ignoring any of the true applications. I'm only talking about our Maven artifacts. If we had a dedicated person that could focus on these tasks it would be wonderful. But I'm not holding my breath.

Jim