IMPORTANT - Ansible Repository Merge

[James Cammarata]

Hi all, just wanted to announce/remind people about the upcoming repo merge.

The Plan:

-------------------------------------------------

Next week, we'll be re-merging the module repositories back into ansible/ansible on GitHub. The two module repos will be essentially locked, though they will be kept in place for the existing 2.1 and 2.2 dependencies. Once 2.2 moves out of official support (in about a year), we'll completely remove those repositories. Until then, any issues/PRs opened there will be auto-closed with a note to open it on ansible/ansible.

Why Are We Doing This (Again...)?

-------------------------------------------------

For those who've been using Ansible long enough, you know that originally we started with a single repository. The original intention of the core vs. extras split was that core would be better supported/tested/etc. Extras would have been a bit more of a "wild-west" for modules, to allow new modules to make it into the distribution more quickly. Unfortunately this never really worked out, as well as the following:

1. Many modules in the core repo were also essentially "grand-fathered" in, despite not having a good set of tests or dedicated maintainers from the community.

2. The time in queue for modules to be merged into extras was not really any different from the time to merge modules into core.

3. The split introduced a few other problems for contributors such as having to submit multiple related PRs for modules with tests, or for those which rely on action plugins. 

4. git submodules are notoriously complicated, even for contributors with decent git experience. The constant need to update git submodule pointers for devel and each stable branch can lead to testing surprises and really buys us nothing in terms of flexibility.

5. Users can already be confused about where to open issues, especially when the problem appears to be with a module but is actually an action plugin (ie. template) or something more fundamental like includes. Having everything back in one repo makes it easier to link issues, and you're always sure to open a bug report in the right place.

Issues and PRs:

-------------------------------------------------

We're working on creating a tool which will allow users to migrate their issues from ansible-modules-core and ansible-modules-extras, details of that will be available next week as well. PRs will be a bit more tricky, though we are working on a script/process to make it easier for users to move their PRs from the existing repos into the ansible/ansible code base post-merge, however in most cases PRs will need to be resubmitted or left in the respective module repository until we can revisit them ourselves.

Metadata - Support/Ownership and Module Status:

----------------------------------------------------------------------

As part of this move, we will be introducing module metadata, which will contain a couple of pieces of information regarding modules:

1. Support Status: This field indicates who supports the module, whether it's the core team, the community, the person who wrote it, or if it is an abandoned module which is not receiving regular updates. The Ansible team has gone through the list of modules and we have marked about 100 of them as "Core Supported", meaning a member of the Ansible core team should be actively fixing bugs on those modules. The vast majority of the rest will be community supported. This is not really a change from the status quo, this just makes it clearer.

2. Module Status: This field indicates how well supported that module may be. This generally applies to the maturity of  the module code however, not necessarily its bug status.

The documentation pages for modules will be updated to reflect the above information as well, so that users can evaluate the status of a module before committing to using it in playbooks and roles.

Summary:

-------------------------------------------------

For the vast majority of Ansible end-users, we expect this change to have little impact. The impact on developers will definitely be higher, though we're really trying to make it much smoother than when we made the original split.

If you have any questions or concerns regarding this, please be sure to let us know ASAP.

Thanks!

James Cammarata

Ansible Lead/Sr. Principal Software Engineer

Ansible by Red Hat

twitter: @thejimic, github: jimi-c

[Toshio Kuratomi]

Greetings all, just writing to let you all know the present status of repomerge and a problem that we have encountered that you're likely to run into later as well.

== Executive summary ==

* We probably have to rewrite git history to fix problems caused by the repomerge script. This will break existing checkouts. 

* Contributors: Please hold off on moving PRs.
  The steps we have to take to correct the repository may mean that the PRs will need to be migrated again (or they may just need to be rebased.  We'll test and let you know).  We've disabled  prmover.pythonanywhere.com for now so no more prs are migrated.

* git log and git bisect are currently broken in the ansible/ansible devel branch.

* Checkouts since repomerge (Yesterday, December 6, 2016 UTC) may need to use some extra git flags to pull afterwards.  We'll test and let you know once we've fixed the history.

* The Ansible Core Team is working through the fixed merging strategies to figure out which of them is needed to get things working..

== Detail ==

Yesterday (Tuesday, December 6, 2016) we ran a script which merged the repositories, we performed some minimal checking, and then pushed the branch to the devel branch on github.

Overnight someone commented that the history (via git log) didn't appear to work quite right (the history of commits existed as shown by git blame but git log --follow FILENAME was unable to find most of the commits).

Further testing showed that git bisect was also having difficulties with the migrated repository: git bisect seemed to get stuck on some commits instead of jumping to a new commit after tagging something as good or bad.  It also jumped far back into history instead of another commit within the initial good/bad range.

Today (Wednesday, December 7, 2016) we've diagnosed the following problems with the migrated repository:

* The log problems are caused because the merge script imported the files at locations relative to the top of the module repos.  At those locations, git was unable to determine that the files should share a common history with the files in their final locations (even when --follow was used).  We prototyped a fix for that which rewrote the file path prior to importing the modules and found that we could fix git log --follow with that.

* The bisect problem was more mysterious for a while.  Eventually we realized that the merge script was interleaving the state of the ansible/ansible tree with the state of the ansible-modules-core and ansible-modules-extras tree.  That means that at any given checkout, we might have all of the ansible/ansible files and none of the modules or all of ansible-modules-core and none of ansible/ansible or the same for ansible-modules-extras.  When trying to git bisect this tree, we then would run across commits that we couldn't test accurately because it didn't have all of the files necessary to test.

The latter problem doesn't appear to be fixable without removing these commits.  So we're probably going to need to reset the tree to before the repomerge apply a fixed method of merging, and then git push --force the rewritten tree to github.  This will invalidate checkouts of the ansible/ansible repository made after the repomerge:

"""

commit 38b621590311643a6c5867249415cddffc2400b9

Author: James Cammarata <ji...@sngx.net>

Date:   Tue Dec 6 10:57:39 2016 -0500

    Das merge

"""

At the moment we're exploring two options as a fixed merge strategy.  Both start by resetting to the commit prior to the repomerge commit.

* The first option will be to simply copy the module files into the new location in the ansible/ansible tree at this point.  One commit will then commit all of hte current modules.  This strategy will mean that trying to find history for the modules will require going back to the ansible-modules-core and ansible-modules-extras repositories from time to time.

* The second option is to try exporting patches from the ansible-modules-core and ansible-modules-extras repositories (either via git format-patch or git cherry-pick) and then reapplying those to a new tree inside of ansible/ansible.  This would preserve history but is more work to implement.  We're currently trying to find out if this is viable or not.

--
You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tkuratomi]

If you're just on ansible-project list and need more background information look at this mailing list thread: https://groups.google.com/forum/#!topic/ansible-devel/mIxqxXRsmCI

= Repomerge on repomerge, check this thread:  Update

Following the outline of a plan by willthames and a lot of  of manual merging by jimi-c, the repo merge has been redone and appears to work much better.  PRMover, and merging to ansible/ansible is now happening again.  The update required removing the original repomerge commits and force pushing the corrected repository to github.  We are sorry that this causes a few inconveniences for users and contributors outlined below.

== How to sync your checkout against the new branch

In most cases, this should be sufficient:

$ git pull --rebase origin devel 

But in case you have any problems:

    **IMPORTANT** Make sure to do step 1.  If you don't make a backup you MAY LOSE ANY WORK that you've done on the devel branch.

    1. Clone your current devel to a new branch, ie. `git checkout devel; git checkout -b devel_backup`

    2. Reset git to a commit prior to the repo merge: `git reset --hard 1a4484be611bdbcaea5f14d1105b7af1ce42552a`

    3. Finally, `git pull --rebase` and you should be on the most up to date devel,

    4. cherry-pick any commits you may have had on devel_backup to the new devel branch.

== Re-migrating PRs

We are extremely sorry but since we had to force push to the repository the PRs which have already been migrated will have to be migrated again.  You will receive a dedicated comment on the PR with a link to instructions if your PR falls into this category.  We apologize profusely.

== Fixes to PRMover

* PRMover now properly checks that the user has a clone of ansible/ansible to migrate the PR to.   This will give them a more informative error message to fix the problem.

* PRMover now does a three-way diff which makes some PRs apply cleanly when they wouldn't apply cleanly with the default merge strategy.

== Why did we have to force push?

The original repomerge script merged the commits from the ansible-modues-core and ansible-modules-extras repositories by using read-tree to push the actual state of the modules repositories into the ansible/ansible repository.  The drawback of this was that trying to git bisect would frequently end up on commits where there was only code from the modules repository.  Without the ansible/ansible code, it was very hard to test that the commit was valid or not.  Rather than subject users trying to diagnose any bugs via bisect to this problem we decided that we needed to rewrite git history to get rid of these commits and do the merge a different way.

Thanks for bearing with us,

Toshio