Skip revisions

[Svend Hansen]

Hello,

I've had to stop and restart my SubGit import a few times now, when it's turned out there were (accidental) copies of the root folder to branches, that were then immediately deleted.

This was easily enough to fix by adding "excludeBranches=branch/accidentalBranchName" to the config when restarting the import.

However, I have now come across a more difficult to fix situation. A tag was created from the root rather than trunk. It was then deleted again. But then a new tag, with the same name, was created from trunk. This means that if I use excludeBranches to exclude the incorrect tag, I would also exclude the correctly constructed tag.

What I would like to do instead would be to just tell SubGit to exclude or skip the two revisions where the incorrect tag was created and deleted, respectively.

This StackOverflow question and answer suggests that this is not possible:

https://stackoverflow.com/questions/18918215/skip-revisions-when-installing-subgit-repository

Is that the case? Is the suggested method of interrupting the import, setting the branches-maxRev and tags-maxRe valid and safe? Seems a bit hacky? Hasn't it already included some of the revision, or will it not include anything from the revision until it has completed it (I don't know how it could possibly "partially include" a revision, but then I don't know how it works, which is why I'm asking here :))?

It would be much nicer to be able to exclude revisions, and I wouldn't mind having to restart it, even though it might delay the import a day or two, rather than do something that might leave it in a dirty state.

[Svend Hansen]

I decided to try out the solution suggested in the SO post, as I'd otherwise have to restart anyway, and after setting the tags-maxRev = 128254 and branches-maxRev = 128254 in the <git-project>/svn/.metadata file, which is the revision it was stuck on, I got this error when restarting the import:

SubGit version 3.2.5 ('Bobique') build #3694
Continuing after an interrupted import.
Translating Subversion revisions to Git commits...
IMPORT FAILED
error: There're new Git commits, not present in SVN repository, can't import.
error: If you want continuous synchronization between SVN and Git repositories, use "subgit install" command instead.

I'm not sure I understand why this would happen because of making that change?

Now it also happens when I reset them to the values they were after I interrupted the import.

[TMate Software Support]

Hello Svend,

Indeed, the way described in SO response is hacky but valid if the revisions you skip this way have no other changes than creation and immediate removal of that wrong tag.

This is not the only way to solve the problem, another way (not hacky but much more difficult) is to remove the relevant revisions using svnadmin dump/load approach:

1. Create 2 dumps: for changes before and after these 2 revisions (tag creation and removal).

2. Create new SVN repository.

3. Apply the first dump to it.

4. Manually create (with "svn commit" or "svn mkdir SVN_URL/of/tag") that wrong tag but this time not as a copy of root but just as an empty directory.

5. Manually remove that tag like in the previous step.

6. Apply the second dump including changes after those 2 revisions.

7. Use this new repaired SVN repository instead of original one.

I don't describe the steps in detail by intention as I think this approach is too complicated and time consuming than SO recommendation. But if you don't think so, contact us again, and I'll provide the commands for this approach.

>Is that the case?

You wrote that r128254 is that wrong revision where the tag is added as a copy of the root, right? Is it true that at r128255 that tag got removed? If yes, the SO post should help you.

The idea is that ".metadata" file contains information about the latest SVN revision translated to Git. SubGit always skips translation of revisions which are less or equal to it. If 2 subsequent revisions "cancel" each other (like tag addition and removal) and the changes are not used in the later revisions (i.e. no other later revision deals with that version of tag created from repository root), they can be skipped.

>Is the suggested method of interrupting the import, setting the branches-maxRev and tags-maxRev valid and safe?

Yes, it's valid and safe, though it's not documented.

> Seems a bit hacky?

Also true, it's a hacky way.

>Hasn't it already included some of the revision, or will it not include anything from the revision until it has completed it (I don't know how it could possibly "partially include" a revision, but then I don't know how it works, which is why I'm asking here :))?

Sorry, I don't understand the question.

> I decided to try out the solution suggested in the SO post, as I'd

> otherwise have to restart anyway, and after setting the tags-maxRev = 128254

> and branches-maxRev = 128254 in the <git-project>/svn/.metadata file, which

> is the revision it was stuck on, I got this error when restarting the

> import:

May I ask you to describe r128254 and r128255 changes? To use the work-around they should they should be the tag addition and the tag removal respectively. And if this is so, you should skip both of them and set

  tags-maxRev = 128255

  branches-maxRev = 128255

Otherwise you will translate a revision (r128255) that deletes a tag that you don't have in Git because you skipped its creation at r128254.

>

> SubGit version 3.2.5 ('Bobique') build #3694

>

> > Continuing after an interrupted import.

> > Translating Subversion revisions to Git commits...

> > IMPORT FAILED

> > error: There're new Git commits, not present in SVN repository, can't

> > import.

> > error: If you want continuous synchronization between SVN and Git

> > repositories, use "subgit install" command instead.

> I'm not sure I understand why this would happen because of making that

> change?

Probably this happens because you skipped revisions that don't cancel out (e.g. tag creation and subsequent tag removal). If you really don't have Git commits that are not translated from SVN, you may run

  subgit install --rebuild-from-revision REV path/to/git/repository

where REV is some older revision, e.g. 128200. Then the translation process will roll back to that revision and restart from it. Then again press Ctrl+C and skip revisions that mutually cancel out.

Note, that "subgit import" = "subgit install" + "subgit uninstall", so if you don't need continuous synchronization, you'll need to run "subgit uninstall path/to/git/repository" upon translation completion.

Dmitry Pavlenko,

TMate Software,

http://tmatesoft.com/ git-svn import & mirror

--
You received this message because you are subscribed to the Google Groups "SubGit Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to subgit-users+unsubscribe@googlegroups.com.
To post to this group, send email to subgit...@googlegroups.com.
Visit this group at https://groups.google.com/group/subgit-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/subgit-users/08cb2322-fa23-4b34-8606-6af85cab1d5d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Svend Hansen]

Hi Dmitry,

Thank you for your reply.

Based on what you say I think I can see what the problem is. While the tag was added in r128255, it was not removed again until several days later when the mistake was discovered, so the two revisions are not consecutive.

What I have decided to do (for now), is to simply use excludeBranches and accept that the tag isn't included in the git repository's history. It's a tag of a release over a year ago, so it's very unlikely to be needed, and I guess it could still be recreated in git if the correct commit corresponding to the revision it was created for can be identified, and otherwise it could always be pulled out of the old SVN repo, which would probably still stick around (if it was ever to be required, which I highly doubt).

I don't know if it would be possible to implement a "skipRevisions" config setting for SubGit, and what the implecations of using it would be. I guess it would require those revisions to be "atomic" in the sense that no other revisions change anything that they change (like in the case the creation and deletion of the tag)... but even then I don't know enough about how SVN works (or how SubGit works) to know if it would still not work in this case...

Thank you for your replies and help!

Cheers,

   Svend.

[TMate Software Support]

Hello Svend,

For that case there's still a work-around based on svnadmin dump/load but it's even more complicated and more time-consuming than the one I described in the previous letter, so I think the solution you've chosen is the best one in your situation.

What about skipping revisions, you're right, it's important that these revisions should be isolated from other changes. I don't think we will implement such a config option because the number of users that could benefit from it would be too low. Just imagine:
 - the number of users that have some branches/tags created by mistake and then deleted is not big;
 - the number of users with those problem and that can't solve them with 'excludeBranches' is really low;
 - the number of users with those problem and that can't solve them with 'excludeBranches', and having these changes "atomic"/isolated is close to zero.

On the other hand the explanation of this "atomic" concept and why I can't skip REV1 without skipping REV2 or why arbitrary REV can't be skipped in general would complicate documentation for all users.

Let me know if you need further assistance and have a nice day!

Dmitry Pavlenko,

TMate Software,

http://tmatesoft.com/ git-svn import & mirror

To view this discussion on the web visit https://groups.google.com/d/msgid/subgit-users/dc624813-1704-4203-8a94-f539c284b633%40googlegroups.com.

[Svend Hansen]

Hi again, Dmitry!

I went with leaving out the tag, and everyone seemed happy with that.

Now I'm getting very close to completing the import, but have reached another tag created from root. However, in this case no one has noticed, and the tag is still there (was created in July and no one noticed that it contains everything). I thought, since it was never deleted and never modified (I've checked its log, and the latest change is its creation), I would be able to use the "skipping" method in this case. So I interrupted the import again, updated the branches-maxRev and tags-maxRev to the revision of the creation of the tag, and restarted the import. However, I'm still getting the error about "new Git commits, not present in SVN repository"... Any idea why this might be?

Cheers,

   Svend.

On Monday, 11 September 2017 15:27:12 UTC+2, TMate Software Support wrote:

Hello Svend,

For that case there's still a work-around based on svnadmin dump/load but it's even more complicated and more time-consuming than the one I described in the previous letter, so I think the solution you've chosen is the best one in your situation.

What about skipping revisions, you're right, it's important that these revisions should be isolated from other changes. I don't think we will implement such a config option because the number of users that could benefit from it would be too low. Just imagine:
 - the number of users that have some branches/tags created by mistake and then deleted is not big;
 - the number of users with those problem and that can't solve them with 'excludeBranches' is really low;
 - the number of users with those problem and that can't solve them with 'excludeBranches', and having these changes "atomic"/isolated is close to zero.

On the other hand the explanation of this "atomic" concept and why I can't skip REV1 without skipping REV2 or why arbitrary REV can't be skipped in general would complicate documentation for all users.

Let me know if you need further assistance and have a nice day!

Dmitry Pavlenko,

TMate Software,

http://tmatesoft.com/ git-svn import & mirror

[TMate Software Support]

Hello Svend,
The message "new Git commits, not present in SVN repository" may be misleading, actually it is
printed on any change between SVN and Git versions, i.e. not only in case of new Git commits
compared to SVN but also on absent Git commits/references.

In your case the tag is present in SVN but absent in Git due to the trick you did.

Please, do the following:
1. Among GIT_REPO/refs directory find

refs/tags/thatTag

2. Then inside

GIT_REPO/refs/svn/root/

find file corresponding to your tag (it might be GIT_REPO/refs/svn/root/tags/thatTag ).

3. Inside GIT_REPO/svn/refs directory find

refs/tags/thatTag/.rev_map.<UUID>

file.

And send all three to us (they only contain SHA-1 and revision numbers) or report if any are absent.
I would like to analyze the issue closer.

In either case you know the work-around:
1. Add thatTag to 'excludeBranches'.
2. Run

  subgit install --rebuild-from-revision REV path/to/git/repository

for revision older than thatTag creation.

Dmitry Pavlenko,

TMate Software,

http://tmatesoft.com/ git-svn import & mirror

To view this discussion on the web visit https://groups.google.com/d/msgid/subgit-users/336a5dff-3b47-4485-a8e0-0377bcfef283%40googlegroups.com.

[Svend Hansen]

Hi Dmitry,

1. There are only tags up until the tag before the problem tag.

2. There are only two tags in here, one is the one before, and the other is an earlier tag.

3. This seems to contain all tags before the problem tag as well.

I've already (while moving the failed import folder sideways, so I could still do this investigation and try anything else you might suggest) started it over with another excludeBranches config entry :)

I started it from the beginning again. When I try the install with the rebuilding option, it fails with this error:

error: Subversion and Git repositories are out of sync and can not be synchronized:
error: Subversion repository '<svn-path>' has 1778 unsynced revisions;
error: Git repository '<git-repo-path>' has 1 unsynced ref:
error:   tag '<tag-name>' removed from b9998e5
error:
error: To attempt automated recover from the out of sync state, run the following command:
error:
error:   subgit install --recover <git-repo-path>
error:
error: In case there are conflicts that prevent automated recovery,
error: instructions will be provided on how to resolve conflicts manually.
error: Unexpected error has occurred; please report along with the logs ('C:\Projects\subgit-install-20170913-095130.zip')
error:   to http://issues.tmatesoft.com/, thank you!

It's at 55% now, so I guess maybe tomorrow morning it will have finished :D

I finally had the idea of running svn log in the "tags" folder, and scanning through all the remaining tags, and none of those seem to be borked, at least :)

Cheers,

   Svend.

On Wednesday, 13 September 2017 14:33:42 UTC+2, TMate Software Support wrote:

Hello Svend,

The message "new Git commits, not present in SVN repository" may be misleading, actually it is
printed on any change between SVN and Git versions, i.e. not only in case of new Git commits
compared to SVN but also on absent Git commits/references.

In your case the tag is present in SVN but absent in Git due to the trick you did.

Please, do the following:
1. Among GIT_REPO/refs directory find

refs/tags/thatTag

2. Then inside

GIT_REPO/refs/svn/root/

find file corresponding to your tag (it might be GIT_REPO/refs/svn/root/tags/thatTag ).

3. Inside GIT_REPO/svn/refs directory find

refs/tags/thatTag/.rev_map.<UUID>

file.

And send all three to us (they only contain SHA-1 and revision numbers) or report if any are absent.
I would like to analyze the issue closer.

In either case you know the work-around:
1. Add thatTag to 'excludeBranches'.
2. Run

  subgit install --rebuild-from-revision REV path/to/git/repository

for revision older than thatTag creation.

Dmitry Pavlenko,

TMate Software,

http://tmatesoft.com/ git-svn import & mirror