Scripting an svn:externals change
BRM
have been working through the recovery process.
Part of that is going through and modifying a number of projects for the change
in the URL - namely changing from a non-standard port for apache to a standard
port.
We have started using svn:externals in some projects, but did not use the
relative path values - instead, specifying the whole URL; so I need to update
the URLs in the svn:externals.
I know I can get the values using 'svn pget svn:externals'; but the SVN redbook
recommends against using pset/propset to set the values.
So, how do I set the values from a script? "svn help pset" seems to show a "-F"
parameter for reading the values from a file. This seems like it would take care
of the concern mentioned issue - that the externals are multi-line.
If that is the case, then shouldn't the note say to use that parameter with pset
or use pedit instead of flatly saying use pedit?
Or is there another reason why I shouldn't use pset?
TIA,
Ben
http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html
Andy Levy
> I recently had my subversion server die - the box, not subversion itself - and
> have been working through the recovery process.
> Part of that is going through and modifying a number of projects for the change
> in the URL - namely changing from a non-standard port for apache to a standard
> port.
> We have started using svn:externals in some projects, but did not use the
> relative path values - instead, specifying the whole URL; so I need to update
> the URLs in the svn:externals.
>
> I know I can get the values using 'svn pget svn:externals'; but the SVN redbook
> recommends against using pset/propset to set the values.
Read that more carefully - it's a special case. svn propset is only
discouraged if you have a *multiline* value. That is, you have one
directory which needs to reference multiple externals. If you're only
referencing a single external for each directory, svn propset will be
fine.
> So, how do I set the values from a script? "svn help pset" seems to show a "-F"
> parameter for reading the values from a file. This seems like it would take care
> of the concern mentioned issue - that the externals are multi-line.
> If that is the case, then shouldn't the note say to use that parameter with pset
> or use pedit instead of flatly saying use pedit?
Most of the time, using svn propedit is enough because people don't
often make changes to their whole repository via a script as you're
doing here.
> Or is there another reason why I shouldn't use pset?
If you only have one external per directory, I don't see why you can't use pset.
Ryan Schmidt
If you can afford to have the repository offline, make a dumpfile, twiddle it, load it into a new repository, and have everybody check out new working copies, you may want to consider doing that. The twiddling step would be done with the svndumptool script. The advantage would be that you could correct the externals definitions in all past revisions of the repository too, not just the HEAD, so that anybody needing to check out a past revision would still have working externals in the repository's new location. And do use relative externals this time so you won't have to do this kind of surgery ever again, noting that the relative externals syntax is only compatible with Subversion 1.5 and higher.
BRM
That's a good thought. I might try doing that tonight. I'm aware of they syntax
compat issue, and have no problem
saying that we have to use 1.5 or higher; it's just a matter of getting the
update through.
Ben
BRM
Okay - well, I decided to try this route, but was advised to try it on a test
repository first.
Good thing - as there is a bit of an issue, and one I need to avoid if I can...
The issue is that an existing working copy's externals are not being updated
when 'svn update' runs,
even though the externals have changed on the server through the update.
Here's my test:
1. Create a file-based repository:
$ cd /
$ mkdir svnTest && cd svnTest
$ svnadmin create Example1
$ mkdir -p ~/testsvn && cd ~/testsvn
$ export SVNROOT=/svnTest
$ svn co ${SVNROOT}/Example1 example1.wc
$ cd example1.wc
$ mkdir sub1 sub2
$ svn add sub1 sub2
$ svn commit -m "Adding some structure for test"
$ touch sub1/file1
$ svn add sub1/file1
$ svn commit -m "Adding a file for test"
$ svn pedit svn:externals sub2
set the svn:externals to:
sub2a file:///svnTest/Example1
save and quit
$ svn update
$ svn commit -m "Adding an external for test"
2. Make a backup of the working copy
$ cd ~/testsvn
$ tar -jcvvf example1.wc.tar.bz2 example1.wc
3. Dump the repository
$ cd /svnTest
$ svnadmin dump Example1 > Example1.dump
4. Switch the external reference from Example1 to Example2
$ cat Example1.dump | sed
's/file:\/\/\/svnTest\/Example1/file:\/\/\/svnTest\/Example2/' >
Example1-edited.dump
5. Load the edited dump into a new repository (Example2)
$ cd /svnTest
$ svnadmin create Example2
$ svnadmin load Example2 < Example1-edited.dump
6. Rename the Example1 repository so it looks like it was removed
$ cd /svnTest
$ mv Example1 Example1.removed
7. Relocate the Example1 working copy, then update it
$ cd ~/testsvn
$ cd example1.wc
$ svn switch --relocate file:///svnTest/Example1 file:///svnTest/Example2
$ svn update
At this point 'svn update' fails outright complaining that it cannot find
file:///svnTest/Example1 (since it was removed).
So I can live with sub2a not getting updated without an svn switch --relocate
occurring on it - integrity and all; however, I would have expected that
svn update the svn:externals on sub2. I could even live with having to rename
an existing external (sub2a -> sub2a.orig) and updating (it's kinda of
acceptable, but not ideal).
At this point, I would expect that the svn:externals property for 'sub2' match
what is in the Example2 repository.
However, an 'svn pget svn:externals sub2' reveals that it did not get updated.
I even tried 'svn update --force'.
I kind of expected that 'svn update --ignore-externals' would not update it
either - which is what I found when I tried it.
Now I tried this with a 1.5.1 and 1.6.6 versions of the svn toolchain; though if
it is working in something newer okay. However, I'm kind of stuck with those two
versions - ubuntu hardy-backports provides 1.5.1 and lucid uses 1.6.6.
Now why is this a "big issue" for me? B/c I and a number of my team members
have a lot of projects with svn:externals, and we need to update our working
copies.
I need to make this as seamless as possible. Scripting the svn switch
--relocate command is easy; but I shouldn't have to also edit the svn:externals
on each project
since that was already changed in the repository.
Now I may be able to alleviate my issue some through temporarily restoring an
alternate port that gets redirected to the server via proxy[1] (the old server
ran on example.com:82 and the new one is on example.com); but forcing people to
have to completely delete their working copies is not an easy option -
especially since there may be some changes in the working copies that have not
yet made it to the server.
Or am I missing some? At this point, I am not going to touch my primary
repository until I know the right solution - but that also means we're stuck.
TIA,
Ben
Ryan Schmidt
> ----- Original Message ----
>
>> From: BRM <bm_wi...@yahoo.com>
>> To: Ryan Schmidt <subversi...@ryandesign.com>
>> ----- Original Message ----
>>> On Sep 2, 2010, at 11:27, BRM wrote:
>>> If you can afford to have the repository offline, make a dumpfile, twiddle it, load it into a new repository, and have everybody check out new working copies, you
>>> may want to consider doing that.
> Okay - well, I decided to try this route, but was advised to try it on a test
Do not use sed to edit a dumpfile. Use a tool designed for the task, like svndumptool.
http://svn.borg.ch/svndumptool/
> 5. Load the edited dump into a new repository (Example2)
>
> $ cd /svnTest
> $ svnadmin create Example2
> $ svnadmin load Example2 < Example1-edited.dump
Because this repository is now different than it was before (the history has been edited), you must give the new repository a new UUID.
svnadmin load --ignore-uuid Example2 < Example1-edited.dump
Since the UUID has changed, everyone must check out new working copies; it is not possible to update or switch existing working copies to this new changed repository.
BRM
> From: Ryan Schmidt <subversi...@ryandesign.com>
> > $ cat Example1.dump | sed
> > 's/file:\/\/\/svnTest\/Example1/file:\/\/\/svnTest\/Example2/' >
> > Example1-edited.dump
> Do not use sed to edit a dumpfile. Use a tool designed for the task, like
>svndumptool.
> http://svn.borg.ch/svndumptool/
Why? B/c it might change data in files too? That's actually something I want
done.
My prime repository has some scripts in it that store the repository information
for auto-checkouts as part of build environments, and I want those changed too -
not just the svn:externals.
Or would sed 'break' the dump file format? svndumptool does look useful to
verify that sed did exactly what I wanted - standard diff won't work well on a
2.1 GB dumpfile.
> > 5. Load the edited dump into a new repository (Example2)
> >
> > $ cd /svnTest
> > $ svnadmin create Example2
> > $ svnadmin load Example2 < Example1-edited.dump
> Because this repository is now different than it was before (the history has
>been edited), you must give the new repository a new UUID.
> svnadmin load --ignore-uuid Example2 < Example1-edited.dump
I wouldn't say 'must' but 'should' - as in 'highly recommended'.
> Since the UUID has changed, everyone must check out new working copies; it is
>not possible to update or switch existing working copies to this new changed
>repository.
Or I could just ignore the history, and just go back to my original question of
changing the values.
As I said, I need to make this as seamless as possible for everyone - preferably
doing an 'svn relocate'/'svn update' on the working copies.
Question: If I checked them out again would the client realize that I have a
series of files locked and retain the locks on the new working copy? (That would
be a PITA to track down if it didn't.)
TIA,
Ben
Andy Levy
> ----- Original Message ----
>
>> From: Ryan Schmidt <subversi...@ryandesign.com>
>> > $ cat Example1.dump | sed
>> > 's/file:\/\/\/svnTest\/Example1/file:\/\/\/svnTest\/Example2/' >
>> > Example1-edited.dump
>> Do not use sed to edit a dumpfile. Use a tool designed for the task, like
>>svndumptool.
>> http://svn.borg.ch/svndumptool/
>
> Why? B/c it might change data in files too? That's actually something I want
> done.
> My prime repository has some scripts in it that store the repository information
> for auto-checkouts as part of build environments, and I want those changed too -
> not just the svn:externals.
>
> Or would sed 'break' the dump file format? svndumptool does look useful to
> verify that sed did exactly what I wanted - standard diff won't work well on a
> 2.1 GB dumpfile.
If you modify the contents as well, it'll break the dumpfile. The
dumpfile contains an MD5 hash of each file for integrity checking.
Daniel Shahaf
> ----- Original Message ----
>
> > From: Ryan Schmidt <subversi...@ryandesign.com>
> > > $ cat Example1.dump | sed
> > > 's/file:\/\/\/svnTest\/Example1/file:\/\/\/svnTest\/Example2/' >
> > > Example1-edited.dump
> > Do not use sed to edit a dumpfile. Use a tool designed for the task, like
> > svndumptool.
> > http://svn.borg.ch/svndumptool/
>
> Why?
Because dump files have structure --- e.g., they contain checksums (length,
md5, sha1) of their contents.
Ryan Schmidt
>> Do not use sed to edit a dumpfile. Use a tool designed for the task, like
>> svndumptool.
>> http://svn.borg.ch/svndumptool/
>
> Why? B/c it might change data in files too? That's actually something I want
> done.
> My prime repository has some scripts in it that store the repository information
> for auto-checkouts as part of build environments, and I want those changed too -
> not just the svn:externals.
>
> Or would sed 'break' the dump file format? svndumptool does look useful to
> verify that sed did exactly what I wanted - standard diff won't work well on a
> 2.1 GB dumpfile.
As others said, because if you change the content of files, the checksums won't match. Just use svndumptool to change the svn:externals properties, and for any file content changes, make them and commit them from a working copy as normal.
>>> 5. Load the edited dump into a new repository (Example2)
>>>
>>> $ cd /svnTest
>>> $ svnadmin create Example2
>>> $ svnadmin load Example2 < Example1-edited.dump
>> Because this repository is now different than it was before (the history has
>> been edited), you must give the new repository a new UUID.
>> svnadmin load --ignore-uuid Example2 < Example1-edited.dump
>
> I wouldn't say 'must' but 'should' - as in 'highly recommended'.
If the content of the new repository is different from the content of the old repository (and it is, if you changed the properties), then you must give them different UUIDs. That is the sole purpose of the UUID -- to identify when repositories differ.
>> Since the UUID has changed, everyone must check out new working copies; it is
>> not possible to update or switch existing working copies to this new changed
>> repository.
>
> Or I could just ignore the history, and just go back to my original question of
> changing the values.
>
> As I said, I need to make this as seamless as possible for everyone - preferably
> doing an 'svn relocate'/'svn update' on the working copies.
And as I said, dump and load and changing UUID requires checking out new working copies. You cannot relocate or update existing working copies. If that's too restrictive, then yes, you'll have to skip the dump and load, just change the externals in HEAD, and accept that you can't check out a past revision and have the externals work.
> Question: If I checked them out again would the client realize that I have a
> series of files locked and retain the locks on the new working copy? (That would
> be a PITA to track down if it didn't.)
I'm not sure exactly what you meant (checked out new working copies? from the new repository?), but note that the dump format does not include information about hook scripts, config files or locks. If you dump and load, the new repository will have the default hook script templates (so, no hook scripts), the default configuration, and nothing locked. But I think you can manually bring the relevant files from the old repository to the new one to preserve this data.
Daniel Shahaf
> the dump format does not include information
> about hook scripts, config files or locks. If you dump and load, the new
> repository will have the default hook script templates (so, no hook scripts),
> the default configuration, and nothing locked. But I think you can manually
> bring the relevant files from the old repository to the new one to preserve
> this data.
For hooks and config files, yes.
Ryan Schmidt
I thought I heard others say you could copy the db/locks directory to preserve locks, too. Will that not work?
Daniel Shahaf
The API does not promise that copying db/locks will work.
And if someone tries that anyway? In general, I think the answer to
your question is the same as the answer to the general "Can I rsync my
repository directory?" question:
* under FSFS, the API doesn't allow it but it might work under certain
conditions.
* under BDB, db/locks is a BDB database,
and you probably know better than me when it is/isn't safe to just
copy those :-).
BRM
> From: Ryan Schmidt <subversi...@ryandesign.com>
> To: BRM <bm_wi...@yahoo.com>
> >> Since the UUID has changed, everyone must check out new working copies; it
>is
>
> >> not possible to update or switch existing working copies to this new
>changed
>
> >> repository.
> >
> > Or I could just ignore the history, and just go back to my original question
>of
>
> > changing the values.
> >
> > As I said, I need to make this as seamless as possible for everyone -
>preferably
>
> > doing an 'svn relocate'/'svn update' on the working copies.
>
> And as I said, dump and load and changing UUID requires checking out new
>working copies.
> You cannot relocate or update existing working copies. If that's too
>restrictive, then yes,
> you'll have to skip the dump and load, just change the externals in HEAD, and
>accept
> that you can't check out a past revision and have the externals work.
Having the history broken is not much of a problem that way. It just would have
been nice to have it work well too.
Though, couldn't I setup a script to modify the svn:externals via the pset
--revprop option?
Any how...I've got a lot of work ahead regardless.
> > Question: If I checked them out again would the client realize that I have a
>
> > series of files locked and retain the locks on the new working copy? (That
>would
>
> > be a PITA to track down if it didn't.)
>
> I'm not sure exactly what you meant (checked out new working copies?
> from the new repository?), but note that the dump format does not include
> information about hook scripts, config files or locks. If you dump and load,
> the new repository will have the default hook script templates (so, no hook
> scripts), the default configuration, and nothing locked. But I think you can
> manually bring the relevant files from the old repository to the new one to
>preserve this data.
>
Okay - the basic situation is that I have a branch in SVN that was under
migration. It was a port of our CVS into SVN and needed to be split up into
separate projects which requires a lot of work to do.
So I had a working copy where I locked files that I had moved to the new
structure to help keep from editing those files in the old structure, and it'll
be a PITA to figure out which files were moved and locked between the two.
At least I still have the working copy. If they don't transfer, okay - I just
have to go through the old working copy and relock them in a new working copy.
(I had to use locks as I couldn't just remove the files due to breaking other
projects.)
Thanks - I think I have my plan of action now - it'll just take a while to
implement. If I can get apache to do the proxy thing (http://silmor.de/49) then
I can at least get back to a working state where we can move things about with
our existing working copies - make sure all changes are in, and are at a good
place to re-download them.
Sadly this all occurred due to a server crash, so it was rather unplanned though
it didn't surprise some of us.
Thanks all!
Ben
Ryan Schmidt
>> You cannot relocate or update existing working copies. If that's too
>> restrictive, then yes,
>> you'll have to skip the dump and load, just change the externals in HEAD, and
>> accept
>> that you can't check out a past revision and have the externals work.
>
> Having the history broken is not much of a problem that way. It just would have
> been nice to have it work well too.
> Though, couldn't I setup a script to modify the svn:externals via the pset
> --revprop option?
> Any how...I've got a lot of work ahead regardless.
No; svn:externals is a versioned property, not a revision property.
BRM
> From: Ryan Schmidt <subversi...@ryandesign.com>
> To: BRM <bm_wi...@yahoo.com>
Thanks.
I did get the the Apache proxy stuff working. Still got a lot to do, but at
least we can function now.
Thanks!
Ben