committing changes to multiple svn externals

[gordo...@gmail.com]

I am currently working on projects which are composed of multiple software units (stored on one SVN repository). Projects are "assembled" by checking-out a combination of these shared software units to the development environment, as svn:externals.

The externals are defined in the 'Sources' folder as so:

(local) (url)
<layer>/software_unit_1 <svn server>/components/software_unit_1/source_code
<layer>/software_unit_2 <svn server>/components/software_unit_2/source_code
... ...
<layer>/software_unit_x <svn server>/components/software_unit_x/source_code

and external revisions are only pegged when tagging.


When I make modifications to an external in my working copy, I can commit the changes back to the server by doing 'SVN commit' at its checkout location. Perfect.

Because of the dependencies between software units, I'd like to be able to commit a logical set of changes to multiple externals in the same commit.
However, when I try do 'SVN commit' at the folder with the external definitions, the external changes listed in the commit dialog window are all greyed-out.

Since they are listed in the commit window, it is clear to me that they are from the same repository and be committed in a single commit.

If my understanding is incorrect I'd be very happy to receive an explanation of why this isn't possible.

It would greatly improve our current workflow as, not only would it be faster and more straightforward, the current process of individually committing each external separately means our CI server cannot build automatically on commits as changes for all dependencies cannot be committed.

Thanks and looking forward to heading from you.

[Nathan Hartman]

One really great feature of Subversion is the ability to make atomic commits that can span multiple parts of a software system, provided that all the parts reside in the same (monolithic) SVN repository. It seems that is exactly what you want.

Since you said that your multiple software units are stored in one (monolithic) SVN repository, here is one possible idea:

In Subversion, you can get a sparse checkout. I don't know how this is done in Tortoise because I use the command line client, but it goes something like this:

svn co --depth=empty svn://path/to/repo/and/directory

And you can use one of: empty, immediates, or infinity for the --depth= argument.

This gives the possibility to checkout the root of your Subversion repository, as --depth=immediates, without actually copying potentially gigabytes (or even more) data to your workstation. I am assuming that the root only contains directories, or at least it doesn't contain a lot of files. With --depth=immediates, you'll get those directories but they will be empty in your working copy. Then, you can progressively change to nested directories and "telescope" the depth using: "svn update --set-depth=" with one of empty, immediates, or infinity.

This functionality is called a "viewspec" in some other version control systems.

There is a Python script somewhere that can automate checking out a viewspec, and you can even version the "spec" files in your repository. See, for example:

https://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-viewspec.py

However what we've done (because we didn't know about the Python script and now we're accustomed to doing it this way) is that we simply wrote a .bat file that checks out a working copy of the root with "svn co --depth=immediates" and then telescopically expands the directories we're interested in with "svn update --set-depth=immediates" and/or "infinity". We have such a batch file for each "viewspec" that we'd want to checkout.

Having a working copy that contains all relevant parts of your software system, you can apply changes to the multiple parts and then commit them in one atomic commit. Be mindful to be in the topmost directory when doing so, as Subversion commits apply to the current directory and all nested directories under it.

Atomic changes across a monolithic repository are a very valid idea. You may find the following article interesting. It is about Google's internal monolithic version control system but the general idea transfers over to Subversion (on a much smaller scale):

https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

At our site, we do use externals, but we do not develop inside the externals directories. We develop in the original directories. In all cases, our externals definitions always specify both the path and peg revision so that future changes to repository layout will not affect checkouts of past revisions.

Hope this is helpful.

[Nathan Hartman]

One more thing: I didn't answer your original question, which is: why can't you do an atomic commit that recurses into externals?

If I understand correctly, this is the reason: svn:externals may fetch from multiple different repositories, even potentially located on different servers in different geographic regions. In your case, you are fetching all externals from the same repository, but I think the original idea behind externals was to construct a working copy of software that has various dependencies that come from different places. Rather than requiring you to use some other tool to do this, I think it was added to Subversion as a convenience feature. It is not feasible to make an "atomic" commit that spans across multiple repositories, because each repository has its own separate existence. Working with dependencies that all come from the same repository, as you are doing (and so are we), is a special case of this more general possibility, and Subversion does not treat this special case any differently than the general case.

This is why we work in the original locations of the dependencies and use the externals only for assembling all the different pieces of a software to be compiled. The externals do serve the very useful purpose of recording exactly which dependencies are needed and which exact revision of each dependency, so that in the future we can checkout a past version and recompile it successfully, without losing the knowledge of exactly which bits were used in that build.

On Tuesday, July 17, 2018 at 10:09:31 AM UTC-4, Gordon Jess wrote:

[Johan Corveleyn]

Op woensdag 18 juli 2018 17:38:05 UTC+2 schreef Nathan Hartman:

One more thing: I didn't answer your original question, which is: why can't you do an atomic commit that recurses into externals?

If I understand correctly, this is the reason: svn:externals may fetch from multiple different repositories, even potentially located on different servers in different geographic regions. In your case, you are fetching all externals from the same repository, ...

I'm confused. As far as I know, performing an atomic commit that includes externals (from the same repository) should work. Even performing a commit that includes externals from other repositories should work from the UI (but it won't be atomic, of course, it will perform different commits per repository, under the hood).

At least, on the commandline, 'svn commit' has an option --include-externals:

  --include-externals      : also operate on externals defined by
                             svn:externals properties

Maybe the OP is using an old version of TortoiseSVN? I'm not sure in which svn version the --include-externals option for 'commit' was introduced.

--

Johan

[Nathan Hartman]

I think I'm the one who's confused.

But I'll quote the svn-book (r5320) -- "Perhaps most disappointingly, the working copies created via the externals definition support are still disconnected from the primary working copy (on whose versioned directories the svn:externals property was actually set). And Subversion still truly operates only on nondisjoint working copies. So, for example, if you want to commit changes that you've made in one or more of those external working copies, you must run svn commit explicitly on those working copies—committing on the primary working copy will not recurse into any external ones."

Of course, that was for svn 1.7. Much has changed since then (such as --include-externals).

You're saying there is logic in the client-side commit machinery that resolves the external, discovers it is coming from the same repo, and performs one atomic commit? If that's the case then I'm really glad to hear that! That can simplify things for us and for the OP considerably.

[gordo...@gmail.com]

Hi Nathan,

Thank you for your suggestion and thorough explanation! I tried the python script today and it worked great.

The only downside of this method is that the folder structure of the working copy must be the same that on the repository. Our SVN repository structure is very much management orientated (organised by customer and variation). SVN externals allowed us to specify a check-out structure more suitable for development (e.g. organised by software architecture)

One more thing... can you please further explain this comment?

>Be mindful to be in the topmost directory when doing so,
>as Subversion commits apply to the current directory and
>all nested directories under it.

What would be the problem with committing from a nested directory, as long as it contains all the parts I want to commit?

Thanks again and best regards,
Gordon

[Johan Corveleyn]

Op vrijdag 20 juli 2018 09:23:44 UTC+2 schreef Gordon Jess:

Hi Nathan,

Thank you for your suggestion and thorough explanation!  I tried the python script today and it worked great.  

The only downside of this method is that the folder structure of the working copy must be the same that on the repository.  Our SVN repository structure is very much management orientated (organised by customer and variation). SVN externals allowed us to specify a check-out structure more suitable for development (e.g. organised by software architecture)

As I said in my reply to Nathan: 'svn commit' nowadays has an option --include-externals, which will include any changes to (same repository) externals into the commit (unless they are "pegged" , i.e. pinned externals).

So, again in reply to Nathan: yes, there is logic in the client-side commit machinery that resolves the external, discovers it is coming from the same repo, and performs one atomic commit. This feature has first appeared in 1.8.0. So Gordon, if you use version 1.8 or later of TortoiseSVN, it should be able to include externals in your commit (unless they are pegged externals).

See this issue: https://issues.apache.org/jira/browse/SVN-1167 (Handle commits which include paths pulled in by svn:externals)

And this mailinglist thread that's referenced by the above issue: https://svn.haxx.se/dev/archive-2011-08/0620.shtml

Ah, I also found a reference to it in the 1.8 release notes: http://subversion.apache.org/docs/release-notes/1.8.html#commit-externals

--

Johan

[Nathan Hartman]

On Friday, July 20, 2018 at 3:23:44 AM UTC-4, Gordon Jess wrote:

Hi Nathan,

Thank you for your suggestion and thorough explanation!  I tried the python script today and it worked great.  

The only downside of this method is that the folder structure of the working copy must be the same that on the repository.  Our SVN repository structure is very much management orientated (organised by customer and variation). SVN externals allowed us to specify a check-out structure more suitable for development (e.g. organised by software architecture)

Yes that is a pretty big downside. On further thought, it also probably affects your ability to build the code, because (1) your build system expects to find things in the locations specified by the externals, and (2) if you are using pegged externals, simply doing a sparse checkout of the whole repository will give you the HEAD revision, whereas the pegged externals might be specifying older versions.

One more thing... can you please further explain this comment?

>Be mindful to be in the topmost directory when doing so,
>as Subversion commits apply to the current directory and
>all nested directories under it.

What would be the problem with committing from a nested directory, as long as it contains all the parts I want to commit?

That's what I meant: In order to get the atomic commit you want, you need to be in a directory that contains all the parts you want to commit.

[Nathan Hartman]

On Friday, July 20, 2018 at 7:16:36 AM UTC-4, Johan Corveleyn wrote:

As I said in my reply to Nathan: 'svn commit' nowadays has an option --include-externals, which will include any changes to (same repository) externals into the commit (unless they are "pegged" , i.e. pinned externals).

So, again in reply to Nathan: yes, there is logic in the client-side commit machinery that resolves the external, discovers it is coming from the same repo, and performs one atomic commit. This feature has first appeared in 1.8.0. So Gordon, if you use version 1.8 or later of TortoiseSVN, it should be able to include externals in your commit (unless they are pegged externals).

Thank you for clarifying this and for the links. This is great news! -- Unless, as you say, they are pegged externals.

This makes me think back to Gordon's original question, which was why the externals were grayed out and couldn't be committed. Is it because they are pegged externals?