Questions: svnsync Command

[Audino, Rob]

Hi SVN Folks

 

I am an engineer at a SCADA software company and I'm on the customer-facing side of the company. I have been working to get SVN redundancy set up between two servers on my system, and I use our proprietary software to schedule a Batch script to run every 15 minutes. Built into this batch script is a "kill svnsync" command, and I figured that that would be enough to clear any leftover commands after they ran. However, it seems as though dozens of these svnsync commands are not being killed, and are backing up, lingering for a day and sometimes even longer.

Here is the script:

 

 

I have a few questions:

 

 

1. Why are these svnsync commands not terminating after being ran?

 

2. How long should an svnsync command take to run before terminating?

 

3. How do I set up a batch script that runs those four svnsync commands and then terminates all of them before the next iteration of the batch script?

 

 

Thanks in advance for the help.

 

Regards,

 

Rob Audino

 

 

Rob Audino (he/him)
Project Engineer |  Aspen Technology, Inc.
work: +1-763-404-4042  |  www.aspentech.com

 

[Daniel Sahlberg]

Den tis 4 okt. 2022 kl 19:10 skrev Audino, Rob <Rob.A...@aspentech.com>:

Hi SVN Folks

 

I am an engineer at a SCADA software company and I'm on the customer-facing side of the company. I have been working to get SVN redundancy set up between two servers on my system, and I use our proprietary software to schedule a Batch script to run every 15 minutes. Built into this batch script is a "kill svnsync" command, and I figured that that would be enough to clear any leftover commands after they ran. However, it seems as though dozens of these svnsync commands are not being killed, and are backing up, lingering for a day and sometimes even longer.

Here is the script:

As far as I know, the init command should only be executed once. I have a private mirror of some public repositories and I have scheduled only the svnsync sync command (I did the svnsync init once manually when setting up everything).

From @echo off, I assume this is Windows based. As far as I know, the sleep and kill commands are not standard Windows programs. Are you sure they exist on your system?

Have you tried to execute the batch script manually? Could it be that svnsync is trying to authenticate to the remove system and is just waiting for the user to enter the username/password (which of course won't happen when the scheduler starts the batch file)? I've added --sync-username and --sync-password, but that of course exposes the username and password to anyone with enough privileges on the server. On Windows you can probably save the password to the Windows Credentials store (if you can access the store for the user executing the batch script from the scheduler).

I have a few questions:

 

 

1. Why are these svnsync commands not terminating after being ran?

As far as I know it should start, do its task and then exit.

2. How long should an svnsync command take to run before terminating?

As long as it takes to sync any changes since last run. It is usually quite quick but of course it depends on the size and number of commmits.

  

3. How do I set up a batch script that runs those four svnsync commands and then terminates all of them before the next iteration of the batch script?

It should be up to your scheduling software not to start a second instance of the sync until the first run is completed. I'm not completely sure how svnsync sync would react if you start it a second time while still running.

I hope it works out for you, and please report back if you find a solution!

Kind regards,

Daniel

[Audino, Rob]

Hi Daniel

 

Thanks for your reply. It looks like it was just the “svnsync init” command that was spamming, not the “svnsync sync” command, so I went ahead and removed the “svnsync init” commands (especially considering your statement that “svnsync init” should only be run upon initializing a repository. I plan to check on my server tomorrow to see if the spam has been alleviated.

 

On my Windows system, there are special packages that allow the “sleep” and “kill” commands to work properly. I have ran the script manually before, and as far as I remember, it did not cause commands to linger (only gave an error that the repositories had already been initialized).

 

If the removal of the “svnsync init” commands does not work, then I think I will likely add the username/password parameters that you listed and see if that fixes the issue.

 

Thanks again for your help, and I plan to update you with the results of my testing.

 

Regards,

 

Rob Audino

 

Rob Audino (he/him)
Project Engineer |  Aspen Technology, Inc.
work: +1-763-404-4042  |  www.aspentech.com

 

[Audino, Rob]

Hi Daniel

 

I checked my system this morning, and it seems that the “svnsync” spam has continued, but only for one of my repositories at this point (called “Displays”). When I run the svnsync commands manually, the svnsync for my “Displays” repository works properly (transmitting file data, committing revisions, and copying revision properties), but the svnsync command for my “Tabulars” repository gives the following result:

 

“svnsync: E000022: Destination HEAD (5) is not the last merged revision (4); have you committed to the destination without using svnsync?”

 

Let me know if you have seen this issue before. Running that svn command with the “--sync-username” and “--sync-password” options gave the same output.

 

I will say this: the way my system is intended to be set up means that the two servers with the repositories should be interchangeable (as opposed to one server having a “read-only” copy of the repository). Does that mean that I should put the svnsync init commands back into the batch script?

 

In addition, I realized that the user that runs the “svnsync” commands when scheduled is DIFFERENT than if I just log in and run them manually. This may be a source of the issues, but I don’t know for sure. Also, when I run an svnsync command manually (specifically for the “Displays” repository), it seems to run its course and then terminate within 20 seconds or less.

 

Please let me know if you have any more ideas as to what may be going on here.

 

Thanks again

 

Rob Audino

 

 

Rob Audino (he/him)
Project Engineer |  Aspen Technology, Inc.
work: +1-763-404-4042  |  www.aspentech.com

 

[Daniel Sahlberg]

Den ons 5 okt. 2022 kl 15:51 skrev Audino, Rob <Rob.A...@aspentech.com>:

Hi Daniel

 

I checked my system this morning, and it seems that the “svnsync” spam has continued, but only for one of my repositories at this point (called “Displays”). When I run the svnsync commands manually, the svnsync for my “Displays” repository works properly (transmitting file data, committing revisions, and copying revision properties), but the svnsync command for my “Tabulars” repository gives the following result:

 

“svnsync: E000022: Destination HEAD (5) is not the last merged revision (4); have you committed to the destination without using svnsync?”

 

Let me know if you have seen this issue before. Running that svn command with the “--sync-username” and “--sync-password” options gave the same output.

As asked in the error message, have you committed to the destination repository outside of svnsync?

 I will say this: the way my system is intended to be set up means that the two servers with the repositories should be interchangeable (as opposed to one server having a “read-only” copy of the repository). Does that mean that I should put the svnsync init commands back into the batch script?

I'm quite sure that scenario is not supported by svnsync, in fact the SVN Book [1] says "allowing the revision history in the target repository to change by any mechanism other than the mirroring process is a recipe for disaster."

There are several commercial servers providing synchronisation solutions with writeable slaves. I haven't used any of them, but I know at least two of the vendors listed on the Subversion website [2] have server distributions with replication and writeable slaves.

 

In addition, I realized that the user that runs the “svnsync” commands when scheduled is DIFFERENT than if I just log in and run them manually. This may be a source of the issues, but I don’t know for sure. Also, when I run an svnsync command manually (specifically for the “Displays” repository), it seems to run its course and then terminate within 20 seconds or less.

That is probably an issue. I'm assuming the username/password is cached for the user you are logged in as. (In the previous e-mail I wrote "Windows Credentials store" however I was mistaken. The username/password is probably stored in one of the files in %AppData%\Subversion\auth\svn.simple, with the password encrypted using Windows API functions [3]). If the user executing the svnsync commands doesn't have the cached credentials in the credentials cache it will probably hang waiting for username/password.

Kind regards,

Daniel

[1] https://svnbook.red-bean.com/en/1.7/svn.reposadmin.maint.html#svn.reposadmin.maint.replication

[2] https://subversion.apache.org/packages.html#windows

[3] https://subversion.apache.org/faq.html#plaintext-passwords

[Audino, Rob]

Hi Daniel

 

Thanks for your detailed response here. Here are some answers to your follow-up questions

1. I do not think that I have committed to the destination repository outside of using svnsync. Would committing the display normally count as a commit outside of using svnsync?
2. The company I work for has done configuration like this before, so I believe that we have found a way to make it work.
3. I do not know if the “Process Scheduler” that we are using is running as the user within that “svn.simple” file. I plan on inquiring about this, just to ensure that it is authenticating properly.

 

I can let you know if I have further questions in this regard. Let me know if you come up with or glean anything new for me to try.

 

Regards,

 

Rob Audino

 

Rob Audino (he/him)
Project Engineer |  Aspen Technology, Inc.
work: +1-763-404-4042  |  www.aspentech.com

 

From: Daniel Sahlberg <daniel.l...@gmail.com>
Sent: Wednesday, October 5, 2022 10:14 AM
To: Audino, Rob <Rob.A...@aspentech.com>
Cc: us...@subversion.apache.org
Subject: Re: Questions: svnsync Command

 

Den ons 5 okt. 2022 kl 15:51 skrev Audino, Rob <Rob.A...@aspentech.com>:

Hi Daniel

 

I checked my system this morning, and it seems that the “svnsync” spam has continued, but only for one of my repositories at this point (called “Displays”). When I run the svnsync commands manually, the svnsync for my “Displays” repository works properly (transmitting file data, committing revisions, and copying revision properties), but the svnsync command for my “Tabulars” repository gives the following result:

 

“svnsync: E000022: Destination HEAD (5) is not the last merged revision (4); have you committed to the destination without using svnsync?”

 

Let me know if you have seen this issue before. Running that svn command with the “--sync-username” and “--sync-password” options gave the same output.

 

As asked in the error message, have you committed to the destination repository outside of svnsync?

 

 I will say this: the way my system is intended to be set up means that the two servers with the repositories should be interchangeable (as opposed to one server having a “read-only” copy of the repository). Does that mean that I should put the svnsync init commands back into the batch script?

 

I'm quite sure that scenario is not supported by svnsync, in fact the SVN Book [1] says "allowing the revision history in the target repository to change by any mechanism other than the mirroring process is a recipe for disaster."

 

There are several commercial servers providing synchronisation solutions with writeable slaves. I haven't used any of them, but I know at least two of the vendors listed on the Subversion website [2] have server distributions with replication and writeable slaves.

 

 

In addition, I realized that the user that runs the “svnsync” commands when scheduled is DIFFERENT than if I just log in and run them manually. This may be a source of the issues, but I don’t know for sure. Also, when I run an svnsync command manually (specifically for the “Displays” repository), it seems to run its course and then terminate within 20 seconds or less.

 

That is probably an issue. I'm assuming the username/password is cached for the user you are logged in as. (In the previous e-mail I wrote "Windows Credentials store" however I was mistaken. The username/password is probably stored in one of the files in %AppData%\Subversion\auth\svn.simple, with the password encrypted using Windows API functions [3]). If the user executing the svnsync commands doesn't have the cached credentials in the credentials cache it will probably hang waiting for username/password.

 

Kind regards,

Daniel

 

[1] https://svnbook.red-bean.com/en/1.7/svn.reposadmin.maint.html#svn.reposadmin.maint.replication [svnbook.red-bean.com]

[2] https://subversion.apache.org/packages.html#windows [subversion.apache.org]

[3] https://subversion.apache.org/faq.html#plaintext-passwords [subversion.apache.org]

[Daniel Sahlberg]

Den ons 19 okt. 2022 kl 17:56 skrev Audino, Rob <Rob.A...@aspentech.com>:

Hi Daniel

 

Thanks for your detailed response here. Here are some answers to your follow-up questions

1. I do not think that I have committed to the destination repository outside of using svnsync. Would committing the display normally count as a commit outside of using svnsync?

Not sure I understand "committing the display", but any commit to the "second server" would break the setup - from the error message it seems that this is what happened. I think your best bet is to figure look at svn log from both servers, figure out what changes were committed to the second server, make sure they are committed to the primary server and then delete the repository from the second server and restart the sync (create repo, svnsync init, svnsync sync).

1. The company I work for has done configuration like this before, so I believe that we have found a way to make it work.

I guess YMMW, but I'd trust the quote from the SVN Book. If you need to commit to any of the servers, check the commercial server offerings.

 

1. I do not know if the “Process Scheduler” that we are using is running as the user within that “svn.simple” file. I plan on inquiring about this, just to ensure that it is authenticating properly.

The process schedule fire off a process as _some_ user. That user needs to have an svn.simple file in their profile. Easiest way is probably to login as that user and run the commands manually - you would see the authentication happening. Adding --sync-username and --sync-password to the batch file would be another option, however exposing the username and password in cleartext in both the process scheduler and in task manager to any user with enough credentials.

(You mentioned trying with --sync-username and --sync-password didn't help, but I understood this to be with regards to the error message "svnsync: E000022". That error message is unrelated to the potential authentication problem for your process scheduler.

I can let you know if I have further questions in this regard. Let me know if you come up with or glean anything new for me to try.

Just one last thing: Please use interleaved posting [1] and keep the context intact instead of top-posting. It makes it a lot easier backreference what you are replying to.

Kind regards,

Daniel

[1] https://en.wikipedia.org/wiki/Posting_style