Rsync - Progress on OpenVMS.
John E. Malmberg
I have posted some files on Encompasserve.org for those that want to
help work on getting rsync to run on OpenVMS.
What I did is replaced the fork() call with using pthreads to run the
second process as a thread.
This still needs some work, as rsync uses many global and static
variables, and the second process running as a thread could corrupt them.
The files are at
http://encompasserve.org/~malmberg/rsync/
The readme file has a more detailed explanation.
I was able to produce a binary that appears to be able to properly
transfer files, inspite of the risks of data corruption.
I am not posting the binaries, to insure that no one accidentally tries
to do something important with them.
For those that do not know what rsync is, rsync is a remote
synchronization program.
It allows multiple developers to keep a local directory tree
synchronized with a master, so that they can be building from the same
sources as everyone else.
It is also used for transfering DNS zone information and has other
applications.
If you are working on an open source application, you may notice that in
addition to the ftp and http and cvs access to the source they will have
an rsync access also. The rsync access can be the easiest way for you
to make sure that you are using the same source as the UNIX platforms,
so you can do concurrent releases.
The home page is: http://rsync.samba.org
While the rsync developers are not real interested in OpenVMS specific
modifications, they are interested in if a fully threaded version can be
developed.
Now that vmsnet.vms-posix seems to have been cleaned out, thanks to the
efforts of several comp.os.vms regulars, it looks like a good place to
discuss this.
I have also posted a note on the encompasserve.org VMS conference.
-John
wb8...@qsl.network
Personal Opinion Only
Craig A. Berry
"John E. Malmberg" <wb8...@qsl.network> wrote:
> I have posted some files on Encompasserve.org for those that want to
> help work on getting rsync to run on OpenVMS.
John, this is excellent news. I have my finger in too many open source
pies already and may not be able to help much, but this is a package I
would definitely use.
I managed to get a clean build, but only with MMS, not MMK. I tried a
quick test rsyncing the current Perl source. Here's one edge case that
needs to be dealt with:
mkstemp ..patch.005797 failed: error in version number
This even though I had extended parse enabled and an ODS-5 volume. It
still wants to interpret the 123 in foo.123 as a version number. It
seems like we would want to intercept these and convert to foo^.123.
I didn't get much farther before seeing:
%SYSTEM-F-STKOVF, stack overflow, PC=FFFFFFFF808A4FA4, PS=0000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs
PC
0 FFFFFFFF808A4FA4
FFFFFFFF808A4FA4
0 FFFFFFFF808A3840
FFFFFFFF808A3840
0 FFFFFFFF809C58A4
FFFFFFFF809C58A4
RSYNC VMS_RENAME vms_rename 1795 0000000000000074
0000000000050554
RSYNC UTIL robust_rename 17883 0000000000000DCC
000000000003C58C
RSYNC RSYNC finish_transfer 17757 0000000000000A54
0000000000030A54
RSYNC RECEIVER recv_files 18002 0000000000001944
0000000000041824
RSYNC START_RECV_THREAD recv_thread 24163 0000000000000044
000000000004F064
PTHREAD$RTL 0 000000000003E504
000000007BCD0504
PTHREAD$RTL 0 000000000002C630
000000007BCBE630
0 0000000000000000
0000000000000000
PTHREAD$RTL ?
?
0 FFFFFFFF8028359C
FFFFFFFF8028359C
This is not too surprising given the thread safety issues that you've
only begun to address. BTW, I don't see the /REENTRANCY compiler flag
being used anywhere, though that would only help with CRTL thread
safety, not rsync code specifically.
I see that you have a getpass() implementation. I have one too:
http://cvs.sourceforge.net/viewcvs.py/*checkout*/freetds/freetds/vms/getp
ass.c?content-type=text%2Fplain&rev=1.3
Mine does a couple of things yours doesn't, like finding out what the
default timeout should be from SYI$_LGI_PWD_TMO, and using SMG if the
input device is not a tty or the caller wants line editing. However, I
can see by comparing with yours that I should probably cache some
things between calls, like the assigned channel (rather than
deassigning and reassigning on every call).
So, that's a bit of a ramble but I applaud your efforts on rsync and
will be eager to see things progress.
John E. Malmberg
> In article <5lGmb.129727$qj6.8...@news1.news.adelphia.net>,
> "John E. Malmberg" <wb8...@qsl.network> wrote:
> John, this is excellent news. I have my finger in too many open source
> pies already and may not be able to help much, but this is a package I
> would definitely use.
>
> I managed to get a clean build, but only with MMS, not MMK. I tried a
> quick test rsyncing the current Perl source. Here's one edge case that
> needs to be dealt with:
>
> mkstemp ..patch.005797 failed: error in version number
I need to get a small reproducer for the CRTL folks. From what I can
see, the failure seems to be in the rename() call.
On my 7.3-2 EFT system, I was getting that error until I set the other
UNIX file options. I was seeing the file created, but not renamed. It
appeared that rename is not following the UNIX parse options. But a
reproducer would make sure of it.
The actual fastest workaround might be to change the template to
mkstemp() so that the last part of the name will not be a pure number.
> This even though I had extended parse enabled and an ODS-5 volume. It
> still wants to interpret the 123 in foo.123 as a version number. It
> seems like we would want to intercept these and convert to foo^.123.
The CRTL should already be doing this. I need to get a small reproducer
to get this looked at.
> I didn't get much farther before seeing:
>
> %SYSTEM-F-STKOVF, stack overflow, PC=FFFFFFFF808A4FA4, PS=0000001B
> %TRACE-F-TRACEBACK, symbolic stack dump follows
> image module routine line rel PC abs
> PC
Probably needs a larger thread stack size. Seems to be related to
rename(). This might be CRTL version related.
> This is not too surprising given the thread safety issues that you've
> only begun to address. BTW, I don't see the /REENTRANCY compiler flag
> being used anywhere, though that would only help with CRTL thread
> safety, not rsync code specifically.
I am setting reentrancy. Instead of using the /Reentrancy call, I am
using a lib$initialize module.
See the module RSYNC_VMS_INIT.C. This sets all the UNIX behavior
features and the reentrancy module. This way you should not need to set
behavior logicals before running, or special compile options.
For some of the older versions of OpenVMS, it may need to be tweaked to
use logicals instead of the feature set calls.
> I see that you have a getpass() implementation. I have one too:
>
> http://cvs.sourceforge.net/viewcvs.py/*checkout*/freetds/freetds/vms/getp
> ass.c?content-type=text%2Fplain&rev=1.3
>
> Mine does a couple of things yours doesn't, like finding out what the
> default timeout should be from SYI$_LGI_PWD_TMO, and using SMG if the
> input device is not a tty or the caller wants line editing. However, I
> can see by comparing with yours that I should probably cache some
> things between calls, like the assigned channel (rather than
> deassigning and reassigning on every call).
>
> So, that's a bit of a ramble but I applaud your efforts on rsync and
> will be eager to see things progress.
I yanked the getpass() call out of the Frontport library. Since rsync
only calls the it once, the caching of the channel should not be needed.
That was my idea behind Frontport, one wrapper library to maintain for
multiple programs. However it was falling behind the UNIX portability
work on Alpha, plus it would require quite a bit of work to get it to
also support the large file sizes.
Something like the Frontport library would probably be needed for either
VAX or older Alpha versions.
Craig A. Berry
"John E. Malmberg" <wb8...@qsl.network> wrote:
> Craig A. Berry wrote:
> > In article <5lGmb.129727$qj6.8...@news1.news.adelphia.net>,
> > "John E. Malmberg" <wb8...@qsl.network> wrote:
> <snips>
> >
> > mkstemp ..patch.005797 failed: error in version number
>
> I need to get a small reproducer for the CRTL folks. From what I can
> see, the failure seems to be in the rename() call.
Hmm. The only place I can see that would report an mkstemp failure
with the above message is line 438 of receiver.c, unless you mean
rename() is used internally by mkstemp(). In any case, here's your
reproducer:
$ type try_mkstemp.c
#include <stdlib.h>
#include <stdio.h>
int main() {
char fname[32] = ".bar.XXXXXX";
int fd = mkstemp(fname);
if (fd == -1) {
perror("mkstemp for failed");
fprintf(stderr, "filename result was %s\n", fname);
exit(44);
}
}
$ run try_mkstemp
mkstemp for failed: non-translatable vms error code: 0x186FC
%rms-f-ver, error in version number
filename result was .bar.000457
%SYSTEM-F-ABORT, abort
>
> On my 7.3-2 EFT system, I was getting that error until I set the other
> UNIX file options. I was seeing the file created, but not renamed. It
> appeared that rename is not following the UNIX parse options. But a
> reproducer would make sure of it.
>
> The actual fastest workaround might be to change the template to
> mkstemp() so that the last part of the name will not be a pure number.
Or leave the pure number but make the dot into an underscore in the
template, like the following (untested):
--- receiver.c;-0 Mon Jan 20 17:32:17 2003
+++ receiver.c Sat Nov 1 13:56:15 2003
@@ -179,7 +179,7 @@
rprintf(FERROR,"filename too long\n");
return 0;
}
- snprintf(fnametmp,MAXPATHLEN, "%s/.%s.XXXXXX",tmpdir,f);
+ snprintf(fnametmp,MAXPATHLEN, "%s/.%s_XXXXXX",tmpdir,f);
return 1;
}
@@ -192,11 +192,11 @@
if (f) {
*f = 0;
- snprintf(fnametmp,MAXPATHLEN,"%s/.%s.XXXXXX",
+ snprintf(fnametmp,MAXPATHLEN,"%s/.%s_XXXXXX",
fname,f+1);
*f = '/';
} else {
- snprintf(fnametmp,MAXPATHLEN,".%s.XXXXXX",fname);
+ snprintf(fnametmp,MAXPATHLEN,".%s_XXXXXX",fname);
}
return 1;
Brad McCusker
"Craig A. Berry" <craig...@mac.com.spamfooler> wrote in message
news:f70989528e11f6f0...@news.teranews.com...
> In article <fBGob.596$Bf7.5...@news1.news.adelphia.net>,
> "John E. Malmberg" <wb8...@qsl.network> wrote:
>
> > Craig A. Berry wrote:
> > > In article <5lGmb.129727$qj6.8...@news1.news.adelphia.net>,
> > > "John E. Malmberg" <wb8...@qsl.network> wrote:
> > <snips>
> > >
> > > mkstemp ..patch.005797 failed: error in version number
> >
> > I need to get a small reproducer for the CRTL folks. From what I can
> > see, the failure seems to be in the rename() call.
>
> Hmm. The only place I can see that would report an mkstemp failure
> with the above message is line 438 of receiver.c, unless you mean
> rename() is used internally by mkstemp(). In any case, here's your
> reproducer:
>
Guys - I can't reproduce the problem. Well, actually, I can, on V7.2-2 with
the March 2003 ECO. But, on V7.3 and V7.3-1 with the Sep 2003 ECO, I can't
reproduce it. John already indicated he can't reproduce it on V7.3-2, so,
it appears that this problem is resolved with Sep 2003 ECOs to V7.2-2 (not
released yet), V7.3, V7.3-1. Its also fixed in V7.3-2 (soon to ship) and
E8.1 (end of year ship). I'm not going to look into this any further,
unless someone gives me reason to.
On a related note, regarding John's comments about rename(). there is a
feature logical, DECC$RENAME_NO_INHERIT that really should have been called
DECC$RENAME_FOLLOW_UNIX_SEMANTICS. Setting that switch sends rename down
and entirely different code path than when the switch is clear. And that
new code path tries to follow the XPG5/6 specification for rename(). It
might not have it perfect, but, its a lot closer and much different than
when the switch is clear.
The V7.3-2 version of the C RTL reference will have a bit more information
on what that switch does.
Brad McCusker
OpenVMS Engineering
C RTL Project Leader
Hewlett-Packard Company
Nashua, NH USA
Craig A. Berry
"Brad McCusker" <brad.mcc...@Mcompaq.com> wrote:
> > > > mkstemp ..patch.005797 failed: error in version number
>
> Guys - I can't reproduce the problem. Well, actually, I can, on V7.2-2 with
> the March 2003 ECO. But, on V7.3 and V7.3-1 with the Sep 2003 ECO, I can't
> reproduce it. John already indicated he can't reproduce it on V7.3-2, so,
> it appears that this problem is resolved with Sep 2003 ECOs to V7.2-2 (not
> released yet), V7.3, V7.3-1. Its also fixed in V7.3-2 (soon to ship) and
> E8.1 (end of year ship). I'm not going to look into this any further,
> unless someone gives me reason to.
Brad, the particular error is dependent on what values mkstemp()
chooses to put into the XXXXXX pattern. If it puts in only the digits
0-9, the last chunk including a dot gets interpreted as a version
number. I've seen this on 7.3-1, though I've also seen it not
happening on 7.3-1.
Most of the time it cycles through some combination of a-z as well as
0-9, but I wonder if it doesn't start with 000000 or at least have some
cases where the template gets replaced with all numeric digits? (In
hopes that I could catch it wrapping around to the beginning, I tried
to cycle through all the possibilities and it's *very* slow; each
mkstemp() call takes a full second or two on a DPW 500au and burns a
lot of CPU in the process.)
Aha! The number it starts with is the pid. So if my pid happens to
contain only the digits 0-9, it's easy to reproduce the problem on
7.3-1. Just log out and log in again until you happen to get a pid
that does not have A-F in it:
$ create try_mkstemp.c
#include <stdlib.h>
#include <stdio.h>
int main() {
char fname[32] = ".bar.XXXXXX";
int fd = mkstemp(fname);
fprintf(stderr, "filename result was %s\n", fname);
if (fd == -1) {
perror("mkstemp failed");
exit(44);
}
}
^Z
$ cc try_mkstemp
$ link try_mkstemp
$ write sys$output f$getjpi("","PID")
00001860
$ run try_mkstemp
filename result was .bar.001860
mkstemp failed: non-translatable vms error code: 0x186FC
%rms-f-ver, error in version number
%SYSTEM-F-ABORT, abort
John, on an unrelated matter, you need to check the spelling of
DECC$READDIR_DROPDOTNOTYPE in your rsync code: you have "DOP" instead
of "DOT".
John E. Malmberg
Try $define DECC$FILENAME_UNIX_ONLY ENABLE before running it.
This makes the test succeed for the mkstemp() under 7.3-2 EFT. I have
not tested this on the real 7.3-2, so it may be fixed as Brad has posted.
> John, on an unrelated matter, you need to check the spelling of
> DECC$READDIR_DROPDOTNOTYPE in your rsync code: you have "DOP" instead
> of "DOT".
That was also a duplicate entry in the init module, so effectively did
nothing. I have fixed my copy of the module.
Thanks,
Craig A. Berry
"John E. Malmberg" <wb8...@qsl.network> wrote:
> > $ run try_mkstemp
> > filename result was .bar.001860
> > mkstemp failed: non-translatable vms error code: 0x186FC
> > %rms-f-ver, error in version number
> > %SYSTEM-F-ABORT, abort
>
> Try $define DECC$FILENAME_UNIX_ONLY ENABLE before running it.
>
> This makes the test succeed for the mkstemp() under 7.3-2 EFT. I have
> not tested this on the real 7.3-2, so it may be fixed as Brad has posted.
Yes, thanks, that also takes care of it on 7.3-1 with VMS731_ACRTL
V1.0. It doesn't seem like it would hurt, though, to make the pattern
always have at least one non-numeric in it.