SCP2 and large files

[Jeremy Begg]

Hi!

A customer running HP OpenVMS V8.4 with TCP/IP Services V5.7-ECO5 is
having a problem with SCP2.

The nightly VMS backup job creates BACKUP savesets on a local disk.
When each saveset is completed (i.e. when the BACKUP/IMAGE finishes) a
batch job is submitted to transfer the saveset to a Linux system using
SCP2. (From there it goes to a large archiving system.)

Those transfers work fine for most of the savesets, which vary in size
from 9GB to 52GB. However there's one saveset which is close to 500GB
for which the transfer never works.

It appears that SCP2 does the following:

1. Connects to the remote (Linux) server and logs in.
2. Does a stat() on the saveset (presumably to get its exact size).
3. Starts transferring the file.

I think where it's going wrong is step 2. The stat() on the 500GB
saveset takes a very long time (seems to be about 90 minutes), by which
time the Linux end has given up waiting for the transfer to begin.

Therefore two possible solutions spring to mind:

a. Can the CRTL stat() routine be told to guess the file size from the
RMS file header (ALQ perhaps)? I thought there might be a DECC$ feature
logical to do this, but I can't find it.

b. Can SCP2 be told to enable TCP keepalive on the connection?

Thanks,

Jeremy Begg

[BlackCat]

On 12.08.20 08:47, Jeremy Begg wrote:
> Can SCP2 be told to enable TCP keepalive on the connection?

Jeremy,

Just set TCP Keepalive as a default.

$ pipe sysconfig -q inet | sea sys$pipe keep
tcp_keepalive_default = 1
tcp_keepcnt = 4
tcp_keepidle = 40
tcp_keepinit = 40
tcp_keepintvl = 20

To define these permanently use:
TCPIP$ETC:SYSCONFIGTAB.DAT

[Scott Dorsey]

Jeremy Begg <jer...@vsm.com.au> wrote:
>I think where it's going wrong is step 2. The stat() on the 500GB
>saveset takes a very long time (seems to be about 90 minutes), by which
>time the Linux end has given up waiting for the transfer to begin.
>
>Therefore two possible solutions spring to mind:
>
>a. Can the CRTL stat() routine be told to guess the file size from the
>RMS file header (ALQ perhaps)? I thought there might be a DECC$ feature
>logical to do this, but I can't find it.
>
>b. Can SCP2 be told to enable TCP keepalive on the connection?

Isn't there a BACKUP option that will cause BACKUP to make multiple smaller
savesets which might be easier to handle for your script?
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[Simon Clubley]

On 2020-08-12, Jeremy Begg <jer...@vsm.com.au> wrote:
>
> b. Can SCP2 be told to enable TCP keepalive on the connection?
>

Are you sure it's a TCP keepalive problem and not a protocol-level
timeout problem ?

Have you also checked the files after they have been transferred to the
remote system to make sure they survived the journey intact ?

I don't know about VMS SCP2, but I have had problems with TCP/IP
Services sftp in the past.

You might want to do compare hashes on the saveset on both the Linux
side and the VMS side to make sure they match.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

[John Reagan]

On Wednesday, August 12, 2020 at 2:47:52 AM UTC-4, Jeremy Begg wrote:

>
> a. Can the CRTL stat() routine be told to guess the file size from the
> RMS file header (ALQ perhaps)? I thought there might be a DECC$ feature
> logical to do this, but I can't find it.
>

You mean "st_size" in the stat buffer, yes? We do use RMS (if the file is on network device, we'll fall back to a $QIOW/IO$_ACCESS and dig through the FAT but I doubt you are doing that path)

1 51613 if (fhc->xab$l_ebk != 0) /* Calculate file size in bytes */
1 51614 stat_buf->st_size = (off64_t)(fhc->xab$l_ebk - 1) * RMS_BLOCKSIZE + fhc->xab$w_ffb;
1 51615 else
1 51616 stat_buf->st_size = (off64_t)fhc->xab$l_hbk * RMS_BLOCKSIZE + fhc->xab$w_ffb;

[Arne Vajhøj]

One should just remember that the relationship between "size of file on
disk" and "bytes that will be read from file" can be rather complex on
VMS.

Arne

[Phillip Helbig (undress to reply)]

In article <rh03ak$121p$1...@gioia.aioe.org>, Jeremy Begg

<jer...@vsm.com.au> writes:

> I think where it's going wrong is step 2. The stat() on the 500GB
> saveset takes a very long time (seems to be about 90 minutes), by which
> time the Linux end has given up waiting for the transfer to begin.

Why not do ZIP/VMS/LEVEL=9 on the saveset?

[johnwa...@yahoo.co.uk]

Would you care to, er, expand on that a little bit?

I *think* I know the answer, and it's already been
hinted at, but maybe a DIR /FULL and/or a DUMP /HEADER
would add a bit more to the discussion, perhaps.

[Stephen Hoffman]

On 2020-08-12 07:17:48 +0000, Jeremy Begg said:

> A customer running HP OpenVMS V8.4 with TCP/IP Services V5.7-ECO5 is
> having a problem with SCP2.

> ...

Poke the Linux box with ssh or some other mechanism, and trigger an scp
over there; pull the file to the Linux system?
Linux scp permits easier access to the keepalive, too.But the OpenVMS
TCP/IP Services ssh_config file does implement a KeepAlive option,


--
Pure Personal Opinion | HoffmanLabs LLC

[bill...@navistar.com]

As a work around to get smaller savesets have you tried backup's compression on the savesets?

I've been doing backups on one server to remote savesets on another VMS server lately (using DECnet-plus FAL, not SCP after the saveset creation). My bottleneck is the network, not CPU or disk I/O and I've found that the reduced saveset size it really helps reduce the elapsed time. I've seen between 50% and almost 90% compression on a saveset depending on the contents of the source.

$ backup /data_format=compress=deflate/group=0 <source disk>/image -
remote::<save_set>/save_set

From a recent log file:
%BACKUP-I-COMPRESS, data compressed by 74%
%BACKUP-I-COMPRESS, data compressed by 49%
%BACKUP-I-COMPRESS, data compressed by 89%

[Jeremy Begg]

Hi John,

That should be easy enough to test so I'll see if that does indeed fix
the problem. (I'm not certain it's a keepalive issue but having seen
*no* packets transferred between client and server while the large
backup saveset is being scanned by the SCP client, I have to assume it's
a possibility.)

Thanks,

Jeremy Begg

[Jeremy Begg]

Hi Simon,

On 12/8/20 22:01, Simon Clubley wrote:
> On 2020-08-12, Jeremy Begg <jer...@vsm.com.au> wrote:
>>
>> b. Can SCP2 be told to enable TCP keepalive on the connection?
>>
>
> Are you sure it's a TCP keepalive problem and not a protocol-level
> timeout problem ?

Not 100% certain but I see *no* packets between client and server while
the file is being processed on the VMS side. Keepalive should show
*some* packets, I think.

> Have you also checked the files after they have been transferred to the
> remote system to make sure they survived the journey intact ?
>
> I don't know about VMS SCP2, but I have had problems with TCP/IP
> Services sftp in the past.
>
> You might want to do compare hashes on the saveset on both the Linux
> side and the VMS side to make sure they match.

Yes, I have successfully retrieved these savesets (the smaller ones!)
from the Linux host and restored files from them. The RMS file
attributes of the saveset had to be fixed after retrieval, no surprises
there.

Jeremy Begg

[Jeremy Begg]

Hi John,

I don't know what the stat() function is actually be asked to find; all
I noticed was that in the SCP2 debug log there is a reference to calling
stat() on the file, and that's when it starts reading the saveset
without transferring any data. Here's the relevant part of the log:

tcpip$ssh_scp2.exe:SshFileCopy/SSHFILECOPY.C:1301: Connection to remote
host 'kp-backup01', ready to serve requests.
tcpip$ssh_scp2.exe:SshFCTransfer/SSHFC_TRANSFER.C:498: Source file is
"raw", and it needs to be parsed.
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3481: Received SSH_FXP_STAT
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3501: home directory:
DSA2:[SSS.JEREMY]
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3568: Statting file
'/backup_savesets/dsa1_200812.bck'

In fact it does it *twice*! The log file continues ...

tcpip$ssh_scp2.exe:SshFCTransfer/SSHFC_TRANSFER.C:663: Next source file
is /backup_savesets/dsa1_200812.bck .
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3481: Received SSH_FXP_STAT
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3501: home directory:
DSA2:[SSS.JEREMY]
tcpip$ssh_scp2.exe:Ssh2SftpServer/SSHFILEXFERS.C:3568: Statting file
'/backup_savesets/dsa1_200812.bck'

Somewhat broken, IMHO. Note this is HP TCP/IP Services so not likely to
be fixed. I should try a similar exercise using MultiNet and VSI TCP/IP
to see how they behave.

Jeremy

[Jeremy Begg]

Because it takes too long.

This backup job initially used BACKUP/DATA=COMPRESS but the time
required to compress the savesets meant the daily backup job was taking
over a day to run!

I know BACKUP/DATA=COMPRESS uses Zlib not Info-ZIP but I would be very
surprised if one turned out to be twice as fast as the other. I'm also
not sure that it would yield a useful improvement in the file size.

Jeremy

[Jeremy Begg]

Hi Stephen,

Telling Linux to pull the file is an interesting suggestion, and easy
enough to see if it works. (I have interactive shell access to that box
so I can at least test the transfer without having to change any
procedures.)

Thanks!

Jeremy Begg

[Phillip Helbig (undress to reply)]

In article <rh21lt$195r$1...@gioia.aioe.org>, Jeremy Begg

<jer...@vsm.com.au> writes:

> > Why not do ZIP/VMS/LEVEL=9 on the saveset?
>
> Because it takes too long.
>
> This backup job initially used BACKUP/DATA=COMPRESS but the time
> required to compress the savesets meant the daily backup job was taking
> over a day to run!
>
> I know BACKUP/DATA=COMPRESS uses Zlib not Info-ZIP but I would be very
> surprised if one turned out to be twice as fast as the other.

Give it a try. Of course, don't try to compress something which is
already compressed.

> I'm also
> not sure that it would yield a useful improvement in the file size.

Depends on what the data are. Experiment with /LEVEL to get the
trade-off between speed and size you need.

[Jan-Erik Söderholm]

BACKUP/DATA=COMPRESS is only meaningful if you have spare CPU cycles.
It will use more CPU to produce a smaller saveset. If you already are
CPU constrainted, it will take a longer total time, yes.

The saved size can be estimated from the kind of files backed up.
Or you can test on a few example files.