Potential loss of data problem in sftp client in TCP/IP Services ?

[Simon Clubley]

I am experiencing a file truncation problem with a TCP/IP engineering
image version of the TCP/IP Services sftp client and I am wondering
if anyone here can reproduce it.

Try using the sftp client to send something small (so that it's easy to
check; I am using my login.com) to a Linux sftp server.

When I do this (both in "ascii dos" mode and binary mode) the last
few dozen bytes of my login.com are lost during transfer.

This worked on TCP/IP V5.6 ECO5 (at least in binary mode; ascii dos was
documented then but not implemented) without data loss, but it does not
work for me with the latest TCP/IP Engineering sftp image.

I'm trying to find out if someone else can get this to fail using the
latest published V5.7 ECOs or if it's just a unreleased TCP/IP Engineering
image problem (or something that triggers under very specific
circumstances only).

I've only just discovered this today after another unrelated problem was
fixed which stopped me from getting this far. I've passed it back to
support, and I am waiting for a response.

Thanks,

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Albrecht Schlosser]

On 12.03.2012 16:12, Simon Clubley wrote:
> I am experiencing a file truncation problem with a TCP/IP engineering
> image version of the TCP/IP Services sftp client and I am wondering
> if anyone here can reproduce it.
>
> Try using the sftp client to send something small (so that it's easy to
> check; I am using my login.com) to a Linux sftp server.
>
> When I do this (both in "ascii dos" mode and binary mode) the last
> few dozen bytes of my login.com are lost during transfer.

I can't help with your particular question, but is this "few dozen
bytes" potentially related to the number of lines in the file, maybe
with a factor of 1, 2, or 4? This would be a hint that it has to do
with file organization and/or [CR/]LF "translation". It could also
be related to even or odd record sizes (1 byte padding in text files).

I'd do another test with a stream lf file to check as well.

Just a thought...

Albrecht

[Ken Fairfield]

On Monday, March 12, 2012 8:12:48 AM UTC-7, Simon Clubley wrote:
> I am experiencing a file truncation problem with a TCP/IP engineering
> image version of the TCP/IP Services sftp client and I am wondering
> if anyone here can reproduce it.
>
> Try using the sftp client to send something small (so that it's easy to
> check; I am using my login.com) to a Linux sftp server.
>
> When I do this (both in "ascii dos" mode and binary mode) the last
> few dozen bytes of my login.com are lost during transfer.

I don't understand this. Granted, I don't have access to
a VMS sftp client or server anymore, but on my RHEL 5.5 server,
the sftp client knows nothing of the ftp mode commands like
"binary", "image", "ascii", etc... I thought these were
simply dropped from sftp.

Regards, Ken

[Simon Clubley]

It's a feature of the VMS sftp client in order to compensate for the
fact that VMS stores sequential files in various formats. It's been
documented for several versions, but only became active in the most
recent images. From the sftp help (this is the latest TCP/IP
Engineering image):

$ sftp
sftp> help
SFTP client Sftp2
) Copyright 1976, 2003 Hewlett-Packard Development Company, L.P.

Type 'help <topic>', where <topic> is one of the following commands:

open lopen close quit cd
lcd pwd lpwd ls lls
get mget put mput rm
lrm mkdir lmkdir rmdir lrmdir
rename lrename readlink lreadlink symlink
lsymlink ascii binary auto setext
getext lsroots debug verbose help

sftp> help binary
binary
Files will be transfered in binary mode.
sftp> help ascii
ascii [-s] [-f] [<remote_nl_conv>] [<local_nl_conv>]
With -s option, shows current newline convention. With "-f" option,
favors this configuration over what server notifies during connection
(this option mainly for testing). <remote_nl_conv> sets remote
newline convention. <local_nl_conv> operates on local side, but is
not as useful (the correct local newline convention is usually
compiled in, so this is mainly for testing). You can set either of
these to "ask", which will cause sftp to prompt you for the
newline convention when needed. With the exception of "-s" option,
this command sets transfer mode to ascii. Available conventions are
"dos", "unix" or "mac", using "\r\n", "\n" and "\r"
as newlines, respectively.

[Jan-Erik Soderholm]

Simon Clubley wrote 2012-03-12 17:11:

> From the sftp help (this is the latest TCP/IP
> Engineering image):

Should I read that as not the latest *released* version ?

And is it your opinion that this is a specific issue
in the sftp image as such?

Just trying to get the context right. We are planning for
an 8.2 => 8.4 upgrade with the latest 5.7 ECO3 TCPIP...

[Simon Clubley]

Congratulations, that appears to be the common factor.

> I'd do another test with a stream lf file to check as well.
>

I converted my login.com to stream_lf using convert and a FDL and the
full length transferred ok.

Can anyone else duplicate this on V5.7 ECO3 ?

I heard back from the support person and it doesn't appear to be
truncating the file on V5.7 ECO2 (he hasn't tried ECO3 yet), although
(reading between the lines) it appears V5.7 ECO2 still had the old
transfer code in the sftp client.

In the old transfer code (present up to the most recent sftp client
versions) all input files, regardless of format, got converted to
stream_lf format during transfer, resulting in a LF as a line
terminator in the destination file. The ascii command, while
documented, was ignored.

It appears that in the latest engineering image something is going
wrong during the conversion which results in the file been truncated
both in binary and ascii dos mode. I don't know if it's some timing
issue (which might explain way it wasn't caught during testing) or a
more general problem.

[Simon Clubley]

On 2012-03-12, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
> Simon Clubley wrote 2012-03-12 17:11:
>
>> From the sftp help (this is the latest TCP/IP
>> Engineering image):
>
> Should I read that as not the latest *released* version ?
>

Correct. I was asked to install the latest engineering image to
test my original problem against before they would work on my
original sftp problem.

Although I am still on V5.6 ECO5, the Engineering supplied sftp
image has significant enhanced functionality. I don't know how
much of that functionality (namely the ascii mode conversion) is
present in V5.7 ECO3 which is the latest released version. If this
new functionality is present in ECO3, I don't know if that version
works correctly or not.

> And is it your opinion that this is a specific issue
> in the sftp image as such?
>
> Just trying to get the context right. We are planning for
> an 8.2 => 8.4 upgrade with the latest 5.7 ECO3 TCPIP...

That's a very good question; I don't know if ECO3 is affected.
It's what I am trying to find out by asking my questions here.

[VAXman-]

sftp> help
SFTP client Sftp2

© Copyright 1976, 2003 Hewlett-Packard Development Company, L.P.

Type 'help <topic>', where <topic> is one of the following commands:

open lopen close quit cd
lcd pwd lpwd ls lls
get mget put mput rm
lrm mkdir lmkdir rmdir lrmdir
rename lrename readlink lreadlink symlink
lsymlink ascii binary auto setext
getext lsroots debug verbose help

However, I've never done anything but a put or get and sftp seems to
properly handle all the rest.

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

Well I speak to machines with the voice of humanity.

[Jan-Erik Soderholm]

Simon Clubley wrote 2012-03-12 18:02:
> On 2012-03-12, Jan-Erik Soderholm<jan-erik....@telia.com> wrote:
>> Simon Clubley wrote 2012-03-12 17:11:
>>
>>> From the sftp help (this is the latest TCP/IP
>>> Engineering image):
>>
>> Should I read that as not the latest *released* version ?
>>
>
> Correct. I was asked to install the latest engineering image to
> test my original problem against before they would work on my
> original sftp problem.
>
> Although I am still on V5.6 ECO5, the Engineering supplied sftp
> image has significant enhanced functionality. I don't know how
> much of that functionality (namely the ascii mode conversion) is
> present in V5.7 ECO3 which is the latest released version. If this
> new functionality is present in ECO3, I don't know if that version
> works correctly or not.
>
>> And is it your opinion that this is a specific issue
>> in the sftp image as such?
>>
>> Just trying to get the context right. We are planning for
>> an 8.2 => 8.4 upgrade with the latest 5.7 ECO3 TCPIP...
>
> That's a very good question; I don't know if ECO3 is affected.
> It's what I am trying to find out by asking my questions here.
>
> Simon.
>

OK. I have currently 8.4 and TCPIP 5.7 ECO3 in my office/hobbyist
system, but I do not know what is needed to run a reproducer. And
I do not think that I have the right server to run against.

If you do have some publicaly reachable server system, I could
give you access to my 5.7 ECO3 system if you'd like to run
some tests.

B.t.w, I'm on Alpha, I do not remember what you where running
(of if it matters).

Anyway,
thanks for your clarifications about "Engineering image". :-)

Jan-Erik.

[Simon Clubley]

On 2012-03-12, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
> Simon Clubley wrote 2012-03-12 18:02:
>> On 2012-03-12, Jan-Erik Soderholm<jan-erik....@telia.com> wrote:
>>>
>>> Should I read that as not the latest *released* version ?
>>>
>>
>> Correct. I was asked to install the latest engineering image to
>> test my original problem against before they would work on my
>> original sftp problem.
>>
>> Although I am still on V5.6 ECO5, the Engineering supplied sftp
>> image has significant enhanced functionality. I don't know how
>> much of that functionality (namely the ascii mode conversion) is
>> present in V5.7 ECO3 which is the latest released version. If this
>> new functionality is present in ECO3, I don't know if that version
>> works correctly or not.
>>
>>> And is it your opinion that this is a specific issue
>>> in the sftp image as such?
>>>
>>> Just trying to get the context right. We are planning for
>>> an 8.2 => 8.4 upgrade with the latest 5.7 ECO3 TCPIP...
>>
>> That's a very good question; I don't know if ECO3 is affected.
>> It's what I am trying to find out by asking my questions here.
>>
>

> OK. I have currently 8.4 and TCPIP 5.7 ECO3 in my office/hobbyist
> system, but I do not know what is needed to run a reproducer. And
> I do not think that I have the right server to run against.
>

Although my more detailed testing was against a local Linux server,
I have also seen the same failure against a Windows based sftp
server as well.

To try and reproduce:

1) On the VMS sftp client system, identify a small text file a few
blocks long; this file should be a variable length file in order to
reproduce my test environment as closely as possible. I have found
the problem does not occur when the input file is a stream_lf file.

I used my ~4 blocks long login.com file for detailed testing, but make
sure the file doesn't exist on the destination system (I don't want to
be responsible for you destroying one of your files by mistake while
following my instructions. :-)).

(BTW, input files other than stream_lf are supposed to be fully
supported. They certainly worked just fine, but with a forced
conversion to stream_lf during transfer, using V5.6 ECO 5)

2) sftp user...@remote.system.domain.name

(You may need to use quotes around the parameter.)

Supply password when prompted.

3) At the "sftp> " prompt enter "put filename".

4) On the remote system, look to see if the last few bytes of the
file are missing after transfer.

> If you do have some publicaly reachable server system, I could
> give you access to my 5.7 ECO3 system if you'd like to run
> some tests.
>

Unfortunately, the servers I am using are either local access only
or have very strict security policies, so this won't be possible.
Thanks for the offer though.

> B.t.w, I'm on Alpha, I do not remember what you where running
> (of if it matters).
>

Alpha V8.3, TCP/IP V5.6 ECO 5, but with the sftp and ssh client images
replaced by the latest TCP/IP Engineering images.

> Anyway,
> thanks for your clarifications about "Engineering image". :-)
>

You are welcome. :-)

[Simon Clubley]

On 2012-03-12, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>
> 3) At the "sftp> " prompt enter "put filename".
>

BTW, I left out the mode switching commands before the put because I
found with the current Engineering sftp image, it doesn't make any
difference to the failure mode.

However, if you are having problems reproducing it, try entering
"ascii dos" or "binary" as a seperate command before the put.

BTW, thanks in advance to anyone making the effort to try and identify
the scope of this problem.

[JF Mezei]

Simon Clubley wrote:

> However, if you are having problems reproducing it, try entering
> "ascii dos" or "binary" as a seperate command before the put.

I've never used sftp. Figured it woudl have same inerface as FTP with
similar commands but just using encrypted links. What is "ascii dos" ?


You mention variable length files. Is it possible that the VMS sftp
client would have trouble counting the number of bytes in the file and
place the "end of file" at the wrong place ?

If you CONVERT a RFM=VAR,RAT=CR file to RMF=VAR,RAT=NONE, there should
be 2 bytes of overhead per record which would match the file size since
<CR><LF> are also 2 bytes long at the end of each line. When using SFTP,
would the fille be sent completely ? and if it, is it the same missing
part ?

[Simon Clubley]

On 2012-03-12, JF Mezei <jfmezei...@vaxination.ca> wrote:
> Simon Clubley wrote:
>
>> However, if you are having problems reproducing it, try entering
>> "ascii dos" or "binary" as a seperate command before the put.
>
>
> I've never used sftp. Figured it woudl have same inerface as FTP with
> similar commands but just using encrypted links. What is "ascii dos" ?
>

No. Non-VMS clients just transfer a stream of bytes. The VMS TCP/IP sftp
client has been altered to support various VMS sequential file formats.
In the old version it did this by unconditionally converting the input
file to stream-lf format during transfer.

The new version has a more refined approach allowing greater control
over the line terminator encoding during transfer. "ascii dos" means
the input file is converted to use CRLF terminators during transfer.

>
> You mention variable length files. Is it possible that the VMS sftp
> client would have trouble counting the number of bytes in the file and
> place the "end of file" at the wrong place ?
>

If it's doing this, which is entirely possible, it's a massive regression
which is silently truncating previously supported file types during
transfer.

> If you CONVERT a RFM=VAR,RAT=CR file to RMF=VAR,RAT=NONE, there should
> be 2 bytes of overhead per record which would match the file size since
><CR><LF> are also 2 bytes long at the end of each line. When using SFTP,
> would the fille be sent completely ? and if it, is it the same missing
> part ?

It's not that simple; there's potentially a line termination conversion
going on during transfer and my experiments show that it doesn't matter
what line termination is active; the input file is still truncated at the
same place.

[glen herrmannsfeldt]

JF Mezei <jfmezei...@vaxination.ca> wrote:

(snip)

> I've never used sftp. Figured it woudl have same inerface as FTP with
> similar commands but just using encrypted links. What is "ascii dos" ?

Similar commands, but not exactly the same.

ftp by default transfers text files, converting to CRLF when sending,
and back again on receiving. With the bin option, or maybe also
some others, it transfers the bytes exactly as on disk.

sftp doesn't have an option to send text files while translating
line termination.

Also, ftp has, and defaults to, prompt mode where it asks for each
file in an mget or mput.

For sftp, the get and put commands allow for multiple files, with
no option for prompt mode. This might cause problems on some systems
if * and ? are used in file names.

There are more differences, but those are the ones you will notice
if you are used to using ftp.

-- glen

[David Froble]

Simon Clubley wrote:
> On 2012-03-12, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
>> Simon Clubley wrote 2012-03-12 17:11:
>>
>>> From the sftp help (this is the latest TCP/IP
>>> Engineering image):
>> Should I read that as not the latest *released* version ?
>>
>
> Correct. I was asked to install the latest engineering image to
> test my original problem against before they would work on my
> original sftp problem.
>
> Although I am still on V5.6 ECO5, the Engineering supplied sftp
> image has significant enhanced functionality. I don't know how
> much of that functionality (namely the ascii mode conversion) is
> present in V5.7 ECO3 which is the latest released version. If this
> new functionality is present in ECO3, I don't know if that version
> works correctly or not.
>
>> And is it your opinion that this is a specific issue
>> in the sftp image as such?
>>
>> Just trying to get the context right. We are planning for
>> an 8.2 => 8.4 upgrade with the latest 5.7 ECO3 TCPIP...
>
> That's a very good question; I don't know if ECO3 is affected.
> It's what I am trying to find out by asking my questions here.
>
> Simon.
>

I'm still trying to understand the problem. You indicate that you had been using V5.6
ECO5. Are you saying the problem exists in this version?

Last year I implemented a small application that uses SFTP to get and put files from and
to a non-VMS system. I needed to use a batch command file for SFTP since things ware
happening in a batch job, not from a terminal. I have not had any problems in either
direction. On the non-VMS system I do not know either the system, nor the file format. I
suspect some form of Unix as the directory / path structure used "/".

Now, the batch file I use for input to SFTP gets automatically translated into stream_lf
by SFTP, but as far as I can see, the actual files transfered do not.

The files I transfer all have records of the same length, but are sequential variable
files on VMS regardless of direction of transfer.

So what I'm trying to understand is, are you having trouble with all sequential variable
files, or just those files where individual records are of differing length?

[JF Mezei]

Simon Clubley wrote:

> It's not that simple; there's potentially a line termination conversion
> going on during transfer and my experiments show that it doesn't matter
> what line termination is active; the input file is still truncated at the
> same place.

When OS-X server (unix) would try to access VMS RFM=VAR files via NFS, I
found that the auto conversion to standard unix termination only worked
for the first portion of the file, after which, the data was sent raw
without record delimiters (but the 2 byte length was part of the "blob"
of data. I assumed it had to do with some buffering where the software
was only able to convert from RFN=VAR to RFM=STMLF within a fixed size
buffer and if the file was larger, then the rest didn't get converted.


But this predated 8.4 and the move of VMS to India. And does not seem to
be what you are experiencing.


And because this is sftp, you can't really use wireshark to see what is
being sent since it is encrypted. You'd need to somehow enable debug
logging on the unix host to see what is being sent.

[JF Mezei]

glen herrmannsfeldt wrote:

> sftp doesn't have an option to send text files while translating
> line termination.

I just checked on OS-X. I am surprised that there is no apparent comands
to dictate text or binary transfers.

The server side does have logging and debugging levels, so you should be
able to start the server sftp side with logging enabled to the max to
see what it sees from VMS.


Does the problem occur on any text file size ? Or only fil sizes larger
than a certain amount ?

[JF Mezei]

At one point, VMS did add a "file size" hint for RFM=VAR files so that
unix like utilities could get a file size of the file without having to
use the C-RTL to count the bytes.

But we were warned that this file size hint was not 100% reliable.

Is it possible that the software uses this , but the guys who write it
weren't told the file size hint is not reliable ?

[Steven Schweda]

> Can anyone else duplicate this on V5.7 ECO3 ?

If I still had patch access, then I might give it a try,
but I don't.

> [...] TCP/IP engineering image version of the TCP/IP
> Services sftp client [...]

And I certainly don't have one of those.

> Although I am still on V5.6 ECO5, the Engineering supplied

> sftp image has significant enhanced functionality. [...]

This is the program which is truncating your files? This
must be some new meaning for "enhanced".

> To try and reproduce:
> [...]

You can't create a guaranteed-to-fail file, and make it
available?


> [...] What is "ascii dos" ?

You couldn't read the HELP text provided?

> sftp doesn't have an option to send text files while
> translating line termination.

The protocol spec may not describe such a mode, but that
can't stop a client (or server) program from trying to adjust
the line endings.

> I converted my login.com to stream_lf using convert and a FDL
> and the full length transferred ok.

This should not be amazing. The VMS C RTL does things
with Stream_LF files which it doesn't do with even Stream or
Stream_CR, let alone non-Stream, variable-length-record
files. The exact behavior depends on the file attributes,
and on how the file is opened, and almost none of the details
is intuitive. With no access to the SFTP client source code,
it's hard to do more than guess, but many non-obvious
problems are easy to encounter. (Recent experience with
GnuPG v. text files, and with Zip v. Stream or Stream_CR with
long records, suggests that there are many more ways to do
things wrong than there are to do them right, and that the
variety of files needed for thorough testing is larger than
one might expect.)

> Can anyone else duplicate this on V5.7 ECO3 ?

I wouldn't worry much about that. If you can take N
files with different attributes which BACKUP /COMPARE and/or
DIFFERENCES (and/or GNU "diff") says are equivalent, and the
SFTP client can't send the data properly for M of them (where
M > 0), then you would seem to have found a bug/limitation in
the SFTP client.

Knowing nothing, my guess would be that the old code
handled the file-attribute differences by doing an effective
CONVERT-to-Stream_LF, and the new code thinks that it's
clever enough to skip that step, but it's not. And the
testing (before yours) was inadequate to catch the
problem(s). (But what do I know?)

[glen herrmannsfeldt]

JF Mezei <jfmezei...@vaxination.ca> wrote:

(snip)

> When OS-X server (unix) would try to access VMS RFM=VAR files via NFS, I
> found that the auto conversion to standard unix termination only worked
> for the first portion of the file, after which, the data was sent raw
> without record delimiters (but the 2 byte length was part of the "blob"
> of data. I assumed it had to do with some buffering where the software
> was only able to convert from RFN=VAR to RFM=STMLF within a fixed size
> buffer and if the file was larger, then the rest didn't get converted.

When NFS requests data from the server, it gives the offset into
the file for the data it wants. Personally, I have never run an NFS
server that did any conversion. You can see, though, that if you didn't
read from the beginning sequentially through, that it wouldn't know
where in the file a specific offset was without reading from the
beginning.

> But this predated 8.4 and the move of VMS to India. And does not seem to
> be what you are experiencing.

> And because this is sftp, you can't really use wireshark to see
> what is being sent since it is encrypted. You'd need to somehow
> enable debug logging on the unix host to see what is being sent.

Well, unix sftp pretty much writes the decrypted data stream to disk.
If anything is wrong, it is pretty likely that the VMS end sent
the wrong thing.

-- glen

[glen herrmannsfeldt]

JF Mezei <jfmezei...@vaxination.ca> wrote:

> At one point, VMS did add a "file size" hint for RFM=VAR files so that
> unix like utilities could get a file size of the file without having to
> use the C-RTL to count the bytes.

(snip)

Well, you really need more than just size, you need some map from
unix file offsets to VMS record offsets.

IBM's MVS also has a conversion problem, though there aren't
so many different formats. There are fixed length records, where
all are the same size, variable with a length field in the front
of each record (and block), and undefined where there are a bunch
of blocks with no known structure. Unlike the unix byte offset,
MVS just keeps track of blocks. If all the tracks are full, then
in the fixed block form you can compute the track/block number,
but they aren't always full. For V and U pretty much the only
way is to read from the beginning, counting bytes.

-- glen

[JF Mezei]

The big question is whether the sftp protocol requires a file size be
sent at the start of transmission, or whether it can send data until end
of file is reached and then indicate "got to EOF" to the remote server.

Consider the following snipped of pseudo code:


mysize = filesize("myfile.txt")
x = 0

while (x < mysize)
{ read_byte("myfile.txt")
write_byte("network_link")
x ++ ;
}


In the above case, if the filesize has been underestimated, then the
loop would end before the end of file has been reached.

The problem with having a bunch of newbies maintain VMS is that one is
never sure if they truly understand the intricacies of the file system
and how the C RTL interacts with RMS and how conversions are done on the
fly by the C RTL. They already showed their lack of knowledge wit the
DIR command bug about directory files.

[Simon Clubley]

On 2012-03-12, David Froble <da...@tsoft-inc.com> wrote:
>
> I'm still trying to understand the problem. You indicate that you had been using V5.6
> ECO5. Are you saying the problem exists in this version?
>

No, the file truncation problem does not exist in the sftp image supplied
with ECO5, but other different problems do exist, which is why the latest
Engineering images were loaded onto the system in question.

This file truncation problem is clearly a very recent problem. I am still
waiting to hear back from HP, so in the meantime I am trying to find out
if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
ECO3).

> Last year I implemented a small application that uses SFTP to get and put files from and
> to a non-VMS system. I needed to use a batch command file for SFTP since things ware
> happening in a batch job, not from a terminal. I have not had any problems in either
> direction. On the non-VMS system I do not know either the system, nor the file format. I
> suspect some form of Unix as the directory / path structure used "/".
>
> Now, the batch file I use for input to SFTP gets automatically translated into stream_lf
> by SFTP, but as far as I can see, the actual files transfered do not.
>

That's very different from my experiments. On the older versions of the sftp
client, all input files are converted to stream_lf during transfer. HP even
have a command procedure floating around internally which allows you to send
a file as a stream (CRLF terminated) format file using this older version.

It works by converting the file to stream format and then altering the file
attributes to make it appear as a stream_lf file so that it doesn't get
converted during transfer.

> The files I transfer all have records of the same length, but are sequential variable
> files on VMS regardless of direction of transfer.
>
> So what I'm trying to understand is, are you having trouble with all sequential variable
> files, or just those files where individual records are of differing length?

All the files I need to transfer have records of different length so that
is what I have been testing.

[Simon Clubley]

On 2012-03-12, JF Mezei <jfmezei...@vaxination.ca> wrote:

> glen herrmannsfeldt wrote:
>
>> sftp doesn't have an option to send text files while translating
>> line termination.
>
>
> I just checked on OS-X. I am surprised that there is no apparent comands
> to dictate text or binary transfers.
>

In Unix-land, which is what OS-X basically is, the convention is just to
send the file as a stream of bytes without any internal conversion.

Given all the various possible internal file structures for VMS sequential
files, this is not possible on VMS if you want the file to be usable on
the destination (non-VMS) system.

>
> Does the problem occur on any text file size ? Or only fil sizes larger
> than a certain amount ?

It also fails with my login.com which is about 3 to 4 blocks in size.

[Jan-Erik Soderholm]

Simon Clubley wrote 2012-03-13 10:48:
> On 2012-03-12, David Froble<da...@tsoft-inc.com> wrote:
>>
>> I'm still trying to understand the problem. You indicate that you had been using V5.6
>> ECO5. Are you saying the problem exists in this version?
>>
>
> No, the file truncation problem does not exist in the sftp image supplied
> with ECO5, but other different problems do exist, which is why the latest
> Engineering images were loaded onto the system in question.
>
> This file truncation problem is clearly a very recent problem. I am still
> waiting to hear back from HP, so in the meantime I am trying to find out
> if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
> ECO3).

I tried to run a few quick tests on my 8.4 (Alpha) TCPIP 5-7 ECO3 system,
but gave up after a number of different errors like "wrong SSH2 version".
I never understood what was the problem... :-)

In my case, since the thought is that this it not a server-side problem,
I simply runed against user@localhost. That might not work anyway if you
think that one has to run against another box to get the error.

Jan-Erik.

[Simon Clubley]

On 2012-03-13, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
>
> I tried to run a few quick tests on my 8.4 (Alpha) TCPIP 5-7 ECO3 system,
> but gave up after a number of different errors like "wrong SSH2 version".
> I never understood what was the problem... :-)
>

Was this during authentication ?

Regardless of which point this was at, that doesn't sound healthy...

Yet another problem I discovered during my early testing was that the
VMS sftp client would not switch to password mode after trying and
failing (as expected) to perform a certificate based authentication.

This was against one specific sftp server, but the Linux sftp client
had no problem with that server.

In the end, I had to force the use of password based authentication only
by using the option

sftp -o "AllowedAuthentications password"

on the command line. You may have more luck using the same option.

[Jan-Erik Soderholm]

Simon Clubley wrote 2012-03-13 11:21:
> On 2012-03-13, Jan-Erik Soderholm<jan-erik....@telia.com> wrote:
>>
>> I tried to run a few quick tests on my 8.4 (Alpha) TCPIP 5-7 ECO3 system,
>> but gave up after a number of different errors like "wrong SSH2 version".
>> I never understood what was the problem... :-)
>>
>
> Was this during authentication ?
>
> Regardless of which point this was at, that doesn't sound healthy...
>
> Yet another problem I discovered during my early testing was that the
> VMS sftp client would not switch to password mode after trying and
> failing (as expected) to perform a certificate based authentication.
>
> This was against one specific sftp server, but the Linux sftp client
> had no problem with that server.
>
> In the end, I had to force the use of password based authentication only
> by using the option
>
> sftp -o "AllowedAuthentications password"
>
> on the command line. You may have more luck using the same option.
>
> Simon.
>

OK.
Now, I guess it was not the intention to check *my* system... :-)

FWIW, here is what I get :

$ sftp -o "AllowedAuthentications password" "<user>@localhost"

warning: Authentication failed.
FATAL: ssh2 client failed to authenticate. (or you have too
old ssh2 installed,)
Disconnected; connection lost (Connection closed by remote host.).

I *can* use Reflection FTP in SFTP mode against the same user
on the VMS box. Refletion says (about the server) :

"SSH-2.0-3.20 SSH OpenVMS V5.5 VMS_sftp_version 3"

Anyway, I don't feel I'm of any greater help here... :-)

Jan-Erik.

[Jan-Erik Soderholm]

Simon Clubley wrote 2012-03-13 11:21:

> On 2012-03-13, Jan-Erik Soderholm<jan-erik....@telia.com> wrote:
>>
>> I tried to run a few quick tests on my 8.4 (Alpha) TCPIP 5-7 ECO3 system,
>> but gave up after a number of different errors like "wrong SSH2 version".
>> I never understood what was the problem... :-)
>>
>
> Was this during authentication ?
>
> Regardless of which point this was at, that doesn't sound healthy...
>
> Yet another problem I discovered during my early testing was that the
> VMS sftp client would not switch to password mode after trying and
> failing (as expected) to perform a certificate based authentication.
>
> This was against one specific sftp server, but the Linux sftp client
> had no problem with that server.
>
> In the end, I had to force the use of password based authentication only
> by using the option
>
> sftp -o "AllowedAuthentications password"
>
> on the command line. You may have more luck using the same option.
>
> Simon.
>

In the 5.7 ECO3 releazase notes it says :

4.21.60 Data appears to be truncated on the remote end

Problem:
The SFTP and SCP utilities are not properly 'put'ing fixed
record format files to non-VMS systems. The data appears to
be truncated on the remote end.

Solution:
This problem is corrected in this release.

But I guess you have checked all rellevant relnotes, right?

Jan-Erik.

[Simon Clubley]

On 2012-03-13, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
> Simon Clubley wrote 2012-03-13 11:21:
>> On 2012-03-13, Jan-Erik Soderholm<jan-erik....@telia.com> wrote:
>>>
>>> I tried to run a few quick tests on my 8.4 (Alpha) TCPIP 5-7 ECO3 system,
>>> but gave up after a number of different errors like "wrong SSH2 version".
>>> I never understood what was the problem... :-)
>>>
>>
>> Was this during authentication ?
>>
>> Regardless of which point this was at, that doesn't sound healthy...
>>
>> Yet another problem I discovered during my early testing was that the
>> VMS sftp client would not switch to password mode after trying and
>> failing (as expected) to perform a certificate based authentication.
>>
>> This was against one specific sftp server, but the Linux sftp client
>> had no problem with that server.
>>
>> In the end, I had to force the use of password based authentication only
>> by using the option
>>
>> sftp -o "AllowedAuthentications password"
>>
>> on the command line. You may have more luck using the same option.
>>
>> Simon.
>>
>
> OK.
> Now, I guess it was not the intention to check *my* system... :-)
>

The idea is to test the sftp _client_ image supplied as part of
TCP/IP V5.7 ECO3 against a sftp server which can be reasonably
assumed to be a "known good" sftp server.

> FWIW, here is what I get :
>
> $ sftp -o "AllowedAuthentications password" "<user>@localhost"
>
> warning: Authentication failed.
> FATAL: ssh2 client failed to authenticate. (or you have too
> old ssh2 installed,)
> Disconnected; connection lost (Connection closed by remote host.).
>

Thanks for trying. When I had this problem, with the same error
message, forcing password authentication as the only valid
authentication option worked. If you are interested in finding out
why it doesn't work for you, adding "-D 5" after the sftp command
will reveal _lots_ of debugging information.

> I *can* use Reflection FTP in SFTP mode against the same user
> on the VMS box. Refletion says (about the server) :
>
> "SSH-2.0-3.20 SSH OpenVMS V5.5 VMS_sftp_version 3"
>
> Anyway, I don't feel I'm of any greater help here... :-)
>

I do appreciate you trying.

I've just tried connecting to the TCP/IP VMS sftp server (up until
now all my tests have been with sftp servers on other hosts) and
I am also seeing a loss of connection, but after the password has
been accepted.

Looking in [tcpip$ssh]tcpip$ssh_run.log shows:

/sys$system/tcpip$ssh_sftp-server2: non-translatable vms error code: 0x2A14
%system-f-exbytlm, exceeded byte count quota
%TCPIP-E-SSH_ERROR, non-specific error condition

error, which is _really_ informative. (It would be nice if VMS
output the _actual_ quota been exceeded. :-))

BTW, I've now tried it with Eisner as the remote sftp server and I am
seeing file corruption there as well. Support have come back to me and
confirmed the corruption problem in the Engineering images. No word yet
on if V5.7 ECO3 is affected.

Thanks for having tried,

[Simon Clubley]

I'm working with images which are the most recent images (and are
more recent than 5.7 ECO3), so either a truncation problem has been
re-introduced into these images (which is more than a little
disturbing), or I've found yet another file truncation problem which
still remains to be fixed.

[VAXman-]

In article <jjn54r$vek$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>On 2012-03-12, David Froble <da...@tsoft-inc.com> wrote:
>>
>> I'm still trying to understand the problem. You indicate that you had been using V5.6
>> ECO5. Are you saying the problem exists in this version?
>>
>
>No, the file truncation problem does not exist in the sftp image supplied
>with ECO5, but other different problems do exist, which is why the latest
>Engineering images were loaded onto the system in question.
>
>This file truncation problem is clearly a very recent problem. I am still
>waiting to hear back from HP, so in the meantime I am trying to find out
>if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
>ECO3).

I just transferred my LOGIN.COM file to one of my Ubuntu laptops:

$ sftp -oPort=###### vax...@192.168.#.###

sftp> cd Desktop
/home/vaxman/Desktop
sftp> put login.com
login.com (src): got EOF reading file (server msg: 'Encountered EOF.')
sftp>

On the Ubuntu Laptop, 'gedit' opened the file just fine and all of its
contents were available. If you can give me a couple of "reproducers"
which exhibit your issue, I'll give them a try.

I've been using 'sftp' extensively to transfer files to and from my VMS
systems to my Linux systems and I've never seen any problems exhibited.
I'm not saying there aren't any issues, just that my files just haven't
exhibited problems that yours have.

[Simon Clubley]

On 2012-03-13, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
> In article <jjn54r$vek$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>On 2012-03-12, David Froble <da...@tsoft-inc.com> wrote:
>>>
>>> I'm still trying to understand the problem. You indicate that you had been using V5.6
>>> ECO5. Are you saying the problem exists in this version?
>>>
>>
>>No, the file truncation problem does not exist in the sftp image supplied
>>with ECO5, but other different problems do exist, which is why the latest
>>Engineering images were loaded onto the system in question.
>>
>>This file truncation problem is clearly a very recent problem. I am still
>>waiting to hear back from HP, so in the meantime I am trying to find out
>>if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
>>ECO3).
>
> I just transferred my LOGIN.COM file to one of my Ubuntu laptops:
>

Thanks for doing this. Just for confirmation, was this V5.7, ECO3 ?

> $ sftp -oPort=###### vax...@192.168.#.###
>
> sftp> cd Desktop
> /home/vaxman/Desktop
> sftp> put login.com
> login.com (src): got EOF reading file (server msg: 'Encountered EOF.')
> sftp>
>
> On the Ubuntu Laptop, 'gedit' opened the file just fine and all of its
> contents were available. If you can give me a couple of "reproducers"
> which exhibit your issue, I'll give them a try.
>
> I've been using 'sftp' extensively to transfer files to and from my VMS
> systems to my Linux systems and I've never seen any problems exhibited.
> I'm not saying there aren't any issues, just that my files just haven't
> exhibited problems that yours have.
>

I've just created a simple file which reproduces the problem for me:

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
LLLLLLLLLLLLLLLLLLLLLLLL
MMMMM

When transferred to a Linux sftp server, this is what is received:

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
LLLLLLLLLLLLLLLLLL

Note the M's and the last few L's are missing.

[Simon Clubley]

On 2012-03-12, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>

> I'm trying to find out if someone else can get this to fail using the
> latest published V5.7 ECOs or if it's just a unreleased TCP/IP Engineering
> image problem (or something that triggers under very specific
> circumstances only).
>

I've just had a response back from support.

While they can duplicate the problem in the Engineering images, they
cannot duplicate the problem with the test input file in TCP/IP V5.7 ECO 3.

Therefore, this specific problem looks like it's a just a problem in
the TCP/IP Engineering image. However, I would suggest you still carry
out your own tests on any ECO changes just to be on the safe side in
case your specific files trigger any other similar bugs.

Thanks to everyone for the comments and help you offered.

[David Froble]

Simon Clubley wrote:
> On 2012-03-13, Jan-Erik Soderholm <jan-erik....@telia.com> wrote:
>> In the 5.7 ECO3 releazase notes it says :
>>
>> 4.21.60 Data appears to be truncated on the remote end
>>
>> Problem:
>> The SFTP and SCP utilities are not properly 'put'ing fixed
>> record format files to non-VMS systems. The data appears to
>> be truncated on the remote end.
>>
>> Solution:
>> This problem is corrected in this release.
>>
>> But I guess you have checked all rellevant relnotes, right?
>>
>
> I'm working with images which are the most recent images (and are
> more recent than 5.7 ECO3), so either a truncation problem has been
> re-introduced into these images (which is more than a little
> disturbing), or I've found yet another file truncation problem which
> still remains to be fixed.
>
> Simon.
>

Might be nice, if the people in India learned how to use RMS ??

But what the hell, you didn't need those few bytes at the end of the file ....

[VAXman-]

In article <jjnc5c$1l5$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>On 2012-03-13, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
>> In article <jjn54r$vek$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>>On 2012-03-12, David Froble <da...@tsoft-inc.com> wrote:
>>>>
>>>> I'm still trying to understand the problem. You indicate that you had been using V5.6
>>>> ECO5. Are you saying the problem exists in this version?
>>>>
>>>
>>>No, the file truncation problem does not exist in the sftp image supplied
>>>with ECO5, but other different problems do exist, which is why the latest
>>>Engineering images were loaded onto the system in question.
>>>
>>>This file truncation problem is clearly a very recent problem. I am still
>>>waiting to hear back from HP, so in the meantime I am trying to find out
>>>if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
>>>ECO3).
>>
>> I just transferred my LOGIN.COM file to one of my Ubuntu laptops:
>>
>
>Thanks for doing this. Just for confirmation, was this V5.7, ECO3 ?

Yes.

$ CREATE SIMON.TEST

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
LLLLLLLLLLLLLLLLLLLLLLLL

MMMMM[ Exit ]
$ sftp -oPort=###### vax...@192.168.#.###
sftp> put simon.test

I see everything when I open the file with 'gedit'.

[Simon Clubley]

On 2012-03-13, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
> In article <jjnc5c$1l5$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>
>>Thanks for doing this. Just for confirmation, was this V5.7, ECO3 ?
>
> Yes.
>

[snip]

>
> $ CREATE SIMON.TEST
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
> CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
> DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
> EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
> FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
> GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
> IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
> JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
> KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
> LLLLLLLLLLLLLLLLLLLLLLLL
> MMMMM[ Exit ]
> $ sftp -oPort=###### vax...@192.168.#.###
> sftp> put simon.test
>
> I see everything when I open the file with 'gedit'.
>

Thank you for running this test Brian.

This is consistant with the reports I have received from support over
the last few hours that they cannot reproduce it in V5.7 ECO 3 but can
only reproduce it in the Engineering images they sent me.

On the plus side, this means the problem does not appear to be in
any public ECO kit. On the down side, it looks like the Engineering
images have a nice new file truncation bug in them...

(I'm assuming here that once they find a bug in one supported version,
they still check for the same bug in all supported versions.)

Thanks once again,

[Simon Clubley]

I should expand on that last statement since it's clearly open to
misunderstanding; it was just something which occurred to me while posting.

Although, as mentioned, I am running TCP/IP V5.6, and hence the image id
in the engineering image HP supplied has a V5.6 id, the image HP supplied
has been very clearly built from the latest code base because of all the
enhanced functionality in it.

I realised I've been implicitly assuming that HP are fixing common mode
problems in all supported versions at the same time just like DEC have
always done. However it also occured to me, that due to the types of
procedural problems we have seen recently, this may not be the case.

Hence it suddenly occured to me it was possible this was a bug fixed on
the V5.7 stream which didn't get moved into the V5.6 stream even though
the V5.6 stream is clearly using code from the latest code base; therefore
this might be a known bug which didn't get fixed when it should have been.

However, thinking about it, it's far more likely that this is a new
problem which would have also affected the next V5.7 ECO as well if
it wasn't fixed; I have not seen any evidence common mode problems are
not getting fixed across all versions once they are identified.

Sorry for the confusion,

[VAXman-]

In article <jjo8os$fo3$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>On 2012-03-13, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:

>I should expand on that last statement since it's clearly open to
>misunderstanding; it was just something which occurred to me while posting.
>
>Although, as mentioned, I am running TCP/IP V5.6, and hence the image id
>in the engineering image HP supplied has a V5.6 id, the image HP supplied
>has been very clearly built from the latest code base because of all the
>enhanced functionality in it.
>
>I realised I've been implicitly assuming that HP are fixing common mode
>problems in all supported versions at the same time just like DEC have
>always done. However it also occured to me, that due to the types of
>procedural problems we have seen recently, this may not be the case.
>
>Hence it suddenly occured to me it was possible this was a bug fixed on
>the V5.7 stream which didn't get moved into the V5.6 stream even though
>the V5.6 stream is clearly using code from the latest code base; therefore
>this might be a known bug which didn't get fixed when it should have been.
>
>However, thinking about it, it's far more likely that this is a new
>problem which would have also affected the next V5.7 ECO as well if
>it wasn't fixed; I have not seen any evidence common mode problems are
>not getting fixed across all versions once they are identified.
>
>Sorry for the confusion,
>
>Simon.

I wish you'd stop tying them up with this issue so that they can address
my REMOTE access requirement 'ssh' issue. :)

[VAXman-]

In article <00ABE4A1...@SendSpamHere.ORG>, VAXman- @SendSpamHere.ORG writes:

>In article <jjnc5c$1l5$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>On 2012-03-13, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
>>> In article <jjn54r$vek$1...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>>>On 2012-03-12, David Froble <da...@tsoft-inc.com> wrote:
>>>>>
>>>>> I'm still trying to understand the problem. You indicate that you had been using V5.6
>>>>> ECO5. Are you saying the problem exists in this version?
>>>>>
>>>>
>>>>No, the file truncation problem does not exist in the sftp image supplied
>>>>with ECO5, but other different problems do exist, which is why the latest
>>>>Engineering images were loaded onto the system in question.
>>>>
>>>>This file truncation problem is clearly a very recent problem. I am still
>>>>waiting to hear back from HP, so in the meantime I am trying to find out
>>>>if anyone else can duplicate this using the most recent TCP/IP ECO (V5.7
>>>>ECO3).
>>>
>>> I just transferred my LOGIN.COM file to one of my Ubuntu laptops:
>>>
>>
>>Thanks for doing this. Just for confirmation, was this V5.7, ECO3 ?
>

>Yes.

>$ CREATE SIMON.TEST

>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
>CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
>DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
>EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
>FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
>GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
>HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
>IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
>JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
>KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>LLLLLLLLLLLLLLLLLLLLLLLL

>MMMMM[ Exit ]
>$ sftp -oPort=###### vax...@192.168.#.###

>sftp> put simon.test
>
>I see everything when I open the file with 'gedit'.

FWIW, I tried this:

$ CONVERT/FDL="Record; Format VFC" SYS$INPUT SIMON.TEST

And supplied the same data as before. It would appear that the sftp client
on VMS does not handle the VFC and sends it to the remote. 'gedit' will not
open it; more simon.test returns blank lines; and WINE NotePad will display
all of the contents. 'sftp' is not reading the VFC file's VFC; it's simply
sending it to the remote as data.

[Simon Clubley]

On 2012-03-13, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
>
> I wish you'd stop tying them up with this issue so that they can address
> my REMOTE access requirement 'ssh' issue. :)

:-) :-)

I've done all the testing I can do for now until they have another go
at providing a solution, so it's your turn to bother them about whatever
your latest problem is. :-)

[Simon Clubley]

Ouch, that's bad. It's also something I didn't try as I have not needed
to transfer VFC files.

Thanks for the heads up.

[JF Mezei]

Simon Clubley wrote:

> In Unix-land, which is what OS-X basically is, the convention is just to
> send the file as a stream of bytes without any internal conversion.

The FTP protocol was built to support inter system txt excahnges by
having a common record delimiter used during file transfers for "ascii
mode" transfers and have the sender convert their native format to the
common one and the receiver do the same.

This is why I am somewhat susprised that sftp doesn't have any of that.

[JF Mezei]

I've found a solution to your problem of sending your login.com :

It is a small procedure:

$OPEN/APPEND output login.com
$X = 0
$LOOP:
$ write output "$!"
$ x = x + 1
$ if X < 100 then goto LOOP
$!
$!
$CLOSE login.com


By adding a sufficiently large number of comment lines, the sftp will
truncate only comments near the end of file, so all of the meat in your
login.com will get sent well before sftp decides it is tired and should
stop sending before end of file.


At the end of the day, if they have proper change conrol and source code
management, they should be able to see what chances were made between a
working copy and the one which fails and send this to the 5 or 6 people
who know VMS in the group to learn why the new copy fails.

[Bob Koehler]

In article <4f5fb97f$0$9212$c3e8da3$12bc...@news.astraweb.com>, JF Mezei <jfmezei...@vaxination.ca> writes:
>
> This is why I am somewhat susprised that sftp doesn't have any of that.

sftp was born and bread in a land of all byte-stream files.

I used to work with a programmer who would run around claiming
"binary always works" for FTP transfers. He was used to using
dos2unix and unix2dos after transfering files.

He also made that claim while we were instructing a user on how
to use Multinet FTP features to transfer a file from UNIX on to
a VMS system resulting in a fixed record format file. Ha.

[glen herrmannsfeldt]

Bob Koehler <koe...@eisner.nospam.encompasserve.org> wrote:
> In article <4f5fb97f$0$9212$c3e8da3$12bc...@news.astraweb.com>, JF Mezei <jfmezei...@vaxination.ca> writes:
>>
>> This is why I am somewhat susprised that sftp doesn't have any of that.

> sftp was born and bread in a land of all byte-stream files.

More specifically, ftp knows about 36 bit systems. At least
it used to have the options needed.

-- glen

[Henry Crun]

also:
s/bread/bred/ -- look it up

--
Mike R.
Home: http://alpha.mike-r.com/
QOTD: http://alpha.mike-r.com/php/qotd.php
No Micro$oft products were used in the URLs above, or in preparing this message.
Recommended reading: http://www.catb.org/~esr/faqs/smart-questions.html#before

[VAXman-]

In article <d5se39-...@Ubuntu.mike-r.com>, Henry Crun <mi...@rechtman.com> writes:
>On 15/03/12 20:58, glen herrmannsfeldt wrote:

>also:
>s/bread/bred/ -- look it up
>
>--
>Mike R.
>Home: http://alpha.mike-r.com/
>QOTD: http://alpha.mike-r.com/php/qotd.php
>No Micro$oft products were used in the URLs above, or in preparing this message.
>Recommended reading: http://www.catb.org/~esr/faqs/smart-questions.html#before

I love your .sig.

[Paul Sture]

On Sat, 17 Mar 2012 20:39:44 +0000, VAXman- wrote:

> In article <d5se39-...@Ubuntu.mike-r.com>, Henry Crun
> <mi...@rechtman.com> writes:
>>On 15/03/12 20:58, glen herrmannsfeldt wrote:
>>> Bob Koehler<koe...@eisner.nospam.encompasserve.org> wrote:
>>>> In article<4f5fb97f$0$9212$c3e8da3$12bc...@news.astraweb.com>, JF
>>>> Mezei<jfmezei...@vaxination.ca> writes:
>>>>>
>>>>> This is why I am somewhat susprised that sftp doesn't have any of
>>>>> that.
>>>
>>>> sftp was born and bread in a land of all byte-stream files.
>>>
>>> More specifically, ftp knows about 36 bit systems. At least it used to
>>> have the options needed.
>>>
>>> -- glen
>>also:
>>s/bread/bred/ -- look it up
>>
>>--
>>Mike R.
>>Home: http://alpha.mike-r.com/
>>QOTD: http://alpha.mike-r.com/php/qotd.php No Micro$oft products were
>>used in the URLs above, or in preparing this message. Recommended
>>reading: http://www.catb.org/~esr/faqs/smart-questions.html#before
>
> I love your .sig.

:-)

But do you know who Henry Crun is?

https://en.wikipedia.org/wiki/Henry_Crun_and_Minnie_Bannister



--
Paul Sture

[Henry Crun]

I remember bunking school and roaring with laughter in front of a radio¹ in the
late 50's -- early 60's

¹) Like a television, except you don't have to watch a picture.

[Paul Sture]

On Sun, 18 Mar 2012 17:40:52 +0200, Henry Crun wrote:

>>
>> But do you know who Henry Crun is?
>>
>> https://en.wikipedia.org/wiki/Henry_Crun_and_Minnie_Bannister
>>
>>
> I remember bunking school and roaring with laughter in front of a radio¹
> in the late 50's -- early 60's
>
> ¹) Like a television, except you don't have to watch a picture.

I missed the Goons the first time around and only managed to catch some
of the repeats they did in the mid-seventies, but thoroughly enjoyed
those. Spike Milligan went on to do several TV series which were much in
the same vein.

--
Paul Sture

[Phillip Helbig---undress to reply]

In article <jjl3pf$6uf$1...@dont-email.me>, Simon Clubley
<clubley@remove_me.eisner.decus.org-Earth.UFP> writes:

> I am experiencing a file truncation problem with a TCP/IP engineering
> image version of the TCP/IP Services sftp client and I am wondering
> if anyone here can reproduce it.
>
> Try using the sftp client to send something small (so that it's easy to
> check; I am using my login.com) to a Linux sftp server.
>
> When I do this (both in "ascii dos" mode and binary mode) the last
> few dozen bytes of my login.com are lost during transfer.

I've seen this as well, but with SCP. There is no point in worrying
about it unless it is also present in the latest TCPIP version (unless
you have support for an older version and want to stick with it).