Indy FTP Response Codes for NLST
Erik Berry
When I'm in an empty directory on the server, I call TIdFTP.List(StrLst, '',
False) it gets into TIdFTP.InternalGet with the NLST command and calls this:
SendCmd(ACommand, [125, 150, 154]); //APR: Ericsson Switch FTP);
My FTP server returns a response 226 with a text of "Transfer Complete."
Because the code 226 is not in the array of acceptable responses above, it
raises an exception from TIdReplyRFC.RaiseReplyError eventually.
The Linux server is "Linux <hostname>
2.4.29-grsec+w+fhs6b+gr0501+nfs+a32+++p4+sata+c4+gr2b-v6.189 #1 SMP Mon Feb
7 13:23:30 PST 2005 i686 GNU/Linux"
ftp> QUOTE SYST
215 UNIX Type: L8
If the same NLST command is called in a non-empty directory, things seem to
work fine. It seems the latest official release (Indy 10.0.52) acts the same.
It looks like Indy is interpreting the Transfer Complete. result as part of
the directory listing, such that even if I allow the 226 code to pass
through, it hangs on the next LPortSv.Listen call waiting for the 226 it
already consumed.
I haven't had time to look deeper than this, but I thought I'd go ahead and
report this in case I don't get back to it.
The amusing side note is that a user viewing this exception sees an error
that states "Transfer complete.", even the operation fails. :)
Erik
Remy Lebeau (TeamB)
"Erik Berry" <e...@techie.ZZZcom> wrote in message
news:45d2db61$1...@newsgroups.borland.com...
> My FTP server returns a response 226 with a text of "Transfer
Complete."
Then your server is very buggy. 226 must not be the first response
code that is sent in reply to the NLST command. It must send 125 or
150 first. 125 and 150 establish the presence of a valid data channel
connection, and then 26 indicates the success of the transfer over
that connection. RFC 959 is very clear on this matter.
> Because the code 226 is not in the array of acceptable responses
above,
> it raises an exception from TIdReplyRFC.RaiseReplyError eventually.
As it should be, because the server is not following the established
standard in the first place.
Gambit
Erik Berry
> Then your server is very buggy. 226 must not be the first response
> code that is sent in reply to the NLST command. It must send 125 or
> 150 first. 125 and 150 establish the presence of a valid data channel
> connection, and then 26 indicates the success of the transfer over
> that connection. RFC 959 is very clear on this matter.
It returns the 226 to the specific SendCmd function call I mentioned (see
IdFTP.pas line 1692). The server does this for an empty dir:
ftp> dir
200 PORT command successful
150 Opening ASCII mode data connection for file list
226 Transfer complete.
With a non-empty directory, it returns this:
ftp> dir
200 PORT command successful
150 Opening ASCII mode data connection for file list
drwxr-xr-x 2 username groupname 4096 Feb 14 03:56 FooDir
226 Transfer complete.
Erik
Remy Lebeau (TeamB)
"Erik Berry" <e...@techie.ZZZcom> wrote in message
> The server does this for an empty dir:
You are contradicting yourself. What you describe now is not what you
described yesterday. The command sequence you have shown now is
indeed sending a 150 response before sending a 226 response. What you
described yesterday can only occur if that 150 were not being sent.
Given this new information, the SendCmd() in question is guaranteed to
receive the 150 in both situations, not the 226. There is no possible
way that the SendCmd() could be receiving the 226 prematurely.
In both situations, a valid data connection is being created, a 150
response is sent back from the server to confirm that, and then a 226
response is sent back when the transfer is finished. That is how FTP
is supposed to work. The amount of data being transferred over the
data connection is irrelevant. 0 bytes is perfectly acceptable, and
TIdFTP handles that fine. What you are showing is just the
consolidated output of the ftp.exe application on the DOS console
window. But there are two distinct socket connections running in the
background, pushing their respective data to the console window's
output.
Gambit
Erik Berry
> described yesterday. The command sequence you have shown now is
> indeed sending a 150 response before sending a 226 response. What you
> described yesterday can only occur if that 150 were not being sent.
> Given this new information, the SendCmd() in question is guaranteed to
> receive the 150 in both situations, not the 226. There is no possible
> way that the SendCmd() could be receiving the 226 prematurely.
I tried to provide as much information as I could given my understanding of
the FTP protocol. I'm not surprised to learn that the information was not
as good as an Indy expert would be able to provide. Maybe this result from
an empty directory using the ls command line FTP command instead of dir is
more helpful:
ftp> ls
200 PORT command successful
226 Transfer complete.
With a subdirectory present in the same directory it reports this:
ftp> ls
200 PORT command successful
150 Opening ASCII mode data connection for file list
FooDir
226 Transfer complete.
The server is "ProFTPD 1.3.0rc2 Server", and it appears not to send a 150
response to ls/NLST requests in empty directories. This may well violate
the RFC (another FTP server I use does send the 150 response in empty dirs)
and isn't worth working around. If so, it may instead be worth a comment in
the source to assist others that try to connect to similar servers.
Erik
Remy Lebeau (TeamB)
"Erik Berry" <e...@techie.ZZZcom> wrote in message
> I tried to provide as much information as I could given my
understanding
> of the FTP protocol. I'm not surprised to learn that the
information was
> not as good as an Indy expert would be able to provide. Maybe this
result
> from an empty directory using the ls command line FTP command
instead
> of dir is more helpful:
You need to stop relying on the output of the command-line FTP
program. That is not useful enough to diagnose this problem, as it
does not show everything that is happening. You should use a real
packet sniffer, such as Ethereal (http://www.ethereal.com), to see the
actual FTP packet traffic back and forth.
Also, just an FYI, this issue is server-specific as well. For
example, on the FTP server I use (Serv-U), when I retreive a directory
listing on an empty folder, it returns a "550 No files found" response
to the NLST command instead of trying to transfer empty data.
> The server is "ProFTPD 1.3.0rc2 Server", and it appears not to send
> a 150 response to ls/NLST requests in empty directories.
Yes, it does (or you would not be getting the 226 reply). The console
output simply does not show it. That is why you need to use a better
viewer of the FTP commands/replies.
Gambit
Erik Berry
>> it appears not to send
>> a 150 response to ls/NLST requests in empty directories.
>
> Yes, it does (or you would not be getting the 226 reply). The console
> output simply does not show it. That is why you need to use a better
> viewer of the FTP commands/replies.
I've verified with the vendor that the 150 response is not sent by some
older versions of the server, though they claim to have fixed it:
http://bugs.proftpd.org/show_bug.cgi?id=2229
I also debugged into the Indy code, and the server does not send a 150 in
response to an NLST request when in empty dirs the same as it does for
non-empty dirs, and this is what causes causes Indy to hang. I was only
posting the ls output (which I know hides many things) to show this
difference in quick summary. Since the summary isn't convincing, here is a
full log I generated yesterday of the send/recv strings from
TIdTCPConnection.SendCmd and TIdTCPConnection.GetInternalResponse:
...
AOut: 'FEAT'
LLine: '211-Features:'
LLine: 'MDTM'
LLine: 'REST STREAM'
LLine: 'SIZE'
LLine: '211 End'
AOut: 'TYPE I'
LLine: '200 Type set to I'
AOut: 'SYST'
LLine: '215 UNIX Type: L8'
AOut: 'PWD'
LLine: '257 "/" is current directory.'
AOut: 'TYPE A'
LLine: '200 Type set to A'
AOut: 'PORT 192,168,1,131,9,0'
LLine: '200 PORT command successful'
AOut: 'NLST' <note - in a non-empty directory, succeeds>
LLine: '150 Opening ASCII mode data connection for file list'
LLine: '226 Transfer complete.'
AOut: 'TYPE I'
LLine: '200 Type set to I'
AOut: 'CWD XXX' <into empty dir>
LLine: '250 CWD command successful'
AOut: 'TYPE A'
LLine: '200 Type set to A'
AOut: 'PORT 192,168,1,131,9,1'
LLine: '200 PORT command successful'
AOut: 'NLST' <note - in empty directory, fails, no 150 response>
LLine: '226 Transfer complete.'
Then this is raised:
EIdReplyRFCError "Transfer complete." with this call stack:
TIdReplyRFC.RaiseReplyError
TIdTCPConnection.RaiseExceptionForLastCmdResult
TIdTCPConnection.CheckResponse(226,...)
TIdTCPConnection.GetResponse
TIdTCPConnection.SendCmd('NLST',...)
TIdFTP.InternalGet('NLST',...,False)
TIdFTP.List(...,'',False)
Finally, the last line of FinalizeDataOperation in TIdFTP.InternalGet hangs
for several minutes until it times out.
If a workaround in Indy isn't practical, I'd suggest commenting on this in
the Indy code to help others with this server. I'm no longer affected by
this issue, and can't spend more time on it.
Good luck,
Erik
Erik Berry
>> and it appears not to send
>> a 150 response to ls/NLST requests in empty directories.
>
> Yes, it does (or you would not be getting the 226 reply). The console
> output simply does not show it. That is why you need to use a better
> viewer of the FTP commands/replies.
I looked at the vendor bug database, and they have had similar bugs (I
noticed later that this isn't the same bug I found, but this fix may be the
cause of the bug I see):
http://bugs.proftpd.org/show_bug.cgi?id=2229
I debugged into the Indy code, and the server does not appear to send send a
150 in response to an NLST request when in empty dirs the same as it does
for non-empty dirs, and this is what causes causes Indy to hang. I was only
posting the ls output (which I know hides many things) to show this
difference in quick summary. The summary info wasn't convincing, so here is
If a workaround in Indy isn't practical, I'd suggest the Indy team add a
comment on this in the source to help others with this server. I'm no
longer affected by this issue, and won't be able to spend more time on it.
Good luck,
Erik
Erik Berry
>> a 150 response to ls/NLST requests in empty directories.
>
> Yes, it does (or you would not be getting the 226 reply). The console
> output simply does not show it. That is why you need to use a better
> viewer of the FTP commands/replies.
I just got an email and it seems the vendor found the missing 150 response
bug, and fixed it in a later version:
http://bugs.proftpd.org/show_bug.cgi?id=2843
They were not opening a data channel, so there was no 150 response before
the 226 arrived, which matches my command/response log output and the ls
output I posted yesterday (but not dir output I incorrectly quoted).
Erik
Remy Lebeau (TeamB)
"Erik Berry" <e...@techie.ZZZcom> wrote in message
> I looked at the vendor bug database, and they have had similar bugs
(I
> noticed later that this isn't the same bug I found, but this fix may
be the
> cause of the bug I see):
> http://bugs.proftpd.org/show_bug.cgi?id=2229
Technically speaking, having LIST and NLST return a 450 error when
listing an empty directory is allowed by RFC 959.
> I debugged into the Indy code, and the server does not appear to
send
> a 150 in response to an NLST request when in empty dirs the same as
> it does for non-empty dirs, and this is what causes causes Indy to
hang.
It will not cause a hang. If the server sends anything other than
125, 150, or 154, an exception is raised. So what EXACTLY is the
server really sending? Again, please provide details about the actual
FTP traffic that is being exchanged. That is why I told you to use a
packet sniffer.
> here is a full log I generated yesterday of the send/recv strings
Rather than trying to get that data manually, you could simply attach
a TIdLog... component to TIdFTP.
In any case,
> PORT 192,168,1,131,9,1
> 200 PORT command successful
> NLST
> 226 Transfer complete.
If that is all the server is actually sending, then the server really
is faulty to begin with, and needs to be fixed.
> Then this is raised:
As it should be. The server sent the wrong reply. There is nothing
Indy can do about that, other than to raise an exception.
> Finally, the last line of FinalizeDataOperation in
TIdFTP.InternalGet hangs
> for several minutes until it times out.
As it should be. The server is sending replies in the wrong order, so
it is going to confuse Indy and make it try to read more data than
will actually be sent..
Gambit
Erik Berry
> It will not cause a hang. If the server sends anything other than
If you don't consider a several minute pause a hang, then, no, it does not
hang. That said, I described a several minute pause a few lines down.
> It will not cause a hang. If the server sends anything other than
> 125, 150, or 154, an exception is raised. So what EXACTLY is the
> server really sending? Again, please provide details about the actual
> FTP traffic that is being exchanged. That is why I told you to use a
I've documented the exact responses in the low-level log I provided and also
there is a detailed log of the server's communication direct from the
internal server logging in the link to the bug fix I posted earlier today.
I can't see any value in spending more time documenting this here, since it
is a well-documented and now fixed bug in the server, confirmed by at least
two other people, including the server's main author.
> As it should be. The server sent the wrong reply. There is nothing
> Indy can do about that, other than to raise an exception.
Indy could detect this buggy server and try to work around this bug, but as
I said, it may not be worth it, since it may be a very rare case.
Anyway, you and I don't seem to communicate very well (and never really have
in the past), so I'll leave things as-is and say goodbye now, before I
further clutter this newsgroup.
Erik