FTP buffer overrun

[Serge Girard]

Hello,

When uploading pictures sometimes a "Buffer overrun error" occurs (from oFtp:lastErrorMessage() )

Then I found this:

http://help.cognex.com/Content/KB_Topics/DataMan/SetupTool/4005.htm

Suggesting this:

This error indicates that an image was taken, but could not be placed in memory due to a lack of space for it. FTP and Image Processing share image buffers in the reader's memory, and Single or Burst trigger modes require some additional buffers to be available for Image Processing.
You may need to decrease the maximum number of buffers available for FTP image queuing when using these trigger modes. If you continue to get the Buffer Overflow errors after decreasing the number of buffers allocated to FTP, then it is likely that you are triggering the DataMan faster than the images that it takes can be processed

Then I found this piece of code:
::nDefaultSndBuffSize := 65536

Is this parameter to be decreased or what?

Thx, Serge

[Klas Engwall]

Hi Serge,

I suppose you are talking about TIPClientFTP?

You say that oFtp:lastErrorMessage() returns that error. To me that
sounds like the buffer for the server response is too small for whatever
the server says in response to your upload. It is a known problem that
the GetReply() method sets up a too small buffer (only 128 bytes) and
that some servers talk a lot and easily fill that buffer - especially
when the file names are long - and then the buffer overflows. In that
case, the solution is to increase the buffer size in GetReply().

So, is logging active in the client? What does the log say?

Are you uploading to your own server? If so, what does the server log say?

Regards,
Klas

[Serge Girard]

Hi Klas,

Thank you very much for replying. Yes, indeed: TIPClientFTP. It's not my own server but the one that belongs to my site I'm uploading to.

I really don't have a clue how to put the logging on or off. Found StartCleanLogFile() ...So please show me how to log!

Anyway I changed 128 bytes in the GetReply() to 256 and this seems to work.

Serge

Op woensdag 2 september 2020 om 19:12:30 UTC+2 schreef Klas Engwall:

[Klas Engwall]

Hi Serge,

> Thank you very much for replying. Yes, indeed: TIPClientFTP. It's not my
> own server but the one that belongs to my site I'm uploading to.
>
> I really don't have a clue how to put the logging on or off. Found
> StartCleanLogFile() ...So please show me how to log!

Look at the New() method in the FTP client:
METHOD New( oUrl, xTrace, oCredentials ) CLASS TIPClientFTP

<xTrace> defaults to .F., so pass .T., and logging will be turned on
with a standard log filename. That is the quickest solution for
temporary use. Or, as a more advanced alternative with your own log
filename, you can setup the TIPLog class separately and then pass the
log object in a code block in <xTrace>.

There is a discussion about the TIPLog class and a subclass with more
advanced options (that was in an SMTP context, but logging is identical)
in this thread:
https://groups.google.com/forum/#!searchin/harbour-users/tiplog%7Csort:date/harbour-users/K-gT12M3jtg/w1i1AQU-AwAJ

> Anyway I changed 128 bytes in the GetReply() to 256 and this seems to work.

That's good. Now, with logging turned on, try to upload one of those
files that failed before, and check the size of the server response in
the log. I am pretty sure it will be longer than 128 bytes.

Regards,
Klas

[Serge Girard]

Thanks! The logging says "INETERRORCODE -3" with 128 and with 256 "INETERRORCODE 0" . Ik you want I sent the logging (4kb) of 128 bytes version.
Serge

Op donderdag 3 september 2020 om 10:48:02 UTC+2 schreef Klas Engwall:

[Klas Engwall]

Hi Serge,

> Thanks! The logging says "INETERRORCODE -3" with 128 and with 256
> "INETERRORCODE 0" . Ik you want I sent the logging (4kb) of 128 bytes
> version.

-3 means HB_INET_ERR_BUFFOVERRUN and 0 means that there is no error.
What I was referring to in the log was the log line beginning with the
150 response code *after* increasing the buffer size. That is where the
server says what it is doing. "150 Opening BINARY mode data connection"
or similar is the standard response, but the server you use apparently
has a lot more to say.

Now we know that the error code has been eliminated with the larger
buffer, and that is what we really wanted to know. Reading the "150"
message is more for you to see what caused the overflow.

Regards,
Klas

[Serge Girard]

Thanks Klas ! Great help!!

Serge

Op donderdag 3 september 2020 om 12:33:13 UTC+2 schreef Klas Engwall:

[Serge Girard]

Klas, Can I send you a logfile?

Serge

Op donderdag 3 september 2020 om 13:27:05 UTC+2 schreef Serge Girard:

[Klas Engwall]

Hi Serge,

I thought we had solved this back in September. Are you still using the
version with the 128 byte receive buffer? Or has the problem returned
with the increased buffer? It could be that the server has even more to
say than can fit in 256 bytes. So increase it to 1024 and see what happens.

As long as you get a -3 response in the error code you do have a too
small buffer. There is no cost in increasing it well beyond whatever the
server's message size is.

Besides, a simple test you can perform is to use a standard FTP client
like FileZilla, for example, to upload one of those files and observe
and measure the message the server returns.

Regards,
Klas

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Harbour Users" group.
> Unsubscribe: harbour-user...@googlegroups.com
> Web: http://groups.google.com/group/harbour-users
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Harbour Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to harbour-user...@googlegroups.com
> <mailto:harbour-user...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/harbour-users/8de5ccc0-ea6e-4158-8333-c1806dfdc273n%40googlegroups.com
> <https://groups.google.com/d/msgid/harbour-users/8de5ccc0-ea6e-4158-8333-c1806dfdc273n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Serge Girard]

Hi Klas,

The previous problem is solved but this occurs in a totally different application. I used the FTPCLI.PRG as before. Even changed to 1024... it hangs up.

I send a small piece of the logging

20210202-16:01:45  :INETSENDALL( <pointer>, 63, STOR /domains/XXXXXXXXXXXXXXX.com/public_html/ZZZZZZZZZZZ.php<cr><lf> )

>>         63 <<

20210202-16:01:46  :INETERRORCODE( <pointer> )

>>          0 <<

20210202-16:01:46  :INETRECVLINE( <pointer>, , 1024 )

>> 150 Opening BINARY mode data connection for /domains/XXXXXXXXXXXXXXX.com/public_html/ZZZZZZZZZZZ.php <<

20210202-16:01:46  :INETERRORCODE( <pointer> )

>>          0 <<

20210202-16:01:46  :INETSENDALL( <pointer>, 6, QUIT<cr><lf> )

>>          6 <<

20210202-16:02:26  :INETRECVLINE( <pointer>, , 1024 )

>>  <<

20210202-16:02:26  :INETERRORCODE( <pointer> )

>>         -1 <<

Hope this is enough for you!

Many thanks in advance , Serge

Op dinsdag 2 februari 2021 om 18:37:19 UTC+1 schreef Klas Engwall:

[Klas Engwall]

Hi Serge,

> The previous problem is solved but this occurs in a totally different
> application. I used the FTPCLI.PRG as before. Even changed to 1024... it
> hangs up.
> I send a small piece of the logging
>
> 20210202-16:01:45  :INETSENDALL( <pointer>, 63, STOR
> /domains/XXXXXXXXXXXXXXX.com/public_html/ZZZZZZZZZZZ.php<cr><lf> )
> >>         63 <<
> 20210202-16:01:46  :INETERRORCODE( <pointer> )
> >>          0 <<
> 20210202-16:01:46  :INETRECVLINE( <pointer>, , 1024 )
> >> 150 Opening BINARY mode data connection for
> /domains/XXXXXXXXXXXXXXX.com/public_html/ZZZZZZZZZZZ.php <<
> 20210202-16:01:46  :INETERRORCODE( <pointer> )
> >>          0 <<
> 20210202-16:01:46  :INETSENDALL( <pointer>, 6, QUIT<cr><lf> )
> >>          6 <<
> 20210202-16:02:26  :INETRECVLINE( <pointer>, , 1024 )
> >>  <<
> 20210202-16:02:26  :INETERRORCODE( <pointer> )
> >>         -1 <<

This is an entirely different problem. You are getting a proper 150
message, which by the way is 103 bytes in this case, and the error code
at that point is 0. So far so good.

Then you send a QUIT without first uploading anything, and there is a 40
second timeout before you ask the server for a status message again, and
then it returns nothing and an error code meaning HB_INET_ERR_TIMEOUT at
the end.

How do you upload the file? UploadFile()? Write() from a memory
variable? Have you checked that the content exists? Before the QUIT
message there should be an ...

INETSENDALL( <pointer>, size_of_the_file, content_of_the_file )

... following the same pattern as the other INETSENDALL log lines, but
there is no trace of any such message, or even an attempt to send such a
message, in the log.

Regards,
Klas

[Serge Girard]

You are so right: a type in file-pathname .... I now check before upload with if file(..)

Sorry to have wasted your time! Mea culpa, maxima!

Serge

Op woensdag 3 februari 2021 om 00:26:19 UTC+1 schreef Klas Engwall:

[Klas Engwall]

Hi Serge

> You are so right: a type in file-pathname .... I now check before upload
> with if file(..)
> Sorry to have wasted your time! Mea culpa, maxima!

No problem :-)

Here is a thought: I suspect that you use the UploadFile() method. When
you do that you outsource locating, reading and transmitting the file
"wholesale" to that method and only see the result when you check the log.

I don't think I have ever used UploadFile(). Instead I check with
hb_fileexists(), then check the file size, read it with hb_memoread(),
check the size of the memory variable and then use the Write() method to
write from memory when I am satisfied that I have the entire file in the
variable. Belt and suspenders, you know :-). It either works everytime
or fails early in my tests so I can correct it.

Regards,
Klas