Problem with Indy Write(Stream)
Stefan Meisner
I do have a problem sending the content of a Stream using Indy 10.
At first I was using
Connection.IOHandler.Write(AStream);
Using this method, the client at some point did stop receiving any data
so I did search through some demos and came across the ImageServer
demo which advised me to use
Connection.IOHandler.WriteBufferOpen;
Connection.IOHandler.Write(AStream);
Connection.IOHandler.WriteBufferClose;
Anyway, the same problem occurs. The client receives several thousand
bytes and suddenly nothing more is being received.
I should add that Client and Server in this case are being executed within
the same process (DUnit Tests) but of course each in it's own thread and
I do have the same problem using Indy 9 so I think I am missing something
general here.
Any advise?
--
Viele Grüsse
Stefan Meisner
www.delphi-online.at
Delphi Remoting Components
Delphi Memory Profiler
danny heijl
> I do have the same problem using Indy 9 so I think I am missing something
> general here.
Without seeing actual code it's difficult to judge, but I have several
Indy 9 tcp servers and clients exchanging hundreds of thousands of
messages a day without ever missing a single byte.
The pattern I use is very is simple:
- The indy server reads a message:
BytesToRead := Athread.Connection.ReadCardinal(True);
if BytesToRead = 0 then // end-of-conversation flag
begin
Athread.Connection.Disconnect;
break;
end;
SetLength(Data, BytesToRead);
Athread.Connection.ReadBuffer(Data[1], BytesToRead);
- The indy server sends the answer:
Athread.Connection.OpenWriteBuffer(1024);
try
BytesToWrite := Length(msg);
Athread.Connection.WriteCardinal(BytesToWrite, True);
Athread.Connection.Write(msg);
Athread.Connection.FlushWriteBuffer;
Athread.Connection.CloseWriteBuffer;
except
on E: Exception do
begin
Athread.Connection.CancelWriteBuffer;
....
end;
end;
- The indy client sends a message:
FTCPClient.OpenWriteBuffer(1024);
try
BytesToWrite := Length(msg);
FTCPClient.WriteCardinal(BytesToWrite, True);
FTCPClient.Write(msg);
FTCPClient.FlushWriteBuffer;
FTCPClient.CloseWriteBuffer;
except
FTCPClient.CancelWriteBuffer;
...
end;
- The indy client reads the answer:
BytesToRead := FTCPClient.ReadCardinal(True);
SetLength(Result, BytesToRead);
FTCPClient.ReadBuffer(Result[1], BytesToRead);
Note: most error handling has been removed.
Danny
---
Remy Lebeau (TeamB)
"Stefan Meisner" <stefan....@gmx.net> wrote in message
news:4696...@newsgroups.borland.com...
> At first I was using
> Connection.IOHandler.Write(AStream);
That is all you need.
> I did search through some demos and came across the
> ImageServer demo which advised me to use
<snip>
I do not recommend the use of write buffering with streams, especially if
the streams are large. That will cause socket errors to be reported in your
sending code because the socket can't handle that much data at one time.
Just use Write() by itself, and Indy will send the stream in smaller chunks
as needed.
> Anyway, the same problem occurs. The client receives several
> thousand bytes and suddenly nothing more is being received.
Then I would suspect a problem in the client's reading code before I suspect
a problem in Write().
> I should add that Client and Server in this case are being executed
> within the same process (DUnit Tests)
Makes no difference.
Gambit
Stefan Meisner
Thanks for your answer.
I have posted some more code in response to Remy's answer.
Groetjes!
Stefan Meisner
>> At first I was using
>> Connection.IOHandler.Write(AStream);
>
> That is all you need.
Ok. I will revert that code and remove the buffering.
Thanks for the hint!
> Then I would suspect a problem in the client's reading code before I
> suspect
> a problem in Write().
It's difficult to show the client code as it is partly generalized as it
should not only
use Indy but also other Socket libraries. So I hope the following extract
shows
the basic pattern:
After I have read the number of expected bytes I basically execute the
following loop:
var
lBuffer: array[0..1023] of Char;
lExpected, lBytesRead: Integer;
lDataStream: TStringStream;
[snipped]
repeat
RelinquishThread; // in Windows this is Sleep(0)
lBytesRead := ReadBuffer(lBuffer, SizeOf(lBuffer));
if lBytesRead = 0 then
Continue;
lTime := MeasureTime;
lDataStream.Write(lBuffer, lBytesRead);
until (lDataStream.Size = lExpected) or nTimeOutReached;
lTime and nTimeOutReached just measure and check for a given TimeOut
after that the client should stop reading. This one is 10 secs. in this
case.
So the problem might be in ReadBuffer (not in ReadBuffer but my usage of it
*g*)
which is somewhere deeper down and for Indy implemented as
function TDDOClientIndy10Connection.ReadBuffer(var Buffer;
Length: Integer): Integer;
begin
Result := GReadBuffer(IndyClient.IOHandler, Buffer, Length);
end;
function GReadBuffer(IOHandler: TIdIOHandler;
var Buffer; Length: Integer): Integer;
var
lBuffer: TIdBytes;
begin
Result := Min(Length, IOHandler.InputBuffer.Size);
SetLength(lBuffer, Result);
IOHandler.ReadBytes(lBuffer, Result, False);
CopyMemory(@Buffer, lBuffer, Result);
end;
Stefan Meisner
Thank you for your time reading my flawed code :o)
Remy Lebeau (TeamB)
> After I have read the number of expected bytes
How are you determing what the expected number of bytes are?
> I basically execute the following loop:
You are telling ReadBuffer() to read a fixed-sized buffer every time. Indy
does not use asynchronous sockets. If you ask it to read more bytes than
are actually sent, it will sit there waiting for bytes that will never
arrive. You should be adjusting your reading size according to the number
of bytes already received compared to the number of bytes expected. You
should be doing this generically for all socket libraries you support. For
example:
var
lBuffer: array[0..1023] of Byte;
lExpected, lBytesToRead, lBytesRead: Integer;
lDataStream: TMemoryStream;
begin
//...
repeat
RelinquishThread; // in Windows this is Sleep(0)
lBytesToRead := Min(SizeOf(lBuffer), lExpected);
lBytesRead := ReadBuffer(lBuffer, lBytesToRead);
lTime := MeasureTime;
if lBytesRead = 0 then Continue;
Dec(lExpected, lBytesRead);
lDataStream.Write(lBuffer, lBytesRead);
until (lExpected = 0) or nTimeOutReached;
//...
end;
Then just change ReadBuffer() to read the actual number of bytes specified,
no more and no less. Don't let it calculate its own size internally since
the loop already took care of that:
function GReadBuffer(IOHandler: TIdIOHandler; var Buffer; Length:
Integer): Integer;
var
lBuffer: TIdBytes;
begin
Result := Length;
SetLength(lBuffer, Result);
IOHandler.ReadBytes(lBuffer, Result, False);
CopyMemory(@Buffer, lBuffer, Result);
end;
> So the problem might be in ReadBuffer (not in ReadBuffer
> but my usage of it *g*) which is somewhere deeper down
It is a problem inside of ReadBuffer(). Your use of the IOHandler's InputBu
ffer is completely wrong.
When a new connection is established, the InputBuffer is intially empty.
The IOHander's ReadFromSource() method is the only place where bytes are
read from the socket itself and put into the InputBuffer. ReadBytes() is
the only place where Indy calls ReadFromSource(). Everything else in the
IOHandler calls ReadBytes() internally to get data. ReadBytes() reads from
the InputBuffer, calling ReadFromSource() only when needed to ensure the
InputBuffer has enough bytes to satisfy the length that was passed in. If
the InputBuffer has an adequate number of bytes, the socket is not read from
at all. So querying the InputBuffer.Size property does not tell you the
number of bytes that may be pending inside the socket waiting to be read
into the InputBuffer, but your code is assuming that it does. That is where
you are getting into trouble.
Eventually, you are going to read all of the bytes from the InputBuffer and
its Size will fall to 0. At that point, you will be telling ReadBytes() to
read 0 bytes without checking if the socket has any pending data. That will
cause ReadBuffer() to return 0, which causes your reading loop to skip to
the next iteration, which will read 0 bytes, and so on. At no point are you
doing anything to cause the IOHandler to receive new bytes from the socket,
so the InputSize.Size keeps returning 0, and you keep looping until your
timeout occurs.
If your ReadBuffer() method must be asynchronous, returning whatever is
currently available and not wait for the full amount of data to arrive, then
you need to change ReadBuffer() to something like the following:
function GReadBuffer(IOHandler: TIdIOHandler; var Buffer; Length:
Integer): Integer;
var
lBuffer: TIdBytes;
begin
if IOHandler.InputBuffer.Size = 0 then IOHandler.ReadFromSource(0);
Result := Min(Length, IOHandler.InputBuffer.Size);
if Result = 0 then Exit;
SetLength(lBuffer, Result);
IOHandler.InputBuffer.ExtractToBytes(lBuffer, Result, False);
CopyMemory(@Buffer, lBuffer, Result);
end;
Gambit
Stefan Meisner
Actually in the meantime I found the error using the InputBuffer wrong.
And also have changed the loop not always to read a fixed number of
bytes as you suggested now.
However, thank you very very much for your long and good explanation.
Finally the remaining 3 Unit testcases, where I did experience that
problem turned to "green" :)
Now I only do have one problem, but that's not related to Indy but
to Synapse and SecureBlackbox.