CryptoStream\NetworkStream Flushing Issue
Brian E. Dean
questions. I'm a short time listener first time caller to this group.
I'm a Network Engineer working for a worldwide financial software company.
My primary responsibility is network security. I cut my development teeth
on VB 4 almost 8 years ago. Wow! When I say it like that it makes me feel
old. Having some experience with some development and knowledgeable about
network security I have been tasked with designing and writing an end-to-end
solution for transferring financial data from a to b over an unsecured
network. Some other things need to happen to this data which would
unfortunately preclude us from using some canned software package. One
would think that working for a software company I would have plenty of
resources at my disposal. Enter politics into the computing world. Yeah
like that ever happens. Anyway I thought a little background would be good
for my first post. Now to the reason of my post ...
I have read some historical post on my issue which is cache of data by the
CryptoStream when is sits on top of a NetworkStream. I have used Rich
Blum's example to write to a MemoryStream first, which works great Rich
thanks for the example, I'm off to the bookstore as we speak. How would
this work with large files? Say a half a gig compressed. Not good I
suspect. It looks as if all the data would need to be written to memory
before transmitting the data across the wire.
In my testing although I have read that calling FlushFinalBlock() flushes
the CryptoStream it doesn't actually transmit anything even after I call
Flush() on the NetworkStream. It only seams to get transmitted when the
underlying Socket is closed. Only from an academic standpoint why is this?
I didn't think the Socket class buffered anything. Is the Socket notifying
the NetworkStream of its intent to close giving it an opportunity to flush
before closing. If so can that be called programmatically? I could use two
Sockets for my solution like FTP, one control and one data. But being a
Network Engineer the fewer ports I need to open through a firewall the
better. I would prefer not to do that. SSH does it why can't I. Don't
answer that. lol
Would the best way to solve my problem be to build upon Rich's example and
just simply write x bytes to the MemoryStream from the FileStream via The
CryptoStream and then flush the MemoryStream to the NetworkStream and then
just loop until the EOF? Or am I making this harder than what It needs to
be?
Thanks in advance,
Brian
Rich Blum
>
> In my testing although I have read that calling FlushFinalBlock() flushes
> the CryptoStream it doesn't actually transmit anything even after I call
> Flush() on the NetworkStream. It only seams to get transmitted when the
> underlying Socket is closed. Only from an academic standpoint why is this?
The FlushFinalBlock() method just notifies the underlying stream
about the status of the crypto buffer. It is up to the underlying
stream what to do with that information. Usually the NetworkStream
attempts to buffer data to efficiently transfer the data across the
network. Unfortunately, the final encrypted block can get caught in
this mess. I have not seen a case where Flush()ing the NetworkStream
didn't complete the transaction. Its possible that in your case there
was nothing left to flush - the final block may have already been
sent.
> Would the best way to solve my problem be to build upon Rich's example and
> just simply write x bytes to the MemoryStream from the FileStream via The
> CryptoStream and then flush the MemoryStream to the NetworkStream and then
> just loop until the EOF? Or am I making this harder than what It needs to
> be?
For this case I don't think that would be necessary. The problem
is identifying the start and end of the encrypted data stream. There
are three techniques (that I know of) that can be used:
1. Sending the size of the data to the receiver so it knows how
much data to receive.
2. Using boundary markers around the data.
3. Closing the stream after the data is sent.
The example in my book used the MemoryStream as a temporary holding
bin for determining the size of the transmitted data. If you are
sending an existing file you can easily get the file size
(FileInfo.Length), and send the file size to the receiver, then send
the file directly from the FileStream to the CryptoStream (pointing to
the NetworkStream). Of course the other reads the file size first,
then knows how much data to expect to receive. It can then feed the
received data straight into the CryptoStream pointing to a FileStream
to store the data.
Alternatively, if you are only sending one file per TCP session,
you can just close the NetworkStream when the encrypted data is sent.
The receiver can then just loop on the Read() until it sees the
network stream is closed (receives 0 bytes of data). That way you
don't have to mess with the file size.
Hope this gives you some more material to work with on your
project. Good luck.
Rich Blum
Author of "C# Network Programming" (Sybex)
http://www.sybex.com/sybexbooks.nsf/Booklist/4176
Brian E. Dean
size. After staring at this code for hours it gets hard to see the forest
through the trees. As I sit and look at cleartext byte arrays, ciphertext
byte arrays, memory streams wondering how I'm going to tell the server how
much data I need to send I forget, hey why don't I just get the size from
the filesystem. Too funny. I work on this and let you know. Thank you
this is very helpful.
"Rich Blum" <rich...@juno.com> wrote in message
news:ccdac0da.03042...@posting.google.com...
Jack
I personally reported the NetworkStream/CryptoStream issue to
Microsoft about 5 months ago. After about 4 days they did confirm
this is a bug and both the Networkstream wrapped in a Cryptostream
does not work properly. The last piece of data is never sent until
the Networkstream is closed.
Most of the samples on the net work because small pieces of data are
sent and the streams are closed, however if you're interested in
making a stream that can respond back to the originating stream it
will never happen.
According to MS the fix will be made but they are not sure when. It is
not fixed in .NET 1.1
It is possible to use the ICryptoTransform to process the data over a
networkstream, however I found the performance of ICryptoTransform to
be unacceptable for large amounts of data.
"Brian E. Dean" <brian...@cowww.com> wrote in message news:<u$Z0MMnCD...@TK2MSFTNGP12.phx.gbl>...
Rich Blum
> Just thought I'd throw a little info into the mix here.
>
> I personally reported the NetworkStream/CryptoStream issue to
> Microsoft about 5 months ago. After about 4 days they did confirm
> this is a bug and both the Networkstream wrapped in a Cryptostream
> does not work properly. The last piece of data is never sent until
> the Networkstream is closed.
>
> Most of the samples on the net work because small pieces of data are
> sent and the streams are closed, however if you're interested in
> making a stream that can respond back to the originating stream it
> will never happen.
>
I'm not sure I would call this a bug (and I'm surprised that
someone at Microsoft did). It just reflects the normal behavior of TCP
communication. You must compensate for that in your program. If you
feed the receiving NetworkStream directly to the CryptoStream, bad
things can happen.
Symmetric encryption methods handle data in blocks. They emulate a
data stream by chaining blocks together (called cipher block
chaining). When the decryptor doesn't receive a complete block it
can't decrypt the data and throws an exception.
Enter TCP, and how it handles data. As data is fed from the
encryptor to the NetworkStream, TCP does not guarantee that each block
of data will be received by the other end as it was transmitted (i.e.
partial blocks may be received, with a delay between data packets).
This just throws the decryptor into fits. As it decodes the stream "on
the fly", it excpects complete blocks of data. To solve the problem,
you must work to guarantee that all of the data is available for the
decryptor to do its job.
One method, as you state, is to close the TCP connection at the
end of the data stream. This allows the receiving end to recognize the
end of the data stream and feed it into the decryptor. Of course this
is impractical for many applications.
The best solution I've found is to feed the receiving
NetworkStream data to a MemoryStream. When you know you've received
all the data, feed that stream to the CryptoStream. Of course the
trick then becomes knowing when you've received all the encrypted
data. One method is by sending the size of the encrypted data before
the actual data. This allows the receiving end to read the size, and
know exactly how much data to receive before feeding it into the
decryptor. In Chapter 17 of my book I show an example of this
technique (you can freely download the code from the Sybex web site).
Yes, this does create a middleman and a slight performance hit. In
a perfect world it would be nice to just feed the NetworkStream
directly to the Cryptostream, and if the Microsoft guys can figure out
a way to do it that would be great. But in the meantime, this is an
easy way to use encryption in your network apps.
Jack
> someone at Microsoft did). It just reflects the normal behavior of TCP
> communication. You must compensate for that in your program. If you
> feed the receiving NetworkStream directly to the CryptoStream, bad
> things can happen.
>
Rich -
I agree with your post only taking exception to your surprise on the
bug status. Here's why Microsoft and I thing this qualifies as a bug.
Both CryptoStream and NetworkStream are inherited from the same [base
stream] class. It's assumed in the OOP world all objects which
inherit from the same base object would handle or expose underlying
data the same way. Since a CryptoStream wraps the underlying stream
(in our case a networkstream) it should function in the data layer the
same way all other streams function.
The issue is specific to cryptostream. The networkstream is expecting
(if I remember correctly) a single 0 byte as notification no
additional data is available to send; otherwise the networkstream sits
in a waiting state expecting more data. The cryptostream never sends
the final byte to the underlying networkstream, hence our bug. It
should be noted what moved this to a bug status is all other stream
objects work perfectly with a NetworkStream, which tells me the
cryptostream does not send the end of data byte to the networkstream.
This was confirmed by Microsoft developers who wrote the CryptoStream
and Networkstream. Unfortunately for all of us they never spoke to
each other.
I did attempt to use a memorystream as a middleman (which you
mentioned) but the system I was working on required a very large
amount of data to be transferred via TCP in an encrypted state and
creating a 8MB memorystream probably isn't the best idea. As I
mentioned earlier ICryptoTransform will work but I found the best
solution from a performance standpoint was to take the raw data and
encrypt it with the cryptostream wrapping a filestream creating an
encrypted file on the local system; grab the filestream and send it
out through the networkstream - if the data was not too large, I would
substitute a memorystream for the encrypted file.
Also the idea of passing a small packet describing the size of the
data is a good idea no matter what the application. If data sizes
don't match, the server could request the data again. It's redundant
because TCP will guarantee the data has all been sent, but I always
like the idea of making sure myself.