base64 encoding from Delphi Client
Patrick Morris
I am working on a Delphi SOAP client (d6) that interfaces to a Perl
SOAP::Lite server. Most things work ok, but having problems sending binary
data (ie an image).
Setting the item type to base64 does not appear to work - the type is
actually set as string in the XML that is sent to the server. Sending the
raw data as a string doesnt work since Delphi appears to balk at sending
unprintable characters (not that suprising).
Trying to send as an dynamic byte array hits the problem described else
where with the pointers to dynamic arrays (I have tried the fix for that
posted by Bruneau but that causes the program to lock ....
Any ideas would be most welcome before i go and write this in Perl instead
......
Patrick
Dave Scofield
server. I don't know what your application is doing, but mine is a
"community library", where users of our software can contribute articles
(documents) they've written. They're required to fill out some info about
the document, then they upload it, then it's available to other users.
My first inclination was to put the file into a TByteDynArray, and then send
it as part of a soap method call. However, one problem with that approach
(aside from the minor detail that it doesn't work) is that the user gets no
feedback on the progress of the upload, and has no chance to cancel
mid-upload.
So, in a nutshell, here's what we're doing:
1. Users fills out document info, I call a soap method to "register" the
document with the server. This puts the doc info into a database, and
returns me a document ID (e.g. 102). The record in the database is marked as
not active.
2. Calc the CRC32 of the file.
3. Zlib the file, the encode it with base64 encoding to have a text
representation of the compressed file. This "text" is still far smaller than
the original file (which is in HTML format).
4. Now, in however many chunks it takes, the file is transferred to the
server a chunk at a time (like 8 or 16k, for example). Each call to the
server sends the document ID, and a string containing the chunk-size worth
of the base64 encoded file. Since it's all "legal" text, soap just passes it
through as a string. The server just writes each chunk to a temp directory,
into files like doc102.001, doc102.002, etc.
5. When all the chunks are sent, the client calls a "finalize" that directs
the server to go collect all the doc102.* files, concat them, decode from
base64, de-zlib, and write a completed file. It returns a CRC32 of the file,
which is compared on the client side, and if everything is cool, then the
client calls a last soap method to "activate" the document (just a flag in
the table).
The user is shown a progress bar during the upload, and if they cancel, the
client calls a server method to go delete all the doc102.* files from the
upload directory, and remove the record from the table.
This is working like a charm for us. Although it's slightly slower to call
the soap server multiple times, it's well worth it to give the user feedback
during the process, and allow them to cancel at any point.
Oh, and the upload process is all done in a thread.
HTH,
Dave
"Patrick Morris" <pmo...@fotango.com> wrote in message
news:3bf13eab$1_2@dnews...
Patrick Morris
Dave Scofield wrote in message <3bf14c90_1@dnews>...
>I can tell you what I did for my particular need of sending files to the
>server. I don't know what your application is doing, but mine is a
>"community library", where users of our software can contribute articles
>(documents) they've written. They're required to fill out some info about
>the document, then they upload it, then it's available to other users.
>
[snip]
Thanks for that. User feedback is not important in this application :)
The main problem is that I cant really change the Perl SOAP server end, and
that handles the binary data as a SOAP-ENC:base64 with no problem -
automatically encoding / decoding it.
Having a look though the delphi source code it appears that it will encode
ByteDynArray using base64 (i think!) and mark the type as base64 in the XML.
Now only if the &%*(ing thing didnt crash ...
Currently modifiying it to do it with dynamic integer arrays as well
Patrick
Jean-Marie Babet
> ByteDynArray using base64 (i think!) and mark the type as base64 in the
XML.
Patrick,
If you have update#1 installed, yes Delphi will serialize TByteDynArray base
base64Binary. Are you getting any errors from the server? Or can you post a
copy of the packet we're sending? (You can use TLinkedRIO to quick to that).
If the server is running somewhere accessible, I'll be happy to investigate.
Regards,
Bruneau.
Jean-Marie Babet
The current code (which we got from Kylix) reads:
if BytesRead < Request.ContentLength then
begin
SetLength(Buffer, Request.ContentLength);
Stream.Write(Request.Content[1], BytesRead);
repeat
ChunkSize := Request.ReadClient(Buffer[0],
Request.ContentLength - BytesRead);
if ChunkSize > 0 then
begin
Stream.Write(Buffer[0], ChunkSize);
Inc(BytesRead, ChunkSize);
end;
until ChunkSize = -1;
I did not investigate this so I'm unaware of the issues. My concern is your
explicit mention of IIS - which would not have been part of the equation for
Kylix. Our goal is single source and so far things are looking good (even if
we have to IFDEF like in the SOAPHTTPTrans.pas I posted yesterday). So I
have a question: is there an IIS specific issue that the above will not
handle? I'm making a note to make sure the above is correct for both
platforms but thought I'd ask in case there was something specific to IIS
here.
Regards,
Bruneau.
"Dave Nottage" <da...@removethis.b3.com.au> wrote in message
news:3bf17b8b_2@dnews...
> "Patrick Morris" wrote:
> > Having a look though the delphi source code it appears that it will
> encode
> > ByteDynArray using base64 (i think!) and mark the type as base64 in
> the XML.
> > Now only if the &%*(ing thing didnt crash ...
>
> If you're using IIS and sending files > UploadReadaheadSize (or whatever
> the setting is called), you'll need to use a modified WebBrokerSOAP.PAS:
>
> { BUG ALERT
> if BytesRead < Request.ContentLength then
> begin
> SetLength(Buffer, Request.ContentLength);
> Stream.Write(Request.Content[1], BytesRead);
> repeat
> ChunkSize := Request.ReadClient(Buffer,
> Request.ContentLength - BytesRead);
> if ChunkSize > 0 then
> begin
> Stream.Write(Buffer, ChunkSize);
> Inc(BytesRead, ChunkSize);
> end;
> until ChunkSize = -1;
> end else
> Stream.Write(Request.Content[1], BytesRead);
> }
> // Fixed code:
> if BytesRead < Request.ContentLength then
> begin
> SetLength(Buffer, Request.ContentLength);
> Stream.Write(Request.Content[1], BytesRead);
> repeat
> // --> added "[BytesRead]"
> ChunkSize := Request.ReadClient(Buffer[BytesRead],
> Request.ContentLength - BytesRead);
> if ChunkSize > 0 then
> begin
> // --> added "[BytesRead]"
> Stream.Write(Buffer[BytesRead], ChunkSize);
> Inc(BytesRead, ChunkSize);
> end;
> // --> changed from "until ChunkSize = -1" to
> until (BytesRead = Request.ContentLength) or (ChunkSize <=
> 0);
> end else
> Stream.Write(Request.Content[1], BytesRead);
> // End fixed code
>
> The problem has been reported. Hopefully it'll be fixed in the next
> update.
>
> --
> Dave Nottage
>
>
Dave Nottage
> If you have update#1 installed, yes Delphi will serialize
TByteDynArray base
> base64Binary. Are you getting any errors from the server?
See my other reply: With IIS, if you're sending files >
UploadReadaheadSize, the section of the code where it says "bug alert"
goes into an eternal loop, so the server appears to hang.
--
Dave Nottage
Dave Nottage
> Having a look though the delphi source code it appears that it will
encode
> ByteDynArray using base64 (i think!) and mark the type as base64 in
the XML.
> Now only if the &%*(ing thing didnt crash ...
If you're using IIS and sending files > UploadReadaheadSize (or whatever
Nick Hodges (TeamB)
I am glad to hear that you guys got that worked out. Sounds like a
great solution. Any chance of a community site article on it? <g>
Nick Hodges - TeamB
HardThink, Inc.
Please always follow the newsgroup guidelines --
http://www.borland.com/newsgroups
Dave Nottage
> Thanks Dave. I remember mention of this.
>
> The current code (which we got from Kylix) reads:
I havent even looked at it yet! <g>
> I did not investigate this so I'm unaware of the issues. My concern is
your
> explicit mention of IIS..
Quite right.. it affects any web server that doesnt return BytesRead >
ContentLength.
--
Dave Nottage
Shiv Kumar
The basic logic behind this is:
If BytesRead < ContentLength
the read buffer until
ChunkSize = -1 (no more bytes or connection dropped)
or
accumulator of ChunkSize = ContentLength
The code that Kylix uses looks fine for IIS as well.
--
Shiv Kumar
The Delphi Apostle
http://www.matlus.com
http://www.delphisoap.com
Jean-Marie Babet
than 48K of data? If yes, I'm pretty sure the current code works but I'll
check.
Thanks again.
Bruneau.
"Shiv Kumar" <sh...@erols.com> wrote in message news:3bf1eb98_2@dnews...
Shiv Kumar
I'm not sure "problem" you refer to <g>. That's the code that gets executed
if the amount of data sent is larger than the read ahead size (normally
48K).
As an aside, I've had a question for a while:
What is more efficient
SomeStream.Read(Pointer(sTemp)^, iSize)
or
SomeStream.Read(sTemp[0], iSize)
Jean-Marie Babet
>
> SomeStream.Read(Pointer(sTemp)^, iSize)
> or
> SomeStream.Read(sTemp[0], iSize)
Well the first one yields:
mov edx, [sTemp]
mov ecx, iSize
mov eax, [SomeStream]
mov ebx, [eax]
call dword ptr [ebx+8]
and the second:
mov edx, [sTemp]
mov ecx, iSize
mov eax, [SomeStream]
mov ebx, [eax]
call dword ptr [ebx+8]
I'd say we have a tie :)
Regards,
Bruneau.
"Shiv Kumar" <sh...@erols.com> wrote in message news:3bf1f16d_2@dnews...
Shiv Kumar
I didn't think about using the CPU view (I'm not accustomed to it<g>).
Patrick Morris
Dave :
The server is at the other is not IIS (Apache + SOAP::Lite)
Bruneau :
Yes have update #1 installed.
While it does encode ByteDynArray as base64 the delphi client crashes later
on - the old AV with TDynByteArray - tried using the GetOrdPropEx /
SetOrdPropEx fixes that you have supplied else where but this leads to a
crash at a later point instead - not pin pointed where yet.
I have modified TSOAPDomConv.ConvertNativeArrayToSoap to also encode an
integer dynamic array into base64 (a hack but it gets round the AV with the
byte array) but still see problems at the server end.
Will get a copy of the packet and post it here ...
thanks
Patrick
Jean-Marie Babet wrote in message <3bf177c9$1_2@dnews>...
Patrick Morris
<?xml version="1.0" encoding='UTF-8'?><SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><N
S1:login xmlns:NS1="urn:StoreImages"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:NS2="http://www.fotango.com/Services.xsd"><NS1:body
href="#1"/><NS2:body id="1" xsi:type="NS2:LoginParam"><user_name
xsi:type="xsd:string">pdenis</user_name><password
xsi:type="xsd:string">a</password></NS2:body></NS1:login></SOAP-ENV:Body></S
OAP-ENV:Envelope><?xml version="1.0" encoding='UTF-8'?><SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><N
S1:delete_log xmlns:NS1="urn:StoreImages"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><msg
xsi:type="xsd:string"></msg></NS1:delete_log></SOAP-ENV:Body></SOAP-ENV:Enve
lope><?xml version="1.0" encoding='UTF-8'?><SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><N
S1:delete_log xmlns:NS1="urn:StoreImages"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><msg
xsi:type="xsd:string"></msg></NS1:delete_log></SOAP-ENV:Body></SOAP-ENV:Enve
lope><?xml version="1.0" encoding='UTF-8'?><SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><a
dd_binary
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><data
xmlns:NS1="xsd" xsi:type="NS1:base64">/9j/4AAQSkZJRg ......
Jean-Marie Babet
is failing]. The little section that you posted (at the very end) looks very
suspicious:
<SOAP-ENV:Body>
<add_binary
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<data xmlns:NS1="xsd" xsi:type="NS1:base64">/9j/4AAQSkZJRg ......
If method add_binary has a parameter named 'data' that's a TByteDynArray,
this packet is wrong. The 'NS1' namespace prefix is pointing to what usually
is the prefix for the XMLSchema namespace [i.e. xsd]?? And the 'xsi:type' is
using NS1 as prefix instead of 'xsd'. Something is definitely wrong with
this packet. However, I'd need to see more... maybe it's just that it's a
partial post.
I just realized something: there's a SOAP::Lite WebService at
http://services.soaplite.com/interop2.wsdl that will echo back any
base64Binary data it's sent. It's part of the WebServices Interop. Lab and I
already have a client that can invoke that Server. Actually, I know I've
invoked the Server with base64Binary data. I just don't remember the result
for SOAP::Lite (there are about 17 endpoints that I run against). So what I
need is to know exactly that you're streaming out. If you're crashing in
OpToSOAPDOMConv.pas, I'd like to know where, if possible. If I can make time
today [I'm still at home but will be heading in soon], I'll run the Client
test against SOAP::Lite and note the results of us sending base64Binary data
to it.
NOTE: The S|GetOrdProp[ex] is only relevant if your TByteDynArray is a
member of a TRemotable-derived object that we're [de]serializing.
Regards,
Bruneau.
.
"Patrick Morris" <pmo...@fotango.com> wrote in message
news:3bf24d8d$1_1@dnews...
Jean-Marie Babet
I just ran the Delphi Client against SOAP::Lite and echoBase64 passed.
Here's are excerpts of the packets sent and received (I'm skipping the data
as the test sends big chunks of data - a random size between 64K and 256K -
of bytes).
Request of Delphi Client
================
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><NS1:echo
Base64 xmlns:NS1="http://soapinterop.org/"><inputBase64
xsi:type="xsd:base64Binary">
... Here goes the data
</inputBase64></NS1:echoBase64></SOAP-ENV:Body></SOAP-ENV:Envelope>
Response from SOAP::Lite Server
=======================
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><n
amesp1:echoBase64Response xmlns:namesp1="http://soapinterop.org/"><return
xsi:type="xsd:base64Binary">
.... Here goes the data
</return></namesp1:echoBase64Response></SOAP-ENV:Body></SOAP-ENV:Envelope>
The signature of the routine invoked is:
function echoBase64(const inputBase64: TByteDynArray): TByteDynArray;
stdcall;
Can you give more some additional information? Or better, can you import the
WSDL of the SOAP::Lite service I mentioned earlier and invoke the echoBase64
method? Doing so will let me know whether the issue here is client related
(your client will fail) or server related (your client will work).
Regards,
Bruneau.
Dave Scofield
Yeah, we got it worked out -- just before we tracked you down for some help
:)
A community article ... now wait - you mean *our* community, or the Borland
community??
Actually, it'd be kinda fun to do an article about it, and give some sample
code.
Dave
"Nick Hodges (TeamB)" <nickh...@yahoo.com> wrote in message
news:fg63vts3rn32ckpse...@4ax.com...
Nick Hodges (TeamB)
<spammen...@epiphanysoftware.com> wrote:
>Actually, it'd be kinda fun to do an article about it, and give some sample
>code.
I think that would be a great idea. It seems like you guys used a
cool solution, and you owe it to the rest of us to share. <g>
Patrick Morris
It seems i was changing too many things at once and did go for the simple
option of putting the right type defs in ....
Oh well I have learnt lots of useful stuff anyway.
Thanks for the help Bruneau, will no doubt call on your services again soon
!
Patrick
Dave Scofield
I'm a community member, would I just upload a document to code central?
Thanks,
Dave
"Nick Hodges (TeamB)" <nickh...@yahoo.com> wrote in message
news:ndq6vtg16okg6b4km...@4ax.com...
John Kaster (Borland)
> Do tell, how exactly does one go about publishing a white paper like that?
> I'm a community member, would I just upload a document to code central?
You could do that. However, if you'd like to get paid for the article,
go to
http://community.borland.com/getpublished/
And follow the instructions there.
--
John Kaster, Borland Developer Relations, http://community.borland.com
$1280/$50K: Thanks to my donors!
http://homepages.borland.com/jkaster/tnt/thanks.html
Buy Kylix! http://www.borland.com/kylix * Got source?
http://codecentral.borland.com
The #1 Java IDE: http://www.borland.com/jbuilder
Nick Hodges (TeamB)
<spammen...@epiphanysoftware.com> wrote:
>Do tell, how exactly does one go about publishing a white paper like that?
>I'm a community member, would I just upload a document to code central?
Dave --
You could put it in CodeCentral I guess, but you can get paid to
publish it at
http://community.borland.com/getpublished/