XMLHTTP and accented characters
Mr Beaton
I am building a form handler that will post my forms using XMLHTTP rather
than posting the form normally. All is well, I even made it so dynamical
that it will post virtually any form configuration I will be likely to need
within this project. However, a problem occurs when a form field contains
accented characters. If I post using the form, it works, but posting using
XMLHTTP renders accented characters useless. Code snippet:
<!--- Start Code --->
function fnSubmitForm(oForm, target) {
var strForm = "table=" + oForm.table + "&";
var strURL = oForm.action;
for(x=0; x<oForm.length; x++) {
var oTemp = oForm.elements[x];
var strValue = oForm.elements[x].value;
strValue = strValue.replace(/%/g, "%25");
strValue = strValue.replace(/&/g, "%26");
//strValue = strValue.replace(/+/g, "%2B");
//strValue = strValue.replace(/ /g, "%20");
//strValue = strValue.replace(/ /g, "+");
switch(oTemp.type) {
case "checkbox":
if(oTemp.checked) { strForm += oTemp.name + "=" + strValue + "&"; }
break;
case "radio":
if(oTemp.checked) { strForm += oTemp.name + "=" + strValue + "&"; }
break;
case "select":
if(oTemp.selected) { strForm += oTemp.name + "=" + strValue + "&"; }
break;
default:
strForm += oTemp.name + "=" + strValue + "&";
break;
}
}
if(true) {
if (window.XMLHttpRequest) { // code for Mozilla, etc.
xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() { xmlhttpChanges(xmlhttp,
target); }
xmlhttp.open("POST", strURL, false);
xmlhttp.setRequestHeader("Content-Type",
"application/x-www-form-urlencoded; charset=utf-8");
//Content-Type: application/x-javascript; charset=ISO-8859-1
xmlhttp.send(strForm);
}
else if (window.ActiveXObject) { // code for IE
xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
if (xmlhttp) {
xmlhttp.onreadystatechange = function() {
xmlhttpChanges(xmlhttp, target); }
xmlhttp.open("POST", strURL, false);
xmlhttp.setRequestHeader("Content-Type",
"application/x-www-form-urlencoded; charset=utf-8");
xmlhttp.send(strForm);
}
}
}
return false;
}
<!--- End Code --->
by toggling true/false in the IF and the RETURN, I route the POST through
either the form or the XMLHTTP. As stated, if I post through the form
itself, I get the expected result, while using the XMLHTTP object yields
some rather undesired effects to the accented chars.
Any ideas are most welcome.
//Leif Beaton
Martin Honnen
Mr Beaton wrote:
> I am building a form handler that will post my forms using XMLHTTP rather
> than posting the form normally. All is well, I even made it so dynamical
> that it will post virtually any form configuration I will be likely to need
> within this project. However, a problem occurs when a form field contains
> accented characters. If I post using the form, it works, but posting using
> XMLHTTP renders accented characters useless.
> var strValue = oForm.elements[x].value;
>
> strValue = strValue.replace(/%/g, "%25");
> strValue = strValue.replace(/&/g, "%26");
> //strValue = strValue.replace(/+/g, "%2B");
> //strValue = strValue.replace(/ /g, "%20");
> //strValue = strValue.replace(/ /g, "+");
> strForm += oTemp.name + "=" + strValue + "&";
> xmlhttp.open("POST", strURL, false);
> xmlhttp.setRequestHeader("Content-Type",
> "application/x-www-form-urlencoded; charset=utf-8");
You tell the server with an HTTP request header that the body of your
POST request has the content type application/x-www-form-urlencoded with
the charset UTF-8.
But if you tell the server that then your client-side code also needs to
take care that the string passed to the post method is in that format
and your few attempts to escape characters do not suffice at all to make
that string be application/x-www-form-urlencoded; charset=utf-8.
If you want to use that MIME type with that encoding then modern
JavaScript engines (as found in IE 5.5/Win and later, Netscape 6 and
later, Mozilla 1.0 and later) provide the function encodeURIComponent so
you would need to do remove your few replace calls and instead to e.g.
if (typeof encodeURIComponent != 'undefined') {
strForm += encodeURIComponent(oTemp.name) + "=" +
encodeURIComponent(strValue);
}
to make sure both the name and the value are encoded as you want and
tell the server.
> by toggling true/false in the IF and the RETURN, I route the POST through
> either the form or the XMLHTTP. As stated, if I post through the form
> itself, I get the expected result, while using the XMLHTTP object yields
> some rather undesired effects to the accented chars.
My suggestions above suffice to properly encode the data POSTED with
XMLHTTP in the format your request header declares (as far as
encodeURIComponent is supported, IE5/Win for instance usually does not
support that). However ensuring that the server-side application can
deal with that and in particular that the server-side application
receives the data with the same format when the browser posts your form
data itself requires other steps outside of client-side JavaScript, the
best way to ensure that the browser itself submits form data as
application/x-www-form-urlencoded; charset=utf-8 is the make sure your
HTML documents with the <form> are UTF-8 encoded and the browser is told
about that (e.g. with the Content-Type header text/html; charset=UTF-8
or at least with a <meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">).
And then you need to be able to deal with
application/x-www-form-urlencoded; charset=utf-8 on the server which
depends on what (e.g. ASP or PHP or JSP or ASP.NET) you use there and
potentially on how you further process the data (e.g. if you store it in
a data base you need then to decide whether that is configured and able
to store UTF-8 encoded data respectively has Unicode support for text
you want to store).
--
Martin Honnen --- MVP XML
http://JavaScript.FAQTs.com/
Mr Beaton
I think I am closer to a solution on this issue due to your tips. I encode
each name and value client side using encodeURIComponent, and I have written
a function serverside (ASP/VBScript) to decode the data. The strange part
is, when I insert it into the DB (MSSQL2000), the accented chars appear
corrupted. My suspicions leaned against the database itself, but after
performing a small test, I found that to be a mistake:
For Each Field In Request.Form
With oCmd
.Parameters("@1").Value = URLDecode(Field)
.Parameters("@2").Value = "Å " & URLDecode(Request.Form(Field))
.Execute ,,128
End With
Next 'Field
The leading "Å " is correctly represented in the DB, whilst all other
accented characters gets corrupted. My decoding func looks like this:
Function URLDecode(strText)
strDecoded = strText
Set oRegExpr = Server.CreateObject("VBScript.RegExp")
oRegExpr.Pattern = "%[0-9,A-F]{2}"
oRegExpr.Global = True
Set oMatchCollection = oRegExpr.Execute(strText)
For Each oMatch In oMatchCollection
strDecoded = Replace(strDecoded,oMatch.value,Chr(CInt("&H" &
Right(oMatch.Value,2))))
Next
URLDecode = strDecoded
End Function
I know this is out of bounds as for this group, but I thought you might want
to know where I'm at. :)
Thanks a bunch for your help, it's really appreciated.
//Leif Beaton
Martin Honnen
Mr Beaton wrote:
> I think I am closer to a solution on this issue due to your tips. I encode
> each name and value client side using encodeURIComponent, and I have written
> a function serverside (ASP/VBScript) to decode the data. T
> My decoding func looks like this:
>
> Function URLDecode(strText)
You are trying much more than is needed, ASP should decode the data
itself so simply accessing Request.Form("FieldName") should be the
decoded data. But as I pointed out in my earlier response the main issue
is to have the client-side script and the server-side script agree on
the encoding and the charset used.
So the easiest and best way as far as I understand is to encode your ASP
documents as UTF-8 and indicate the encoding as the code page (UTF-8 is
code page 65001 e.g.
<%@ Language="VBScript" CodePage="65001" %>
<%
Response.ContentType = "text/plain; charset=UTF-8"
For Each Key In Request.Form
Response.Write Key & "=" & Request.Form(Key) & VbCrLf
Next
Response.Write "Original request data: " & Request.Form & VbCrLf
%>
That way ASP will properly decode the data send as
application/x-www-form-urlencoded; charset=UTF-8 automatically, there is
no need for you to try to implement a decoding function.
As for your data base ask in an appropriate group then, with the above
example the VBScript in the ASP page will have the properly decoded data
in the strings e.g. Request.Form("FieldName") gives. What you need to do
then to store that in the data base depends on the data base, the column
type.
Serge Baltic
MH> But as I pointed out in my earlier response the main
MH> issue
MH> is to have the client-side script and the server-side script agree
MH> on the encoding and the charset used.
MH> So the easiest and best way as far as I understand is to encode your
MH> ASP documents as UTF-8 and indicate the encoding as the code page (UTF-8
MH> is code page 65001 e.g.
And why not use text/xml instead of URL-encoding, and send parameters in
a form of an XML Document?
(H) Serge
Bob Barrows [MVP]
> Hello,
>
>> But as I pointed out in my earlier response the main
>> is to have the client-side script and the server-side script agree
>
> And why not use text/xml instead of URL-encoding, and send parameters
> in a form of an XML Document?
>
Bob Barrows
--
Microsoft MVP -- ASP/ASP.NET
Please reply to the newsgroup. The email account listed in my From
header is my spam trap, so I don't check it very often. You will get a
quicker response by posting to the newsgroup.
Martin Honnen
Serge Baltic wrote:
> And why not use text/xml instead of URL-encoding, and send parameters in
> a form of an XML Document?
The original poster described that he has both a HTML form and script
with XMLHTTP submitting data to the same server-side application and he
wanted to have the same processing on the server for the data. That's
why application/x-www-form-urlencoded data is sent with XMLHTTP so that
the server-side app can use Request.Form in ASP to process the data
whether it is the browser submitting the form data or the script.
Bob Barrows [MVP]
I'm not sure where the OP said this was a requirement. In fact, his first
sentence was "I am building a form handler that will post my forms using
XMLHTTP rather
than posting the form normally."
Did I miss something? To accomplish the stated requirement, I would be
building a well-formed xml document in client-side script, using xmlhttp to
pass the document to the server page which parses the document to process
the data elements.
Bob Barrows
--
Microsoft MVP - ASP/ASP.NET
Please reply to the newsgroup. This email account is my spam trap so I
don't check it very often. If you must reply off-line, then remove the
"NO SPAM"
Martin Honnen
Bob Barrows [MVP] wrote:
> I'm not sure where the OP said this was a requirement. In fact, his first
> sentence was "I am building a form handler that will post my forms using
> XMLHTTP rather
> than posting the form normally."
He also said
> by toggling true/false in the IF and the RETURN, I route the POST through
> either the form or the XMLHTTP. As stated, if I post through the form
> itself, I get the expected result, while using the XMLHTTP object yields
> some rather undesired effects to the accented chars.
so at least he tried to have server-side code process data the same way
that was submitted by XMLHTTP and by the browser posting a HTML form.
Not sure whether that was a requirement or simply a lack of knowledge
and skills to build XML and post that and process it on the server.
Bob Barrows [MVP]
the problem arose due to the xmlhttp object or whether a normal form posting
would also cause the problem.
Bob Barrows
--
Mr Beaton
I am sorry for my absence from my own thread for a couple of days, I've been
ill beyond belief. I find, however, to my great satisfaction and somewhat of
a surprise that the thread has had a life of its own in my absence. I must
thank you all, as several elements of my attempts has (correctly) been
deemed less than brilliant. Catching up on the thread has given me several
ideas, not only for the topic at hand, but for a few potential pitfalls in
the near future for my application. I am truely grateful for the input you
all provided. As for the most recent events in the thread;
>>> I'm not sure where the OP said this was a requirement. In fact, his
>>> first sentence was "I am building a form handler that will post my
>>> forms using XMLHTTP rather than posting the form normally."
True, not a requirement. I simply started out at that end of the spectre as
I am far more used to handle form data serverside. However; I often send XML
data in the other direction. Therefore I do believe I'll try it out as means
for posting, as Mr. Barrows have pointed out for me twice, and apparently
Mr. Baltic concurrs. I've gone through the requirements for my project, and
I cannot see a single reason why I wouldn't go for XML transfer. As a matter
of fact, save for binary streams, I am having a hard time coming up with a
single reason not to use XML.
> That way ASP will properly decode the data send as
> application/x-www-form-urlencoded; charset=UTF-8 automatically, there is
> no need for you to try to implement a decoding function.
That's what I thought, but apparently I am messing up somewhere. Rest
assured, even if I am going to use XML transport instead, I will _not_ give
up on this part of my problem, as I know myself far to well to even _try_ to
abandon a method just due to my lack of skill / luck. I could kiss sleeping
goodbye for the better part of a decade if I gave up on it... :)
>> He also said
>>
>>> by toggling true/false in the IF and the RETURN, I route the POST
>>> through either the form or the XMLHTTP. As stated, if I post through
>>> the form itself, I get the expected result, while using the XMLHTTP
>>> object yields some rather undesired effects to the accented chars.
>>
>> so at least he tried to have server-side code process data the same
>> way that was submitted by XMLHTTP and by the browser posting a HTML
>> form.
>>
>> Not sure whether that was a requirement or simply a lack of knowledge
>> and skills to build XML and post that and process it on the server.
Neither, acctually. I'll let lack of thorough use of brain take credit for
my oversight there.
> I think this was his attempt to debug his problem. He was trying to see if
> the problem arose due to the xmlhttp object or whether a normal form
> posting
> would also cause the problem.
Precisely so.
Now, to summarize, you guys have given me a lot of options and a lot of
ideas. I acctually think you'll manage to keep me busy for the next couple
of days... :) I'll work with the ideas and see what I come up with. I will
of course post the final "solution" to the thread, just in the odd event
that somebody wants to read it. I've done quite intense "studying" of this
phenomenon on the web over the last couple of weeks, and it seems like I am
neither the first nor the last to struggle with this, so who knows? Somebody
might find it useful. :)
Mr. Barrows; I will naturally do the same in asp.general.
Again; Thank You All.
//Leif Beaton
Serge Baltic
M> True, not a requirement. I simply started out at that end of the
M> spectre as I am far more used to handle form data serverside.
M> However; I often send XML data in the other direction.
Just a note: before you go, you'd better make sure XMLHTTP works as expected
on all the to-be-supported browsers. I can remember having some problems
with it on IE5.
(H) Serge