Spark Output Encoding

[Michael Edwards]

Is it possible to change the output encoding of Spark?

We are using Spark to output XML, however non-UTF-8 characters are
being output. We want to change the XML encoding to UTF-16 to see if
this resolves the problem but I need to set Spark to output in UTF-16
as well.

Is this possible?

Cheers

Mike

[Rob G]

Hi Mike,

Spark contains a class called Spark.Spool.SpoolWriter which inherits from TextWriter and the Encoding override looks like this:

        public override Encoding Encoding

        {

            get { return Encoding.Default; }

        }

Now if you're just looking to see if this will fix your problem, you could download the source and change it to Unicode and retest your stuff. If that works, and/or if anyone else sees a general need for this, then we can talk about putting this in as some kind of configurable setting.

Thoughts?

Cheers,

Rob

--
You received this message because you are subscribed to the Google Groups "Spark View Engine Dev" group.
To post to this group, send email to spar...@googlegroups.com.
To unsubscribe from this group, send email to spark-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spark-dev?hl=en.

[Asbjørn Ulsberg]

On Sat, 16 Oct 2010 15:35:46 +0200, Rob G <robertg...@gmail.com> wrote:

> public override Encoding Encoding
> {
> get { return Encoding.Default; }
> }

Is this the default encoding used by Spark? If so, I'd call it a bug.
Please read the remark on
http://msdn.microsoft.com/en-us/library/system.text.encoding.default.aspx:

"Different operating systems might use different encodings as the default.
Therefore, data streamed from one operating system to another might be
translated incorrectly. To ensure that the encoded bytes are decoded
properly, your application should use a Unicode encoding, that is,
UTF8Encoding, UnicodeEncoding, or UTF32Encoding, with a preamble. Another
option is to use a higher-level protocol to ensure that the same format is
used for encoding and decoding."

Encoding.Utf8 is a much better and more sane default, imho. Either way, is
there a reason Spark doesn't use the responseEncoding setting of
web.config?

http://msdn.microsoft.com/en-us/library/hy4kkhe0.aspx

In a non-web context, using responseEncoding of course doesn't make sense
(and Encoding.Default may be better for console applications and such),
but Utf8 and responseEncoding should be the default, imho.

-Asbjørn

>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

[Michael Edwards]

Hi Rob

I am reluctant to download a copy of Spark and make a code change just
for this problem, it is lazy but I don't want another code library to
manage. If there was a way to make the configurable that would be
great.

Cheers

Mike

[Rob G]

Hi Mike,

I wasn't suggesting that you manage another code library - I was just curious as to whether or not that actually fixed your problem, or if there is another point in the rendering pipeline where work may be required? If it fixes your problem, and as per Asbjørn's assertion, we can pretty much go ahead with the change anyway unless someone specifically disagrees for any reason.

Regards,

Rob 

[Michael Edwards]

Rob
Cool, thanks for the clarification, I will try to have a look sometime soon.

Cheers

[Henning]

I believe I've found a bug with regards to encoding.
I have had similar problems where I can't output norwegian characters,
æ,ø,å. They display as a question mark.

I finally got sick of this, so I've managed to find a possible bug.
In AbstractSyntaxProvider, the following line of code is the culprit.
It doesn't read the characters properly, especially those files who
are ANSI encoded.
using (TextReader reader = new StreamReader(stream))
{
return new SourceContext(reader.ReadToEnd(),
viewSource.LastModified, fileName);
}

Changing it to the following solves my problems.
using (var reader = new StreamReader(stream,Encoding.Default))
{
return new SourceContext(reader.ReadToEnd(),
viewSource.LastModified, fileName);
};

Haven't run any tests to make sure it works across the board, so take
it for what it is. However, in my small manual tests it does work.

Henning

> > spark-dev+...@googlegroups.com<spark-dev%2Bunsubscribe@googlegroups­.com>
> > .

[Henning]

By the way, setting the Encoding to UTF8 didn't work in my case, as
the default in new StreamReader(stream) seems to be UTF8.

Henning

[Asbjørn Ulsberg]

Encoding.Default shouldn't be used, as it will return a different encoding
on various platforms depending on the system default. Instead, it should
be hard coded to something Spark knows how to handle, namely Encoding.UTF8
or Encoding.Unicode.

Please read the caution here:

http://msdn.microsoft.com/en-us/library/system.text.encoding.default.aspx

If you can test with Encoding.UTF8 or Encoding.Unicode and verify that it
works, it would be good for Spark to move away from Encoding.Default. The
absolute best thing for Spark to do would be to use the encoding defined
in <globalization responseEncoding="..." />. This can be read
programmatically through:

http://msdn.microsoft.com/en-us/library/system.web.configuration.globalizationsection.aspx

-Asbjørn

>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

[Asbjørn Ulsberg]

That's probably because Spark later tries to output the same bytes with
Encoding.Default, which on Windows in the western parts of the world is
Windows-1252 which is a bastard encoding (that's even incompatible with
ISO-8859-1) which will mangle the 8-bits characters of UTF-8. Spark should
use Encoding.Unicode, Encoding.UTF8 or
System.Web.Configuration.GlobalizationSection.ResponseEncoding everywhere.

To get proof of this, just do
Console.WriteLine(System.Text.Encoding.Default.CodePage) -- it will yield
"1252". Rendering text on the web in a 15 year old, ISO-incompatible,
7-bit bastard encoding is imho not such a great idea. ;)

More on Windows-1252 here: http://en.wikipedia.org/wiki/Windows-1252

-Asbjørn

>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

[Henning]

Bah, why is this so freaking difficult. Makes me wonder if the world
is out to get me hehe.

My problem is that VS 2010 saves my file in ANSI, whatever encoding
that is. This wreaks havoc with my norwegian characters when I use the
Html Editor for my spark files.
To manage around that problem I've modified spark in two ways.
Firstly, the one I've mentioned in my previous post. I've also set my
output encoding to UTF8. When I do that I get the right output. I also
know that I can use the Advanced Save Options from the File menu to
specify my file encoding, but that seems a bit cumbersome, especially
if everybody has to do it on every spark file.

I've talked about this before, but another possible solution is to use
the Xml Editor in VS 2010, but then I lose all Html intellisense.
Is it possible to create a spark template in SparkSense that saves it
in the right encoding?

Henning

> >> spark-dev+...@googlegroups.com<spark-dev%2Bunsubscribe@googlegroups­.com>

[Henning]

I've been investigating a bit more. When I save my files as UTF8
through the Advanced save options it seems to be working fine.
I'll try to create a project and file templates that I can send you
guys.

Henning

[Rob G]

I'll look into setting the Encoding on the TextBuffer via SparkSense - not sure if I have access to write to it, but I know I can read it. Templates would probably be good anyway because I'll be using T4 to enable "right-click" add view etc.

To unsubscribe from this group, send email to spark-dev+...@googlegroups.com.

[Matt Ellis]

What happens if you set the default editor to HTML with encoding? Does that help any?

[Henning]

Matt,
Not sure. Can't remember if i've tried it, but it might work. I
converted all my files through advanced save options.

Rob,
Sounds good. Will that T4 template generate a view based on the
viewmodel, or just create an empty shell?

On 28 Okt, 15:53, Matt Ellis <m.t.el...@gmail.com> wrote:
> What happens if you set the default editor to HTML with encoding? Does that help any?
>

> On 28 Oct 2010, at 02:35 PM, Rob G <robertgreyl...@gmail.com> wrote:
>
>
>
> > I'll look into setting the Encoding on the TextBuffer via SparkSense - not sure if I have access to write to it, but I know I can read it. Templates would probably be good anyway because I'll be using T4 to enable "right-click" add view etc.
>

> > For more options, visit this group athttp://groups.google.com/group/spark-dev?hl=en.

[Rob G]

I'd like to try and be a clever as possible with it, but not too clever if you know what I mean.

People have very different ideas of what constitutes a "good view", and what about Spark Bindings - do we hook into those? Or Not?

Anyway, it's a tough one, but I'm sure with some good feedback, we can come up with a "best practices" approach for people getting into Spark, and maybe a way to substitute out T4 templates for your own, or even ReSharper Templates perhaps. Anyway, I'm not quite there yet, but will be soliciting feedback when I get close...

[Rob G]

OK - this one is going to need someone to test the result here. I've ensured that Spark now has a default encoding output of UTF8, and as Asbjørn suggested, I've hooked it up to the "system.web/globalization" responseEncoding setting which itself seems to have a default of UTF8 but I can't be sure that's the same in all regions.

Anyway - if someone can give me feedback on this, it would be great...

Regards,

Rob

[Asbjørn Ulsberg]

Great!

I'm pretty sure that UTF-8 is the default for all regions, it is at least
in Norway, which could have had ISO-8859-1 as its default wrt what
characters we "need" in addition to ASCII.

For other regions, I'd say UTF-8 makes even more sense, given their need
for a wider character space and lack of fitting old-school character
encodings. Either way, if another encoding is required, they would have to
adjust this with the responseEncoding in web.config anyway, so Spark
doesn't make this any harder than it has already been by using the same
value.

I would be very surprised if this change has any negative effects for
anyone.

-Asbjørn

>>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

>>> <spark-dev%2Bunsubscribe@googlegroups .com>
>>> > > > > >> > > .
>>> > > > > >> > > For more options, visit this group at
>>> > > > > >> > >http://groups.google.com/group/spark-dev?hl=en.
>>> >
>>> > > --
>>> > > You received this message because you are subscribed to the Google
>>> Groups "Spark View Engine Dev" group.
>>> > > To post to this group, send email to spar...@googlegroups.com.
>>> > > To unsubscribe from this group, send email to

>>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

>>> .
>>> > > For more options, visit this group athttp://
>>> groups.google.com/group/spark-dev?hl=en.
>>> >
>>> > > --

>>> > > You received this message because you are subscribed to the Google
>>> Groups "Spark View Engine Dev" group.
>>> > > To post to this group, send email to spar...@googlegroups.com.
>>> > > To unsubscribe from this group, send email to

>>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>

>>> .
>>> > > For more options, visit this group athttp://
>>> groups.google.com/group/spark-dev?hl=en.
>>>
>>> --

>>> You received this message because you are subscribed to the Google
>>> Groups
>>> "Spark View Engine Dev" group.
>>> To post to this group, send email to spar...@googlegroups.com.
>>> To unsubscribe from this group, send email to

>>> spark-dev+...@googlegroups.com<spark-dev%2Bunsu...@googlegroups.com>