UTF8 Data

[Ken Collins]

This is interesting. I was testing 2.3.5 with the adapter and noticed this test

# Event.title is limited to 5 characters
e1 = Event.create(:title => "一二三四五")
assert e1.valid?, "Could not create an event with a unique, 5 character title"
e2 = Event.create(:title => "一二三四五六七八")
assert !e2.valid?, "Created an event whose title, with limit taken into account, is not unique"

Even when I changed the column for event title to nvarchar(5), it was still now allowing the insert. I did some research and found this link below. Seems that SQL Server converts the UTF8 data to perhaps UCS=2 which has a different byte length. Has anyone else run into this? I typically do not rely on length validations for national/unicode columns, but wondered if anyone else knew this or could expound on the topic.

http://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=362867

[Nick Tidey]

Hey Ken

You may remember I had issues with this test a while back:

http://groups.google.com/group/rails-sqlserver-adapter/browse_thread/thread/cb37e075abc0c6d7?hl=en#

Adding "client charset = UTF-8" to my freetds.conf server definition
seemed to be the key to fixing the issue for me. What's your server
definition look like?

--
Nick

> http://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?Fee...

[Ken Collins]

I'll try this tomorrow when I'm back at my desk and let you know.

- Ken

> --
>
> You received this message because you are subscribed to the Google Groups "Rails SQLServer Adapter" group.
> To post to this group, send email to rails-sqlse...@googlegroups.com.
> To unsubscribe from this group, send email to rails-sqlserver-a...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/rails-sqlserver-adapter?hl=en.
>
>

[Ken Collins]

OK, seems we were both wrong, here is what this test is all about.
http://github.com/rails-sqlserver/2000-2005-adapter/commit/8d627a9f1b7054863f993ecc6c34965e46887254

The basic gists is this, an adapters quote(value,column=nil) method typically has to rely on having a column object before really forcing certain quote types. It is great that ActiveSupport gives us a way to abstract query a string in 1.8/1.9 for #is_utf8? but we can not really use that as it would cause errors down the way for other SQL that would be quoted that does pass that method and the technical column underneath is not national/unicode. The code snippets shows why the validations where not keeping the orig create object from not saving. Hopefully the rails 3 code and the usage of AREL will allow a better abstraction and I'll have to revisit this then, if anyone has suggestions, I'm all ears.

- Ken


On Dec 1, 2009, at 5:02 PM, Nick Tidey wrote: