Issue 73 in contentmirror: Plone 4 contentmirror has problems with umlaut type of characters like in this word: geïnstalleerd

[conten...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 73 by p.j.m.stamps: Plone 4 contentmirror has problems with
umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

When i change geïnstalleerd (Dutch for installed) into geinstalleerd the
contentmirror works fine...

See error message? But where and how to solve this issue?



sqlalchemy.exc.ProgrammingError: (ProgrammingError) You must not use 8-bit
bytestrings unless you use a text_factory that can interpret 8-bit
bytestrings (liketext_factory = str). It is highly recommended that you
instead just switch your application to Unicode strings. u'INSERT INTO
content (id, content_uid, object_type, status, portal_type,
folder_position, container_id, path, relative_path, title, description,
subject, location, contributors, creators, creation_date,
modification_date, effectivedate, expirationdate, language, rights,
excludefromnav) VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'
['front-page', 'fc76a6f234293051d32b210cd322604e', 'ATDocumentPeer', 'published', 'Document',
59, None, '/Plone/front-page', 'front-page', 'Welkom bij
Plone', 'Gefeliciteerd! Plone is succesvol
ge\xc3\xafnstalleerd.', '', '', '', 'admin', '2010-08-31
22:56:34.812000', '2010-08-31 22:56:35.234000', None, None, 'nl', '', 0]

[conten...@googlecode.com]

Comment #1 on issue 73 by jean.jordaan: Plone 4 contentmirror has problems

with umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

You can switch your application to Unicode strings. I.e. use only Unicode
in your application.

See http://docs.python.org/howto/unicode.html for starters.

[conten...@googlecode.com]

Comment #2 on issue 73 by p.j.m.st...@gmail.com: Plone 4 contentmirror has

problems with umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

Dear Jean,

Thanks for you feedback.

Unfortunately I am using the default functionality of Plone 4 and
contentmirror. I am not the owner of these apps.
I tried to discover in the source code if there was an option to influence
the "unicode" behaviour. But i have no clue where to look for a solution.

The only solution I found was NOT to use ï, á, ú, ë and the likes. Then
everything works fine.
I think the problem might be in SQLAlchemy or in contentmirror itself.
Probably when the software formats the INSERT it should be able to detect
if encoding is needed?



BTW: When I followed the published installation of contentmirror for a
Plone 4 installation then it did not work.

With the following Buildout and ZCML configuration all did work immediately.
--------------------
settings.zcml
--------------------
<configure xmlns="http://namespaces.zope.org/zope"
xmlns:ore="http://namespaces.objectrealms.net/mirror">

<!-- setup a database connection -->
<ore:engine url="sqlite:///c:/plone4/contentmirror.sqlite"
name="mirror-db"
echo="True"/>

<!-- bind the database connection to the contentmirror rdb schema -->
<ore:bind engine="mirror-db" metadata="ore.contentmirror.schema.metadata"/>

</configure>

----------------------
configure.zcml
---------------------
<configure xmlns="http://namespaces.zope.org/zope">

<include file="base.zcml"/>

<!-- Plone Meta -->
<include file="meta.zcml" />

<!-- Plone Settings -->
<include file="settings.zcml" />

<!-- Implements Declarations -->
<include file="implements.zcml"/>

<!-- Archetypes Dependency -->
<include package="Products.Archetypes"/>

<!-- Plone Default Content -->
<include file="content.zcml" />

</configure>

----------------
buildout.cfg
----------------

eggs =
.....
.....
ore.contentmirror


zcml =
.....
.....
ore.contentmirror

[conten...@googlecode.com]

Comment #3 on issue 73 by jean.jordaan: Plone 4 contentmirror has problems

with umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

> I think the problem might be in SQLAlchemy or in contentmirror itself.

The only change contentmirror could make would be to guess at the encoding.
This will sometimes fail. It's much better to be explicit about the
encoding.

> Unfortunately I am using the default functionality of Plone 4

In that case, you should try posting your issue to the Plone Users mailing
list. Plone is supposed to be using unicode internally, so you need to find
out how non-unicode content is making its way into Plone.

[conten...@googlecode.com]

Comment #4 on issue 73 by kapilt: Plone 4 contentmirror has problems with

umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

I just wanted to echo jean's comment that afaics this issue is about
misconfiguration of plone. Contentmirror does handle unicode and does have
several tests to that effect with russian and simplified chinese
characters. The only safe/sane place to interpret encoding and normalize
the byte strings to unicode is the application tier that accepts and
displays inputs, ie. plone. The storage layer can't safely make such
guesses.

http://plone.org/support/forums/i18n

[conten...@googlecode.com]

Comment #5 on issue 73 by develo...@acsr.de: Plone 4 contentmirror has

problems with umlaut type of characters like in this word: geïnstalleerd
http://code.google.com/p/contentmirror/issues/detail?id=73

For MySQL adding the suffix "?charset=utf8"to the engine url in the
settings can fix a lot!

example:

<ore:engine
url="mysql://[user]:[pwd]@localhost:3306/[database]?charset=utf8"

name="mirror-db"
echo="True" />

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings