Unicode characters il url, again

[Luca De Feo]

Hello,

I am migrating a wiki from gitit 0.8.1 to 0.10.3, and I am
encountering problems with URLs containing non-ascii characters.

Viewing pages works fine, but saving edits or writing cached pages
fails with the following error:

Server error: wikidata/Récursivité.page: openBinaryFile: invalid
argument (invalid character)


I installed gitit with the following command:

cabal install --global -fhighlighting pandoc-1.10.1 gitit-0.10.3

using ghc 7.4.1 and cabal 1.14.0 from officieal debian wheezy repositories.


My guess is that it is a problem related with BinaryString.writeFile
and Codec.Binary.UTF8.String.encodeString), as they appear, for
example, in line 86 of Cache.hs.

Indeed, using ghci, the command

> import Data.ByteString as B
> import Codec.Binary.UTF8.String
> liftIO $ do B.writeFile "é" B.empty

creates an empty file named é, while

> liftIO $ do B.writeFile (encodeString "é") B.empty

creates an empty file named é . I shall precise that my locale is en_US.UTF-8.

Any ideas ?

Thanks,
Luca

[John MacFarlane]

I'm curious, can you reproduce the problem on gitit.net?
That's 0.10.3 compiled with ghc 7.6.1. I didn't see any
problems -- but I also see that caching is disabled.

That's a good sign, as it may indicate that the problem
is JUST with the caching code. I suspect it has to do with
a change in how the base libraries handle encodings of
filenames.

+++ Luca De Feo [Mar 20 13 02:17 ]:

> --
> You received this message because you are subscribed to the Google Groups "gitit-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gitit-discus...@googlegroups.com.
> To post to this group, send email to gitit-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/gitit-discuss?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[John MacFarlane]

OK, I've fixed it. Uploading new version now.

+++ Luca De Feo [Mar 20 13 02:17 ]:

[Luca De Feo]

> I'm curious, can you reproduce the problem on gitit.net?
> That's 0.10.3 compiled with ghc 7.6.1. I didn't see any
> problems -- but I also see that caching is disabled.

I couldn't reproduce it on gitit.net .

> That's a good sign, as it may indicate that the problem
> is JUST with the caching code.

It doesn't only affect caching. Even with caching disabled I am not
able to edit or create pages whose names contain non-ASCII characters.
The error happens when I try to save the page.

I installed the new version, but the problem didn't go away. However,
the following code in ghci

> import Text.Pandoc.UTF8 as U
> import Data.ByteString as B
> liftIO $ do B.writeFile (U.encodePath "€") B.empty

creates a file with the right name, as opposed to the old code using
encodeString. So, there must be another similar problem somewhere.
I'll see if I can debug the code and find the responsible line.

Luca

[John MacFarlane]

Have you tried installing the latest version, which I pushed
to Hackage yesterday?

Also, make sure you are using the latest version of filestore.

cabal update
cabal install filestore gitit

should do the trick.

Let me know if the problem persists.

+++ Luca De Feo [Mar 20 13 17:43 ]:

[Luca De Feo]

> Have you tried installing the latest version, which I pushed
> to Hackage yesterday?

I installed from the git sources.

> Also, make sure you are using the latest version of filestore.

I'm using 0.6. cabal says it's the latest

> cabal update
> cabal install filestore gitit

I tried, but cabal says

All the requested packages are already installed:
filestore-0.6
gitit-0.10.3.1
Use --reinstall if you want to reinstall anyway.


I forced reinstall, just in case, but nothing changed. I just noticed
that the error message has changed slightly (probably it had changed
even before the last reinstall):

Server error: cache/Récursvité: openBinaryFile: invalid argument
(invalid character)

(non-ASCII characters are correctly rendered in the error message now).

Luca

[John MacFarlane]

What versions of base and directory is it compiled against?

+++ Luca De Feo [Mar 20 13 18:36 ]:

[Luca De Feo]

> What versions of base and directory is it compiled against?

base: 4.5.0.0
directory: 1.1.0.2

both from the standard debian wheezy packages

Luca

[Luca De Feo]

I tried to create a fresh wiki, just to see what happens, and...
everything works fine, including caching!

So, it seems to be a problem specific to the migration from 0.8.1.

This is driving me crazy!

Luca

[Luca De Feo]

It got even crazier!

I'm using daemontools to start and keep the wiki running, and I
realized that this is responsible for the problem.

Removing the 'supervise' directory and restarting the service
sometimes fixes the problem, but not consistently. Rebooting the box
didn't do any good.

I stopped daemontools and started the wiki manually. Now everything
seems to be fine.

Summarizing, the issue seems to be related to reading/writing a file
whose name contains non-ASCII characters (whether in cache or
wikidata). I don't know what daemontools has to do with that...

Luca




On Thu, Mar 21, 2013 at 1:16 AM, Luca De Feo