How to stop "invalid byte sequence in US-ASCII"?

[Peter Empen]

While browsing, my nanoc view console gets full of the following messages:

ERROR ArgumentError: invalid byte sequence in US-ASCII
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/stream_cleaners/utf8.rb:11:in `gsub'
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/stream_cleaners/utf8.rb:11:in `clean'
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/cleaning_stream.rb:105:in `block in clean`
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/cleaning_stream.rb:105:in `each'
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/cleaning_stream.rb:105:in `inject'
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/cleaning_stream.rb:105:in `clean'
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/cleaning_stream.rb:49:in `<<'
ruby/1.9.1/webrick/httpserver.rb:213:in `block in access_log'
ruby/1.9.1/webrick/httpserver.rb:212:in `each'
ruby/1.9.1/webrick/httpserver.rb:212:in `access_log'
ruby/1.9.1/webrick/httpserver.rb:111:in `run'
ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'

I suspect this is due to non UTF-8 encoding but I was not able to configure Nanoc to generate UTF-8 on Windows. It generates ANSI from UTF-8 sources even if I add

data_sources:
    config:
      encoding: utf-8

to config.yaml. How would you suggest me to resolve these errors?
Peter

[Denis Defreyne]

Hi Peter,

This is an odd issue. I’m not sure what is going wrong exactly, although I suspect WEBrick is passing ASCII data to nanoc, which expects UTF-8 everywhere. Setting the data source encoding won’t help here (even though it is a good thing to do).

Can you try changing the default encoding for cmd.exe to UTF-8? This explains how: http://superuser.com/a/269857.

Cheers,

Denis

> --
> --
> You received this message because you are subscribed to the nanoc discusssion group.
>
> To post to this group, send email to na...@googlegroups.com
> To unsubscribe from this group, send email to
> nanoc+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nanoc?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "nanoc" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nanoc+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Peter Empen]

Hi Denis,
I tried that command line processor setting. On cmd start up I get the message Aktive Codepage: 65001 so it is active but nanoc terminates under this new condition immediately without any output. It doesn't even generates the output.
Shall I generate a trace for you and if so how?
Peter

[Bill Burton]

Hello,

I did a little investigation and found the following:

Try the following from your console window:

    ruby -e 'puts "String encoding: #{__ENCODING__.name}"'

    ruby -e 'puts "Default external encoding: #{Encoding.default_external}"'

Both commands probably return: IBM437 or other code page.

Then execute this command:

    set LC_CTYPE=en_US.UTF-8

Re-execute the above two ruby commands.

Both commands should output: UTF-8

Of course, you can replace the, "en_US" part of the above setting with whatever is appropriate for your language and country.

You may want to run nanoc compile first to ensure the pages are compiled properly and then run nanoc view.

Hope this helps,

-Bill

[Peter Empen]

Hi Bill, so I did:

>ruby -e 'puts "String encoding: #{__ENCODING__.name}"'

String encoding: CP850

>ruby -e 'puts "Default external encoding: #{Encoding.default_external}"'

Default external encoding: CP850
>set LC_CTYPE=en_US.UTF-8

>ruby -e 'puts "String encoding: #{__ENCODING__.name}"'

String encoding: UTF-8

>ruby -e 'puts "Default external encoding: #{Encoding.default_external}"'

Default external encoding: UTF-8
>nanoc compile

At this point nanoc still generates ANSI encoded output and, consequently, on the nanoc view console the same ugly messages appear..:(.

Peter

[Peter Empen]

I must correct myself a little bit: The messages did change with respect to the trace from

ERROR ArgumentError: invalid byte sequence in US-ASCII
gems/1.9.1/gems/nanoc-3.5.0/lib/nanoc/cli/stream_cleaners/utf8.rb:11:in `gsub'

...
to

ERROR ArgumentError: invalid byte sequence in US-ASCII

gems/1.9.1/gems/win32console-1.3.2-x86-mingw32/lib/Win32/Console/ANSI.rb:163:in `sub!'
...

[Denis Defreyne]

Hi Peter,

It seems that win32console is at fault here. Could you try with ANSICON, a modern replacement for win32console? You can find it at https://github.com/adoxa/ansicon.

Alternatively, you could use bash from cygwin/msys (I do it this way when I’m on Windows).

Denis

[Peter Empen]

Hi Denis,

apparently, ANSICON doesn't do it better.

Before, I also implemented a batch converting all text files (html, css, js) in the output folder to UTF-8. Why do I get the same error messages even using UTF-8 files? Are these messages real errors? What is the real cause of these outputs?

Thanks,
Peter

[Denis Defreyne]

Hi Peter,

Do you have any filenames with non-ASCII characters in them? That seems to be the cause of the issue.

Denis

--

[Peter Empen]

Hi Denis,

Do you have any filenames with non-ASCII characters in them? That seems to be the cause of the issue.

No, I don't. You may have a look at the nanoc project, everything in English, at https://github.com/scala-graph/scala-graph-site.

Thanks,
Peter