Problem with character encoding (malformed UTF-8 character (ArgumentError))

[Jack Zelig]

Hi,

When I run the following code:

require 'prawn'
require 'prawn/layout'
Prawn::Document.generate("test.pdf") do
data = [[{ :text => "First Name", :font_style => :bold }, "Günther"]]
table data, :font_size => 10, :position => :left, :border_width =>
0, :horizontal_padding => 5, :vertical_padding => 2, :align => { 0
=> :right, 1 => :left }
end

I get this error:
c:/ruby/lib/ruby/gems/1.8/gems/prawn-core-0.5.0.1/lib/prawn/font/
afm.rb:77:in `unpack': malformed UTF-8 character (ArgumentError)

This is being caused by the 'ü' in Günther.
I thought prawn supported utf-8 out of the box.
Could someone give me a clue as to what i am doing wrong or where I
could start reading up on character encoding so that I can solve this
(hopefully) small problem.

I am using ruby 1.8.6 and prawn 0.5.0.1

[Gregory Brown]

It does work out of the box. However, you must a) make sure to set
$KCODE="U" on Ruby 1.8
and b) use a TTF font that is UTF-8 aware.

-greg

[Gregory Brown]

On Jul 23, 2:45 pm, Gregory Brown <gregory.t.br...@gmail.com> wrote:

> It does work out of the box.  However, you must a) make sure to set
> $KCODE="U" on Ruby 1.8
> and b) use a TTF font that is UTF-8 aware.

Whoops, I spoke too soon. Since ü can be transcoded safely to
winansi, you don't need a TTF font.
I think the most likely culprit here is that your editor is producing
files in an encoding other than UTF-8. Can you verify that?

I was able to get your PDF to generate using the default built in
font, but as an unrelated note, if you plan to set font style on
individual cells, it breaks the cell width calculations,
so you need to set a column width explicitly. something
like :column_widths => { 0 => 100 } worked fine for me.

-greg

[Jack Zelig]

> you must a) make sure to set $KCODE="U" on Ruby 1.8
> and b) use a TTF font that is UTF-8 aware

Thanks for the quick reply.

Code now looks like this:

$KCODE = 'u'

require 'prawn'
require 'prawn/layout'
Prawn::Document.generate("test.pdf") do

font "#{Prawn::BASEDIR}/data/fonts/DejaVuSans.ttf"

data = [[{ :text => "First Name", :font_style => :bold }, "Günther"]]
table data, :font_size => 10, :position => :left, :border_width => 0

end

but unfortunately the error remains the same.
Could the problem be with the font?

[Gregory Brown]

On Thu, Jul 23, 2009 at 2:53 PM, Jack Zelig<jack....@googlemail.com> wrote:

> $KCODE = 'u'
> require 'prawn'
> require 'prawn/layout'
> Prawn::Document.generate("test.pdf") do
> font "#{Prawn::BASEDIR}/data/fonts/DejaVuSans.ttf"
> data = [[{ :text => "First Name", :font_style => :bold }, "Günther"]]
> table data, :font_size => 10, :position => :left, :border_width => 0
> end
>
> but unfortunately the error remains the same.
> Could the problem be with the font?

No, I'm pretty sure the problem is with your text editor producing
non-UTF-8 output.

[Jack Zelig]

> Whoops, I spoke too soon. Since ü can be transcoded safely to
> winansi, you don't need a TTF font.
> I think the most likely culprit here is that your editor is producing
> files in an encoding other than UTF-8. Can you verify that?

I'm using SciTE, the editor that comes with the one click ruby
installer.
I just tried recreating the file using Wordpad and running it from the
command line.

Still same error :-(

How might I be sure that my text editor is saving things in utf-8?

[Jack Zelig]

> No, I'm pretty sure the problem is with your text editor producing
> non-UTF-8 output.

Ok, I got it working.
I changed the encoding in SciTE to "UTF-8 Cookie"and that solved the
problem
(Bizarrely UTF-8 produced all manner of errors)
Thanks a lot for your quick help

I am still having one slight problem however:
I am reading applicant's data from a data base.
I want to generate a pdf and save it as #{applicant.last_name}.pdf

Here is the code:

def pdf
@applicants = Applicant.find(:all, :limit => 67)
@applicants.each do |applicant|
Prawn::Document.generate("public/pdfs/new/#
{applicant.last_name}.pdf") do
data = [[{ :text => "Last name", :font_style => :bold }, "#
{applicant.last_name}"]]
table data
end
end
end

My problem is that the filename becomes gibberish if it contains a
special character (e.g. Strötgens), yet in the same file the name
Strötgens is outputted perfectly to the table cell (no problems with
the ö).

Any idea why this might be happening.

Thanks very much for your help so far.

[Chris Schumann]

On Thu, Jul 23, 2009 at 2:22 PM, Jack Zelig<jack....@googlemail.com> wrote:
> My problem is that the filename becomes gibberish if it contains a
> special character (e.g. Strötgens), yet in the same file the name
> Strötgens is outputted perfectly to the table cell (no problems with
> the ö).

It's possible the file system you run expects a different encoding for
file names.

Chris

[Gregory Brown]

Yeah, I think the windows APIs are weird about this...

Jack, is it a possibility to use the iconv stdlib to transcode to
ASCII w. translit?
That'd preserve your 'ö' as an 'o'. (This is not a Prawn problem,
but a Ruby one, I think)

-greg

[Jack Zelig]

> Jack, is it a possibility to use the iconv stdlib to transcode to
> ASCII w. translit?
> That'd preserve your 'ö' as an 'o'. (This is not a Prawn problem,
> but a Ruby one, I think)

Annoyingly, I've now got this working with pdf::writer.
What I did was to add a method to class writer:

CONVERTER = Iconv.new( 'ISO-8859-15//IGNORE//TRANSLIT', 'utf-8')
module PDF
class Writer
def convert(textto)
CONVERTER.iconv(textto)
end
end
end

Now I can do this:
@name = "public/pdfs/" + pdf.convert(@applicant.last_name) + ".pdf"
File.open(@name, "wb") { |f| f.write pdf.render }

and Ströckens saves as 'Ströckens.pdf" as opposed to "Ströckens.pdf"

I'll carry on playing about with things in prawn and post back here if
I have any more questions.

[Gregory Brown]

On Fri, Jul 24, 2009 at 7:27 AM, Jack Zelig<jack....@googlemail.com> wrote:

> I'll carry on playing about with things in prawn and post back here if
> I have any more questions.

To be clear, it doesn't make much sense to add this to the PDF layer
(in PDF::Writer or Prawn)
This is an issue with Ruby's File class.

You can do this trick before passing your filename to Prawn and it
should work fine.

-greg