Problem writing some strings (UnicodeEncodeError)

[Paulo da Silva]

Hi!

I am using a python3 script to produce a bash script from lots of
filenames got using os.walk.

I have a template string for each bash command in which I replace a
special string with the filename and then write the command to the bash
script file.

Something like this:

shf=open(bashfilename,'w')
filenames=getfilenames() # uses os.walk
for fn in filenames:
...
cmd=templ.replace("<fn>",fn)
shf.write(cmd)

For certain filenames I got a UnicodeEncodeError exception at
shf.write(cmd)!
I use utf-8 and have # -*- coding: utf-8 -*- in the source .py.

How can I fix this?

Thanks for any help/comments.

[Albert-Jan Roskam]

--------------------------------------------
On Sun, 1/12/14, Paulo da Silva <p_s_d_a_...@netcabo.pt> wrote:

Subject: Problem writing some strings (UnicodeEncodeError)
To: pytho...@python.org
Date: Sunday, January 12, 2014, 4:36 PM

======> what is the output of locale.getpreferredencoding(False)? That is the default value of the "encoding" parameter of the open function.
shf=open(bashfilename,'w', encoding='utf-8') might work, though on my Linux macine locale.getpreferredencoding(False) returns utf-8.
help(open)
...
In text mode, if encoding is not specified the encoding used is platform
dependent: locale.getpreferredencoding(False) is called to get the
current locale encoding. (For reading and writing raw bytes use binary
mode and leave encoding unspecified.)
...

[Peter Otten]

You make it harder to debug your problem by not giving the complete
traceback. If the error message contains 'surrogates not allowed' like in
the demo below

>>> with open("tmp.txt", "w") as f:
... f.write("\udcef")
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in
position 0: surrogates not allowed

you have filenames that are not valid UTF-8 on your harddisk.

A possible fix would be to use bytes instead of str. For that you need to
open `bashfilename` in binary mode ("wb") and pass bytes to the os.walk()
call.

Or you just go and fix the offending names.

[Emile van Sebille]

Not sure exactly, but I'd try


shf=open(bashfilename,'wb')

as a start.

HTH,

Emile

[Paulo da Silva]

That is the situation. I just lost it and it would take a few houres to
repeat the situation. Sorry.

>
> you have filenames that are not valid UTF-8 on your harddisk.
>
> A possible fix would be to use bytes instead of str. For that you need to
> open `bashfilename` in binary mode ("wb") and pass bytes to the os.walk()
> call.

This is my 1st time with python3, so I am confused!

As much I could understand it seems that os.walk is returning the
filenames exactly as they are on disk. Just bytes like in C.

My template is a string. What is the result of the replace command? Is
there any change in the filename from os.walk contents?

Now, if the result of the replace has the replaced filename unchanged
how do I "convert" it to bytes type, without changing its contents, so
that I can write to the bashfile opened with "wb"?

>
> Or you just go and fix the offending names.

This is impossible in my case.
I need a bash script with the names as they are on disk.

[Peter Otten]

No, they are decoded with the preferred encoding. With UTF-8 that can fail,
and if it does the surrogateescape error handler replaces the offending
bytes with special codepoints:

>>> import os
>>> with open(b"\xe4\xf6\xfc", "w") as f: f.write("whatever")
...
8
>>> os.listdir()
['\udce4\udcf6\udcfc']

You can bypass the decoding process by providing a bytes argument to
os.listdir() (or os.walk() which uses os.listdir() internally):

>>> os.listdir(b".")
[b'\xe4\xf6\xfc']

To write these raw bytes into a file the file has of course to be binary,
too.

> My template is a string. What is the result of the replace command? Is
> there any change in the filename from os.walk contents?
>
> Now, if the result of the replace has the replaced filename unchanged
> how do I "convert" it to bytes type, without changing its contents, so
> that I can write to the bashfile opened with "wb"?
>
>
>>
>> Or you just go and fix the offending names.
> This is impossible in my case.
> I need a bash script with the names as they are on disk.

I think instead of the hard way sketched out above it will be sufficient to
specify the error handler when opening the destination file

shf = open(bashfilename, 'w', errors="surrogateescape")

but I have not tried it myself. Also, some bytes may need to be escaped,
either to be understood by the shell, or to address security concerns:

>>> import os
>>> template = "ls <fn>"
>>> for filename in os.listdir():
... print(template.replace("<fn>", filename))
...
ls foo; rm bar

[Paulo da Silva]

>
> I think instead of the hard way sketched out above it will be sufficient to
> specify the error handler when opening the destination file
>
> shf = open(bashfilename, 'w', errors="surrogateescape")

This seems to fix everything!
I tried with a small test set and it worked.

>
> but I have not tried it myself. Also, some bytes may need to be escaped,
> either to be understood by the shell, or to address security concerns:
>

Since I am puting the file names between "", the only char that needs to
be escaped is the " itself.

I'm gonna try with the real thing.

Thank you very much for the fixing and for everything I have learned here.

[Peter Otten]

Paulo da Silva wrote:

>> but I have not tried it myself. Also, some bytes may need to be escaped,
>> either to be understood by the shell, or to address security concerns:
>>
>
> Since I am puting the file names between "", the only char that needs to
> be escaped is the " itself.

What about the escape char?

[Paulo da Silva]

Just this fn=fn.replace('"','\\"')

So far I didn't find any problem, but the script is still running.

[Peter Otten]

To be a bit more explicit:

>>> for filename in os.listdir():
... print(template.replace("<fn>", filename.replace('"', '\\"')))
...
ls "\\"; rm whatever; ls \"

[Peter Otten]

Peter Otten wrote:

> To be a bit more explicit:
>

>>>> for filename in os.listdir():

> ... print(template.replace("<fn>", filename.replace('"', '\\"')))
> ...
> ls "\\"; rm whatever; ls \"

The complete session:

>>> import os
>>> template = 'ls "<fn>"'
>>> with open('\\"; rm whatever; ls \\', "w") as f: pass
...
>>> for filename in os.listdir():

... print(template.replace("<fn>", filename.replace('"', '\\"')))
...
ls "\\"; rm whatever; ls \"

Shell variable substitution is another problem. c.l.py is probably not the
best place to get the complete list of possibilities.

[Paulo da Silva]

I see what you mean.
This is a tedious problem. Don't know if there is a simple solution in
python for this. I have to think about it ...
On a more general and serious application I would not produce a bash
script. I would do all the work in python.

That's not the case, however. This is a few times execution script for a
very special purpose. The only problem was the occurrence of some
Portuguese characters in old filenames encoded in another code than
utf-8. Very few also include the ".

The worst thing that could happen was the bash script to abort. Then it
would be easy to fix it using a simple editor.

[Peter Otten]

I looked around in the stdlib and found shlex.quote(). It uses ' instead of
" which simplifies things, and special-cases only ':

>>> print(shlex.quote("alpha'beta"))
'alpha'"'"'beta'

So the answer is simpler than I had expected.

[Paulo da Silva]

Yes, it should work, at least in this case.
Although python oriented, it seems to work to bash also.
I need to remove the "" from the templates and use shlex.quote for
filenames. I'll give it a try.

Thanks