Version 4.3.4 released

[Pierre Quentel]

Hi,

This new version fixes the bug reported by Nicolas : if Python source
encoding is defined (PEP 0263), the script is opened with this
encoding

Cheers,
Pierre

Download : http://code.google.com/p/karrigell/downloads/detail?name=Karrigell-4.3.4.gz

[Nicolas Pinault]

Hi Pierre,

I have remarks :
- Default encoding used is "ascii", not sys.getdefaultencoding(). But
maybe this is what you expect.
- File handler used to guess encoding is not closed.
- Imported scripts have to be decoded also (_import() function).

Best regards,
Nicolas

[Pierre Quentel]

On 18 oct, 21:47, Nicolas Pinault <nico...@famillepinault.fr> wrote:
> Hi Pierre,> Hi,
>
> > This new version fixes the bug reported by Nicolas : if Python source
> > encoding is defined (PEP 0263), the script is opened with this
> > encoding
>
> > Cheers,
> > Pierre
>

> > Download :http://code.google.com/p/karrigell/downloads/detail?name=Karrigell-4....

>
> I have remarks :
> - Default encoding used is "ascii", not sys.getdefaultencoding(). But
> maybe this is what you expect.

This is the default as defined in PEP 0263

> - File handler used to guess encoding is not closed.
> - Imported scripts have to be decoded also (_import() function).

Good points. I also added the management of the BOM header '\xef\xbb
\xbf'
The changes are included in version 4.3.5, just released
>
> Best regards,
> Nicolas
Thanks,
Pierre

[Nicolas Pinault]

>
> On 18 oct, 21:47, Nicolas Pinault<nico...@famillepinault.fr> wrote:
>> Hi Pierre,> Hi,
>>
>>> This new version fixes the bug reported by Nicolas : if Python source
>>> encoding is defined (PEP 0263), the script is opened with this
>>> encoding
>>> Cheers,
>>> Pierre
>>> Download :http://code.google.com/p/karrigell/downloads/detail?name=Karrigell-4....
>> I have remarks :
>> - Default encoding used is "ascii", not sys.getdefaultencoding(). But
>> maybe this is what you expect.
> This is the default as defined in PEP 0263

I made a mistake, default encoding should be UTF-8.
Please, see PEP3120.

>
>> - File handler used to guess encoding is not closed.
>> - Imported scripts have to be decoded also (_import() function).
> Good points. I also added the management of the BOM header '\xef\xbb
> \xbf'
> The changes are included in version 4.3.5, just released

I have not tested BOM header.
With encoding tag, things are working well now.

Thanks,
Nicolas

[Pierre Quentel]

>
> default encoding should be UTF-8.
> Please, see PEP3120.
>

I missed this one, sorry

>
> I have not tested BOM header.
> With encoding tag, things are working well now.
>
> Thanks,
> Nicolas
>

I'll wait until your test with the BOM header before publishing a new
version

Cordialement,
Pierre

[Nicolas Pinault]

I have generated a script with BOM to do tests.

1) replace
if head == '\xef\xbb\xbf' : # BOM for utf-8
with
if head == b'\xef\xbb\xbf' : # BOM for utf-8
bytes compare to bytes.

2) During my tests I have generated a script with illegal characters. As
a consequence, the BOM detection code block was skipped because of the
try/except. I suggest to decode with the "replace" option. In this case,
� are displayed by the browser. I think this is better than an exception
raised at the end of the source() function saying asccii decoding have
failed. I also suggest to use "replace" option in other instances of
decode() function in source().

Regards,
Nicolas

[Pierre Quentel]

>
> I have generated a script with BOM to do tests.
>
> 1) replace
> if head == '\xef\xbb\xbf' : # BOM for utf-8
> with
> if head == b'\xef\xbb\xbf' : # BOM for utf-8
> bytes compare to bytes.
>

Line 244 of current version of Karrigell.py is indeed :

if head == b'\xef\xbb\xbf': # BOM for utf-8

Where did you find the line without the b ?

> 2) During my tests I have generated a script with illegal characters. As
> a consequence, the BOM detection code block was skipped because of the
> try/except. I suggest to decode with the "replace" option. In this case,
> are displayed by the browser. I think this is better than an exception
> raised at the end of the source() function saying asccii decoding have
> failed. I also suggest to use "replace" option in other instances of
> decode() function in source().

The try/except clause is here mainly in case the source file is empty
or only has one line, so that an exception on readline() doesn't crash
the server. If the BOM header is found and decode() raises an
exception, it is ignored, but raised again in line 265 and caught in
the try/except clause starting at line 155

Not sure about decoding with errors='replace'. Why do you think it
better than raising an exception ?
>
> Regards,
> Nicolas
A+
Pierre

[Nicolas Pinault]

Le 22/10/2011 08:19, Pierre Quentel a �crit :

>> I have generated a script with BOM to do tests.
>>
>> 1) replace
>> if head == '\xef\xbb\xbf' : # BOM for utf-8
>> with
>> if head == b'\xef\xbb\xbf' : # BOM for utf-8
>> bytes compare to bytes.
>>
> Line 244 of current version of Karrigell.py is indeed :
> if head == b'\xef\xbb\xbf': # BOM for utf-8
>
> Where did you find the line without the b ?

Well... It's my mistake.
During my tests, I generated a bad file. The characters where encoded
with CP-1212, while the BOM was UTF-8. This file generated an exception
in Karrigell :

Traceback (most recent call last):
File "C:\Python32\lib\site-packages\Karrigell\Karrigell.py", line 156, in handle_data
self.run(func)
File "C:\Python32\lib\site-packages\Karrigell\Karrigell.py", line 296, in run
src = self.source(self.script_path)
File "C:\Python32\lib\site-packages\Karrigell\Karrigell.py", line 265, in source
src = bfo.read().decode(src_enc)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

I searched a long time why the UTF-8 decoding did not took place.
I tried to replace the if head == b'\xef\xbb\xbf' : statement by another
working statement. Of course, the problem was that my script file was
wrong. But the "except :" catch all statement lead me to the wrong way.
I finaly found the problem and rolled back all my test code.
Unfortunately, I guess the undo stack was not deep enought and I thought
the b was l missing. I should have checked the original file. Sorry for
that.

>
>> 2) During my tests I have generated a script with illegal characters. As
>> a consequence, the BOM detection code block was skipped because of the
>> try/except. I suggest to decode with the "replace" option. In this case,
>> are displayed by the browser. I think this is better than an exception
>> raised at the end of the source() function saying asccii decoding have
>> failed. I also suggest to use "replace" option in other instances of
>> decode() function in source().
> The try/except clause is here mainly in case the source file is empty
> or only has one line, so that an exception on readline() doesn't crash
> the server. If the BOM header is found and decode() raises an
> exception, it is ignored, but raised again in line 265 and caught in
> the try/except clause starting at line 155
>
> Not sure about decoding with errors='replace'. Why do you think it
> better than raising an exception ?

Imagine you write a script, and for any reason (code copied from another
editor or byte string coming from a database with wrong encoding), there
is an encoding error.
Imagine this script is imported by another one.
How much time do you think you will need to find the problem. You will
have an exception like the one above.
- ASCII encoding error -> script is UTF-8 encoded, not ASCII encoded.
- Error at line 265 in Karrigell.py -> The actual error is at line 245.
- You don't know which script is faulty.
- You don't know where is the error in the script.

I propose another solution : Do not use "replace" in the decode()
function but enumerate the script line, catch the DecodeError exception
and raise an exception giving the script name and line number in the
message. This way, the problem is clearly identified.

Here is the code :

class ScriptError(Exception):
pass

...
def source(self,script_path):
# manages encoding of Python source code (PEP 0263 and PEP3120)
src_enc = "utf-8" # Default encoding
bfo = open(script_path,'rb')
try:
head = bfo.read(3)

if head == b'\xef\xbb\xbf': # BOM for utf-8

lines = []
for n,l in enumerate(bfo.readlines()) :
lines.append(l.decode('utf-8'))
src = ''.join(lines)
bfo.close()
return src.replace('\r\n','\n')
except UnicodeDecodeError as exc :
raise ScriptError("Error in file %s at line %d" %
(script_path, n+1)) from exc
except:
pass
bfo.seek(0)
try:
mo = re.search(b"coding[:=]\s*([-\w.]+)",bfo.readline())
if mo:
src_enc = mo.groups()[0].decode('ascii')
else:
try:
mo =
re.search(b"coding[:=]\s*([-\w.]+)",bfo.readline())
if mo:
src_enc = mo.groups()[0].decode('ascii')
except:
pass
bfo.seek(0)
except:
pass
try :
lines = []
for n,l in enumerate(bfo.readlines()) :
lines.append(l.decode(src_enc))
except UnicodeDecodeError as exc :
raise ScriptError("Error in file %s at line %d" %
(script_path, n+1)) from exc
src = ''.join(lines)
bfo.close()
return src.replace('\r\n','\n')
...

Best regards,
Nicolas

>> Regards,
>> Nicolas
> A+
> Pierre
>

[Pierre Quentel]

Salut Nicolas,

Thanks for the proposal. I rewrote the method when I realized that
readline() doesn't raise an exception if there is nothing to read, it
just returns an empty string

With the ScriptError class and your patch the method becomes :

def source(self,script_path):
# manages encoding of Python source code (PEP 0263)
src_enc = "utf-8" # default (PEP 3120)
src = open(script_path,'rb')
head = src.read(3)
if not head == b'\xef\xbb\xbf': # BOM for utf-8
head = ''
src.seek(0)
first2lines = [src.readline(),src.readline()]
for line in first2lines:
mo = re.search(b"coding[:=]\s*([-\w.]+)",line)

if mo:
src_enc = mo.groups()[0].decode('ascii')

break
src.seek(len(head))
try:
lines = []
for n,line in enumerate(src):
lines.append(line.rstrip().decode(src_enc))
src.close()
return '\n'.join(lines)

except UnicodeDecodeError as exc:
raise ScriptError("Error in file %s at line %d" %
(script_path, n+1)) from exc

It works with my test scripts, could you also test it ?

Ciao,
Pierre

[Nicolas Pinault]

Bonsoir Pierre,

> Salut Nicolas,

All my tests where successful !

I have one remark : I just discovered that enumerate() accepts a second
argument, a start value for the index count. This way, we can replace
n+1 by n when printing.

I have one question : Why enumerating a file open in binary mode returns
lines of bytes and not bytes ?

Best regards,
Nicolas