f = urllib.request.urlopen(url)
    text = f.read()

then "text" is a bytes object, not a string, and so I can't do a
regexp on it.

Of course, this is easy to patch: just do "f.read().decode()".
However, it strikes me as an obvious bug, which ought to be fixed.
That is, read() should return a string, as it did in py2.5.

But apparently others disagree? This was mentioned in issue 3930
( http://bugs.python.org/issue3930 ) back in September '08, but that
issue is now closed, apparently because consistent behavior was
achieved. But I figure consistently bad behavior is still bad.

This change breaks pretty much every Python program that opens a
webpage, doesn't it? 2to3 doesn't catch it, and, in any case, why
should read() return bytes, not string? Am I missing something?

HOWEVER... a web page request often does know what encoding to use,
since it ostensibly has to parse the header.  It's reasonable that IF
a url request's "Content-type" is text, and/or the "Content-encoding"
is given, for urllib to have an option to automatically decode and
return a string instead of bytes.  (For all I know, it already can do
that.)

On Dec 22, 1:41 pm, ajaksu <aja...@gmail.com> wrote:

Consider: reading a web page is an easy one-liner. Now, no one is
going to write that one-liner, and then spend 20 lines trying to get
the Content-Type and encoding figured out. Instead we're all going to
do it the short, easy, *wrong* way. So every program in the world that
uses urlopen gets to have the same bug. Not good. The *right* way
needs to be the *easy* way.

Everybody in the whole world has to deal with unicode *unless* you are
living in USA and all you have is plain and simple ASCII text. Python
2.x makes no difference between text in ASCII and arbitrary bytes. Both
are stored in the str type. This makes it easy for ASCII country but the
rest of the world suffers the consequences.

Python 3.0 makes a hard break for ASCII people because with 3.0 really
everybody has to deal with encodings. There is no more implicit
conversion between ASCII text and unicode.
http://www.joelonsoftware.com/articles/Unicode.html explains it in great
detail.

I suggest somebody makes a feature request for 3.1. A patch with unit
test increases the changes for the patch by at least one magnitude.

Glenn, does this sketch work for you?

def urlopen_text(url, data=None,
timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    response = urlopen(url, data, timeout)
    _readline = response.readline
    _readlines = response.readlines
    _read = response.read
    charset = response.headers.get_charsets()[0]
    def readline(limit = -1):
        content = _readline()
        return str(content, encoding=charset)
    response.readline = readline
    def readlines(hint = None):
        content = _readlines()
        return [str(line, encoding=charset) for line in content]
    response.readlines = readlines
    def read(n = -1):
        content = _read()
        return str(content, encoding=charset)
    response.read = read
    return response

Any comments/suggestions are very welcome. I could use some help from
people that know urllib on the best way to get the charset. Maybe
after some sleep I can code it in a less awful way :)

I think a neat improvement, besides better docs, names, code, etc.,
would be to implement Carl Bank's idea of checking Content-Encoding.

Oh, and there's a 'print("Using default charset '%s'" %
response.charset)' in there that seemed educational when I wrote it,
but now sounds lame :)

Glenn, can you test the patch[1] there?

Thanks for the encouragement, Chris! I hope we don't regret it :D

Daniel