[Python-Dev] [compatibility-sig] making sure importlib.machinery.SourceLoader doesn't throw an exception if bytecode is not supported by a VM

[Brett Cannon]

I would like to have importlib just work out of the box for all VMs in 3.3 instead of requiring a minor patch in order to not throw an exception when loading from source and there is no bytecode. The relevant code for this discussion can be seen at http://hg.python.org/cpython/file/c2910971eb86/Lib/importlib/_bootstrap.py#l691 .

First question is what are all the VMs doing for imp.cache_from_source()? Are you implementing it like CPython, or are you returning None? And if you implemented it, what does marshal.loads() do? Right now cache_from_source() is implemented in importlib itself, but we can either provide a flag to control what it does or in your setup code for import you can override the function with ``lambda _, __=None: None``.

Second question, what do you set sys.dont_write_bytecode to?

The answers to those questions will dictate if there is anything to actually discuss. =)

[Alex Gaynor]

For PyPy: I'm not an expert in our import, but from looking at the source

1) imp.cache_from_source is unimplemented, it's an AttributeError.

2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
should IMO, but it's not a high priority for me, or anyone else apparently)

Alex

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Brett Cannon]

On Tue, Jun 12, 2012 at 12:38 PM, Alex Gaynor <alex....@gmail.com> wrote:

For PyPy: I'm not an expert in our import, but from looking at the source

1) imp.cache_from_source is unimplemented, it's an AttributeError.

Well, you will have it come Python 3.3 one way or another. =)

 

2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
  should IMO, but it's not a high priority for me, or anyone else apparently)

But doesn't PyPy read and write .pyc files (http://doc.pypy.org/en/latest/config/objspace.usepycfiles.html suggests you do)? So I would assume you are not affected by this. Jython and IronPython, though, would be (I think).

[Brett Cannon]

On Tue, Jun 12, 2012 at 10:52 AM, Brett Cannon <br...@python.org> wrote:

I would like to have importlib just work out of the box for all VMs in 3.3 instead of requiring a minor patch in order to not throw an exception when loading from source and there is no bytecode. The relevant code for this discussion can be seen at http://hg.python.org/cpython/file/c2910971eb86/Lib/importlib/_bootstrap.py#l691 .

First question is what are all the VMs doing for imp.cache_from_source()? Are you implementing it like CPython, or are you returning None? And if you implemented it, what does marshal.loads() do? Right now cache_from_source() is implemented in importlib itself, but we can either provide a flag to control what it does or in your setup code for import you can override the function with ``lambda _, __=None: None``.

I should mention another option is to add sys.dont_read_bytecode (I think I have discussed this with Frank at some point).

-Brett

[Alex Gaynor]

On Tue, Jun 12, 2012 at 11:47 AM, Brett Cannon <br...@python.org> wrote:

On Tue, Jun 12, 2012 at 12:38 PM, Alex Gaynor <alex....@gmail.com> wrote:

For PyPy: I'm not an expert in our import, but from looking at the source

1) imp.cache_from_source is unimplemented, it's an AttributeError.

Well, you will have it come Python 3.3 one way or another. =)

 

Sure, I'm not totally up to speed on the py3k effort.

 

2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
  should IMO, but it's not a high priority for me, or anyone else apparently)

But doesn't PyPy read and write .pyc files (http://doc.pypy.org/en/latest/config/objspace.usepycfiles.html suggests you do)? So I would assume you are not affected by this. Jython and IronPython, though, would be (I think).

This is a compile time option, not a runtime option.  However, it looks like I lied, someone did implement it correctly, so we have the same behavior as CPython.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

[Amaury Forgeot d'Arc]

2012/6/12 Alex Gaynor <alex....@gmail.com>

On Tue, Jun 12, 2012 at 11:47 AM, Brett Cannon <br...@python.org> wrote:

On Tue, Jun 12, 2012 at 12:38 PM, Alex Gaynor <alex....@gmail.com> wrote:

For PyPy: I'm not an expert in our import, but from looking at the source

1) imp.cache_from_source is unimplemented, it's an AttributeError.

Well, you will have it come Python 3.3 one way or another. =)

 

Sure, I'm not totally up to speed on the py3k effort.

It's indeed implemented in pypy's py3k branch.

 

 

2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
  should IMO, but it's not a high priority for me, or anyone else apparently)

But doesn't PyPy read and write .pyc files (http://doc.pypy.org/en/latest/config/objspace.usepycfiles.html suggests you do)? So I would assume you are not affected by this. Jython and IronPython, though, would be (I think).

This is a compile time option, not a runtime option.  However, it looks like I lied, someone did implement it correctly, so we have the same behavior as CPython.

Yes, PyPy seems to respect sys.dontwritebytecode.

--
Amaury Forgeot d'Arc

[fwier...@gmail.com]

On Tue, Jun 12, 2012 at 9:38 AM, Alex Gaynor <alex....@gmail.com> wrote:
> For PyPy: I'm not an expert in our import, but from looking at the source
>
> 1) imp.cache_from_source is unimplemented, it's an AttributeError.

Jython does not (yet) have a cache_from_source.

> 2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
>   should IMO, but it's not a high priority for me, or anyone else apparently)

Jython does support sys.dont_write_bytecode, but doesn't support
sys.dont_read_bytecode yet - do you happen to know when
dont_read_bytecode was added? It should be pretty straightforward, and
so I'll probably add it to our 2.7.

-Frank

[fwier...@gmail.com]

On Tue, Jun 12, 2012 at 10:04 AM, fwier...@gmail.com
<fwier...@gmail.com> wrote:
> Jython does support sys.dont_write_bytecode, but doesn't support
> sys.dont_read_bytecode yet - do you happen to know when
> dont_read_bytecode was added? It should be pretty straightforward, and
> so I'll probably add it to our 2.7.

Looking around it seems dont_read_bytecode came in sometime in 3.x so
I'll wait for 3 (and so I'll probably just use importlib....? :)

[Brett Cannon]

On Tue, Jun 12, 2012 at 1:04 PM, fwier...@gmail.com <fwier...@gmail.com> wrote:

On Tue, Jun 12, 2012 at 9:38 AM, Alex Gaynor <alex....@gmail.com> wrote:
> For PyPy: I'm not an expert in our import, but from looking at the source
>
> 1) imp.cache_from_source is unimplemented, it's an AttributeError.

Jython does not (yet) have a cache_from_source.

> 2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
>   should IMO, but it's not a high priority for me, or anyone else apparently)

Jython does support sys.dont_write_bytecode, but doesn't support
sys.dont_read_bytecode yet - do you happen to know when
dont_read_bytecode was added?

It was never added since it doesn't currently exist; I said *add* sys.dont_read_bytecode, not *use*. =)

Would the flag actually be beneficial? Have you had issues where people assumed that Jython should be able to read bytecode since they just didn't worry about a VM that can't read bytecode? I mean I'm open to suggestions as ways to control bytecode reading to fail gracefully in case someone runs Jython (or IronPython) in a directory where PyPy or CPython was run previously and thus the bytecode exists from them. That's why I asked what marshal.loads() does; if it returns None or raises some exception that can be distinguished from a failure of badly formatted marshal data then I could rely on that as another option.

 

It should be pretty straightforward, and
so I'll probably add it to our 2.7.

I would assume it would just be a flag for you. Importlib and other stdlib code would be where all the work would be to start obeying the flag (if it is added).

[Eric Snow]

On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
> I should mention another option is to add sys.dont_read_bytecode (I think I
> have discussed this with Frank at some point).

Or check for "sys.implementation.cache_tag is None"...

-eric

[Brett Cannon]

On Tue, Jun 12, 2012 at 2:28 PM, Eric Snow <ericsnow...@gmail.com> wrote:

On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
> I should mention another option is to add sys.dont_read_bytecode (I think I
> have discussed this with Frank at some point).

Or check for "sys.implementation.cache_tag is None"...

Perfect! Will that work for Jython (Franke) and IronPython (Jeff)?

This does mean, though, that imp.cache_from_source() and imp.source_from_cache() might need to be updated to raise a reasonable exception when sys.implementation.cache_tag is set to None as I believe right now it will raise a TypeError because None isn't a str. But what to raise instead? TypeError? EnvironmentError?

[Jeff Hardy]

> 1) imp.cache_from_source is unimplemented, it's an AttributeError.

Same for IronPython.

>
> 2) sys.dont_write_bytecode is always false, we don't respect that flag (we really
>   should IMO, but it's not a high priority for me, or anyone else apparently)

Always True for IronPython. You can change it, but it doesn't affect anything.

- Jeff

[Jeff Hardy]

On Tue, Jun 12, 2012 at 12:01 PM, Brett Cannon <br...@python.org> wrote:
> On Tue, Jun 12, 2012 at 2:28 PM, Eric Snow <ericsnow...@gmail.com>
> wrote:
>>
>> On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
>> > I should mention another option is to add sys.dont_read_bytecode (I
>> > think I
>> > have discussed this with Frank at some point).
>>
>> Or check for "sys.implementation.cache_tag is None"...
>
>
> Perfect! Will that work for Jython (Franke) and IronPython (Jeff)?

IronPython will probably never *write* pyc files, but it might *read*
them at some point -- as I understand cache_tag, we'd set it to
whatever version of CPython's pyc files we could read (that seems to
violate the spirit of sys.implementation). The combination of that and
sys.dont_write_bytecode should cover all of the states; I'll just lock
down sys.dont_write_bytecode so that changes are completely ignored.

>
> This does mean, though, that imp.cache_from_source() and
> imp.source_from_cache() might need to be updated to raise a reasonable
> exception when sys.implementation.cache_tag is set to None as I believe
> right now it will raise a TypeError because None isn't a str. But what to
> raise instead? TypeError? EnvironmentError?

NotImplementedError?

- Jeff

[Brett Cannon]

On Tue, Jun 12, 2012 at 3:53 PM, Jeff Hardy <jdh...@gmail.com> wrote:

On Tue, Jun 12, 2012 at 12:01 PM, Brett Cannon <br...@python.org> wrote:
> On Tue, Jun 12, 2012 at 2:28 PM, Eric Snow <ericsnow...@gmail.com>
> wrote:
>>
>> On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
>> > I should mention another option is to add sys.dont_read_bytecode (I
>> > think I
>> > have discussed this with Frank at some point).
>>
>> Or check for "sys.implementation.cache_tag is None"...
>
>
> Perfect! Will that work for Jython (Franke) and IronPython (Jeff)?

IronPython will probably never *write* pyc files, but it might *read*
them at some point -- as I understand cache_tag, we'd set it to
whatever version of CPython's pyc files we could read (that seems to
violate the spirit of sys.implementation).

If you wanted to sneak around not writing your own bytecode but still reading it, then yes, you could do that. And yes, that does somewhat violate the concept of what sys.implementation.cache_tag was supposed to do. =)

 

The combination of that and

sys.dont_write_bytecode should cover all of the states; I'll just lock
down sys.dont_write_bytecode so that changes are completely ignored.

Great!

 

>
> This does mean, though, that imp.cache_from_source() and
> imp.source_from_cache() might need to be updated to raise a reasonable
> exception when sys.implementation.cache_tag is set to None as I believe
> right now it will raise a TypeError because None isn't a str. But what to
> raise instead? TypeError? EnvironmentError?

NotImplementedError?

That's also a possibility. I also realized that importlib checks for None being returned by cache_from_source(), so that's another option (which ties into how the rest of the PEP 302 methods work).

[fwier...@gmail.com]

On Tue, Jun 12, 2012 at 12:01 PM, Brett Cannon <br...@python.org> wrote:
>
>
> On Tue, Jun 12, 2012 at 2:28 PM, Eric Snow <ericsnow...@gmail.com>
> wrote:
>>
>> On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
>> > I should mention another option is to add sys.dont_read_bytecode (I
>> > think I
>> > have discussed this with Frank at some point).
>>
>> Or check for "sys.implementation.cache_tag is None"...
>
>
> Perfect! Will that work for Jython (Franke) and IronPython (Jeff)?

So Jython does actually emit bytecodes, but they are Java bytecodes
instead of Python bytecodes. Right now they end up next to the .py
files just like .pyc files. They have the possibly unfortunate naming
foo.py -> foo$py.class -- If I understand cache_tag (I may not) I
guess Python 3 is putting the .pyc files into hidden subdirectories
instead of putting them next to the .py files? If so we may do the
same with our $py.class files.

Incidentally we also have a mode for reading .pyc files -- though we
haven't implementing writing them yet (we probably will eventually)

I guess what I'm trying to say is that I don't know exactly how we
will handle these new flags, but chances are we will use them (Again
provided my guesses about what they do are anywhere near what they
really do).

>
> This does mean, though, that imp.cache_from_source() and
> imp.source_from_cache() might need to be updated to raise a reasonable
> exception when sys.implementation.cache_tag is set to None as I believe
> right now it will raise a TypeError because None isn't a str. But what to
> raise instead? TypeError? EnvironmentError?

NotImplementedError seems fine for me too if we don't end up using this flag.

-Frank

[Brett Cannon]

On Jun 12, 2012 6:24 PM, "fwier...@gmail.com" <fwier...@gmail.com> wrote:
>
> On Tue, Jun 12, 2012 at 12:01 PM, Brett Cannon <br...@python.org> wrote:
> >
> >
> > On Tue, Jun 12, 2012 at 2:28 PM, Eric Snow <ericsnow...@gmail.com>
> > wrote:
> >>
> >> On Tue, Jun 12, 2012 at 10:48 AM, Brett Cannon <br...@python.org> wrote:
> >> > I should mention another option is to add sys.dont_read_bytecode (I
> >> > think I
> >> > have discussed this with Frank at some point).
> >>
> >> Or check for "sys.implementation.cache_tag is None"...
> >
> >
> > Perfect! Will that work for Jython (Franke) and IronPython (Jeff)?
> So Jython does actually emit bytecodes, but they are Java bytecodes
> instead of Python bytecodes. Right now they end up next to the .py
> files just like .pyc files. They have the possibly unfortunate naming
> foo.py -> foo$py.class -- If I understand cache_tag (I may not) I
> guess Python 3 is putting the .pyc files into hidden subdirectories
> instead of putting them next to the .py files?

Yes, __pycache__. The tag is to allow different versions of bytecode to exist side-by-side (eg for CPython 3.3 it's cpython33 so the file ends up being named foo-cpython33.pyc).

If so we may do the
> same with our $py.class files.

That was part of the hope when it was designed.

>
> Incidentally we also have a mode for reading .pyc files -- though we
> haven't implementing writing them yet (we probably will eventually)

If you can read .pyc files then you should be fine.

>
> I guess what I'm trying to say is that I don't know exactly how we
> will handle these new flags, but chances are we will use them (Again
> provided my guesses about what they do are anywhere near what they
> really do).

IOW it's too soon to be having this discussion. :) I mean regardless of what happens you can always tweak the import lib code as necessary, I just wanted to try to avoid it.

>
> >
> > This does mean, though, that imp.cache_from_source() and
> > imp.source_from_cache() might need to be updated to raise a reasonable
> > exception when sys.implementation.cache_tag is set to None as I believe
> > right now it will raise a TypeError because None isn't a str. But what to
> > raise instead? TypeError? EnvironmentError?
> NotImplementedError seems fine for me too if we don't end up using this flag.

OK, that's 2 votes for that exception.

>
> -Frank

[Nick Coghlan]

On Wed, Jun 13, 2012 at 11:10 AM, Brett Cannon <br...@python.org> wrote:
>> > This does mean, though, that imp.cache_from_source() and
>> > imp.source_from_cache() might need to be updated to raise a reasonable
>> > exception when sys.implementation.cache_tag is set to None as I believe
>> > right now it will raise a TypeError because None isn't a str. But what
>> > to
>> > raise instead? TypeError? EnvironmentError?
>> NotImplementedError seems fine for me too if we don't end up using this
>> flag.
>
> OK, that's 2 votes for that exception.

+ 1 from me as well, both for skipping any implicit reading or writing
of the cache when cache_tag is None (IIRC, that's the use case we had
in mind when we allowed that field to be None in the PEP 421
discussion), and for *explicit* attempts to access the cache when the
tag is None triggering NotImplementedError.

That way people are free to use either LBYL (checking cache_tag) or
EAFP (catching NotImplementedError).

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Brett Cannon]

On Wed, Jun 13, 2012 at 1:18 AM, Nick Coghlan <ncog...@gmail.com> wrote:

On Wed, Jun 13, 2012 at 11:10 AM, Brett Cannon <br...@python.org> wrote:
>> > This does mean, though, that imp.cache_from_source() and
>> > imp.source_from_cache() might need to be updated to raise a reasonable
>> > exception when sys.implementation.cache_tag is set to None as I believe
>> > right now it will raise a TypeError because None isn't a str. But what
>> > to
>> > raise instead? TypeError? EnvironmentError?
>> NotImplementedError seems fine for me too if we don't end up using this
>> flag.
>
> OK, that's 2 votes for that exception.

+ 1 from me as well, both for skipping any implicit reading or writing
of the cache when cache_tag is None (IIRC, that's the use case we had
in mind when we allowed that field to be None in the PEP 421
discussion), and for *explicit* attempts to access the cache when the
tag is None triggering NotImplementedError.

That way people are free to use either LBYL (checking cache_tag) or
EAFP (catching NotImplementedError).

I'm sold: http://bugs.python.org/issue15056 for tracking the change.