cython support for stub files - alternative to pure mode

[Denis Akhiyarov]

Does cython support the stub files, which is an alternative to pure mode?

Here is the background info:

https://www.python.org/dev/peps/pep-0484/#stub-files

http://www.mypy-lang.org/wiki/CreatingStubsForPythonModules

[Denis Akhiyarov]

So pxd files look like stub files:

http://docs.cython.org/src/tutorial/pxd_files.html

[Stefan Behnel]

Not currently. In fact, PEP 484 is mostly uninteresting from the point of
view of a static compiler. Essentially, what it allows users to do is to
state things like "this parameter must receive an iterable" or "this
variable must contain a Python sequence". But that's usually obvious when
looking at the code. If the code that Cython compiles iterates over an
object, then that object must obviously be iterable, whether it was
statically declared upfront or not. And if it's not obvious from the code,
then the declaration is entirely irrelevant in the first place.

PEP 484 doesn't even have a way to specify that a variable contains
*exactly* a given type and not a subtype. Without that, it doesn't matter
if you provide a static declaration or not, because anything that Cython
could optimise for could be overwritten by subtypes. So you would still
need the type check that Cython generates already.

The only interesting bit in PEP 484 is that is also supports declarations
of item types in containers, which Cython doesn't currently allow. But then
again, the item types that you could specify there would still not be
helpful for Cython.

Basically, the sole purpose of PEP 484 is to provide helpful hints to IDEs
and static code checkers ("Look! My IDE knows that an integer is not a
sequence and I don't even have to write a test for it!"). It's explicitly
excluding support for anything that goes beyond that.

Stefan

[Stefan Behnel]

Denis Akhiyarov schrieb am 08.05.2015 um 09:43:
> So pxd files look like stub files:
>
> http://docs.cython.org/src/tutorial/pxd_files.html

Sort of, yes. pxd files allow Cython code to share and reuse declarations
for external code across modules, to export C-level APIs to other Cython
modules, and to provide static declarations for Cython or Python code that
allow the compiler to specialise and optimise the code it generates.

Stefan

[Chris Barker]

On Fri, May 8, 2015 at 3:42 AM, Stefan Behnel <stef...@behnel.de> wrote:

Not currently. In fact, PEP 484 is mostly uninteresting from the point of
view of a static compiler.

This is pretty disappointing -- it seems to me there are two reasons for declaring types -- type safety, and performance.

Personally, while type safety seems to make the "Enterprise" happy, I think it's next to worthless. I have yet to see even anectodal evidence that adding type checking to a dynamic language catches the may and deep bugs people seem to think it will.

Performance on the other hand, is a serious issue with dynamic languages -- at least in some case, hence my use of Cython. Maybe the future for this lies in type inference and/or JIT compilation, but in the meantime, we really do need static compilation.

So it's a shame that Python is adding a big ol' type annotation system that can't be used for what I think is the most critical use case of typing.. It's kind of like getting the downsides of static typing without the major upside. But this has been hashed out on pytohn-ideas, and teh train is moving forward -- it will be interesting to see how actually ends up getting used.

 

 And if it's not obvious from the code,
then the declaration is entirely irrelevant in the first place.

I'm not sure I follow that -- type inference is hard -- if it's not obvious from the code, surely type declarations help. -- well,maybe they only help static type checkers ;-)

 

PEP 484 doesn't even have a way to specify that a variable contains
*exactly* a given type and not a subtype. Without that, it doesn't matter
if you provide a static declaration or not, because anything that Cython
could optimise for could be overwritten by subtypes. So you would still
need the type check that Cython generates already.

Even if it did specify a specific type, then wouldn't you still need the run-time type checking anyway? there is certainly nothing in PEP484 that provides for run time type checking. i.e. a function that is specirfied at taking a specific type could always have any pyton object passed to it.

Or are you getting to Cython calling Cython -- where a function called by another Cyton function could, in theory, be guaranteed to get teh type it needs.

Basically, the sole purpose of PEP 484 is to provide helpful hints to IDEs
and static code checkers ("Look! My IDE knows that an integer is not a
sequence and I don't even have to write a test for it!"). It's explicitly
excluding support for anything that goes beyond that.

I'm not sure about that -- I think it's explicitly saying it does care about supporting anything beyond that -- specifically static compilation. Cython already used its own notation for type defijnitions, it could conceiveably use the PEP 484 syntax, butu intepret it in a differnt way, for isntance, the simiplest example from teh PEP:

def greeting(name: str) -> str:
    return 'Hello ' + name

This states that the expected type of the name argument is str . Analogically, the expected return type is str .

Expressions whose type is a subtype of a specific argument type are also accepted for that argument.

could not Cython slightly redefine that to mean "the argument must be a string -- subtypes are not allowed"

In theory, it would be nice if people could write python code, using PEP 484 for its benefits to type checking and IDE completion, etc, and then run it through Cython for performance enhancement.

Essentially a cleaner way to get pure python mode.

Whether this makes sense depends on how common use of PEP 484 becomes.

 

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Stefan Behnel]

Chris Barker schrieb am 08.05.2015 um 18:01:

> On Fri, May 8, 2015 at 3:42 AM, Stefan Behnel wrote:
>> Not currently. In fact, PEP 484 is mostly uninteresting from the point of
>> view of a static compiler.
>
> This is pretty disappointing

Oh, absolutely.

> -- it seems to me there are two reasons for
> declaring types -- type safety, and performance.
>
> Personally, while type safety seems to make the "Enterprise" happy, I think
> it's next to worthless. I have yet to see even anectodal evidence that
> adding type checking to a dynamic language catches the may and deep bugs
> people seem to think it will.
>
> Performance on the other hand, is a serious issue with dynamic languages --
> at least in some case, hence my use of Cython. Maybe the future for this
> lies in type inference and/or JIT compilation, but in the meantime, we
> really do need static compilation.

But that's not included in PEP 484. As I said, the types that you can
specify via PEP 484 type hints are (mostly) not interesting for
optimisation purposes.

>> And if it's not obvious from the code,
>> then the declaration is entirely irrelevant in the first place.
>
> I'm not sure I follow that -- type inference is hard -- if it's not obvious
> from the code, surely type declarations help. -- well,maybe they only help
> static type checkers ;-)

Sure, but not for optimisation. If the code doesn't use the fact that a
variable contains an Iterable, then declaring it doesn't help. And if the
code makes use of that property by iterating over the object, then the
declaration is redundant because it's clear from the code that the variable
needs to be iterable. That's what I meant.

> Cython already used its own notation for type defijnitions, it could
> conceiveably use the PEP 484 syntax, butu intepret it in a differnt way,
> for isntance, the simiplest example from teh PEP:
>
> def greeting(name: str) -> str:
> return 'Hello ' + name
>
> This states that the expected type of the name argument is str .
> Analogically, the expected return type is str.

But those are not the semantics that PEP 484 defines. (And this is a
particularly bad example as typing text input is almost always a bad idea
in Cython.)

Sure, Cython could reinterpret them, but then it would fail to correctly
compile some user code. And users might not notice because most test suites
will not try to send a string subtype into an interface.

> In theory, it would be nice if people could write python code, using PEP
> 484 for its benefits to type checking and IDE completion, etc, and then run
> it through Cython for performance enhancement.

Well, you can already do something similar:

https://github.com/cython/cython/blob/master/tests/run/annotation_typing.pyx

But that was pre-PEP-484 and is now obsoleted by it - in the sense that
it's incompatible, as PEP 484 does not intend to allow interoperability
with other kinds of annotations.

Stefan

[Chris Barker]

On Fri, May 8, 2015 at 9:31 AM, Stefan Behnel <stef...@behnel.de> wrote:

> Performance on the other hand, is a serious issue with dynamic languages --
> at least in some case, hence my use of Cython. Maybe the future for this
> lies in type inference and/or JIT compilation, but in the meantime, we
> really do need static compilation.

But that's not included in PEP 484.

Exactly -- PEP 484 is all  about static type safety, and secondarily about giving IDEs additional hints for users.

I guess I'm hoping that it can be leveraged to static compilation, but you'd know better than me -- it could be a dead end. After all, a good fraction of it is syntax, rather than concept. 

 

> I'm not sure I follow that -- type inference is hard -- if it's not obvious
> from the code, surely type declarations help. -- well,maybe they only help
> static type checkers ;-)

Sure, but not for optimisation. If the code doesn't use the fact that a
variable contains an Iterable, then declaring it doesn't help. And if the
code makes use of that property by iterating over the object, then the
declaration is redundant because it's clear from the code that the variable
needs to be iterable. That's what I meant.

got it. -- but lower level stuff it could, help, yes? if a variable is used in arithmetic expessions, you can know (or at least be likley correct), that it's a numeric type, but you can't know which one:

def fun(a: List[int]):
    x = 0

    for i in a:

       x += i


If Cython explicitly doesn't allow subtypes, then we know at compile time that i in that loop will be an int, and that a will be a python list -- without the type hint, all we know is that a is an iterable, and that i can be added to something that can be initialized to 0.

surely that buys us smoething?

> def greeting(name: str) -> str:
>     return 'Hello ' + name
>
> This states that the expected type of the name argument is str .
> Analogically, the expected return type is str.

But those are not the semantics that PEP 484 defines.

actually, that is a direct cut and paste from the PEP --- or do you mean the "not a subclass" part?

 

(And this is a
particularly bad example as typing text input is almost always a bad idea
in Cython.)

sure -- it was just the first and simplest example.

Sure, Cython could reinterpret them, but then it would fail to correctly
compile some user code. And users might not notice because most test suites
will not try to send a string subtype into an interface.

Will that put Cython users in any way in a worse position than they already are? Code paths are untested will always be untested. Though I suppose if folks are using a static type checker, they will think that they don't need to test that!

 

> In theory, it would be nice if people could write python code, using PEP
> 484 for its benefits to type checking and IDE completion, etc, and then run
> it through Cython for performance enhancement.

Well, you can already do something similar:

https://github.com/cython/cython/blob/master/tests/run/annotation_typing.pyx

But that was pre-PEP-484 and is now obsoleted by it - in the sense that
it's incompatible, as PEP 484 does not intend to allow interoperability
with other kinds of annotations.

Well, maybe a updated version then -- but I would certainly wait to see how this catches on before doing anything at all.

-CHB