Deriving type checking from doc strings
Paul Hildebrandt
about documenting function parameters. He also bought up using
typecheck (http://oakwinter.com/code/typecheck/) for the times when he
really wants to check parameter type.
That led to a stream of though about possibly having something like
typecheck work with typing information in the doc string. I know that
both Doxygen and Epydoc support a @param tag for the doc string. Epydoc
also supports @type for specifying the parameters type. I am not sure
how it would work but it would be cool if when we wanted type checking
we could specify it in the doc string and it would just magically work.
Maybe what we would have to do is put type checking in the code that the
documentation engine would recognize. That seems a little more doable.
Anyway I am just sharing an idea, maybe a crazy one, in hopes of feedback.
Thoughts?
Paul
Chuck Esterbrook
<Paul.Hil...@disneyanimation.com> wrote:
...
> Thoughts?
>
> Paul
WingIDE performs your last suggestion, which is that it looks for:
assert isinstance(c, Customer)
so that when you type "c." it can pop-up a list of Customer attributes
and methods. So we know it's doable and if you use the same technique
then Wing users don't have use 2 techniques.
Their docs on this are at
http://wingide.com/doc/edit/source-code-analysis and include a tip on
avoiding circular imports.
Or you could use Cobra and get all these features like types and unit
tests built into the language. ;-)
-Chuck
--
http://cobra-language.com/
Paul Hildebrandt
On Sat, May 2, 2009 at 8:49 AM, Paul Hildebrandt <Paul.Hil...@disneyanimation.com> wrote: ...Thoughts? PaulWingIDE performs your last suggestion, which is that it looks for: assert isinstance(c, Customer) so that when you type "c." it can pop-up a list of Customer attributes and methods. So we know it's doable and if you use the same technique then Wing users don't have use 2 techniques. Their docs on this are at http://wingide.com/doc/edit/source-code-analysis and include a tip on avoiding circular imports.
Or you could use Cobra and get all these features like types and unit tests built into the language. ;-)
I've actually brought it up at work but it's hard for another language to get traction.
Michał Kwiatkowski
<chuck.es...@gmail.com> wrote:
> WingIDE performs your last suggestion, which is that it looks for:
>
> assert isinstance(c, Customer)
>
> so that when you type "c." it can pop-up a list of Customer attributes
> and methods. So we know it's doable and if you use the same technique
> then Wing users don't have use 2 techniques.
From what I understood this technique helps Wing to find out types of
variables, and while it would probably be possible to extend this to
function arguments, it would look much worse than the solution with
decorators. "if 0" and all those asserts add too much noise to code
for my taste.
> Or you could use Cobra and get all these features like types and unit
> tests built into the language. ;-)
But then we would have to change the name of the project from
Pythoscope to Cobrascope. ;-)
Cheers,
mk
Michał Kwiatkowski
<Paul.Hil...@disneyanimation.com> wrote:
> Michal and I were chatting back and forth in a Pythoscope code review
> about documenting function parameters. He also bought up using
> typecheck (http://oakwinter.com/code/typecheck/) for the times when he
> really wants to check parameter type.
BTW, for those of you that like doing code reviews, please check out
http://pythoscope.appspot.com/ . We have a single patch under review
right now, but there will be more. You need a google account to
participate.
> That led to a stream of though about possibly having something like
> typecheck work with typing information in the doc string. I know that
> both Doxygen and Epydoc support a @param tag for the doc string. Epydoc
> also supports @type for specifying the parameters type. I am not sure
> how it would work but it would be cool if when we wanted type checking
> we could specify it in the doc string and it would just magically work.
> Maybe what we would have to do is put type checking in the code that the
> documentation engine would recognize. That seems a little more doable.
> Anyway I am just sharing an idea, maybe a crazy one, in hopes of feedback.
As a way to avoid the whole problem, a question. Do we really need a
documentation system? Pythoscope is a tool, not a library, for the
most part[*]. From a coder perspective, having docstrings extracted
and published in HTML somewhere doesn't help at all. I already have
all of this in the code, and more, including comments and the code
itself. OTOH API documentation systems suck for design overviews - the
place for those is not in the code anyway. I understand my point of
view can be biased on this one, since I wrote most of the Pythoscope
source code, but I can easily navigate my way through Pythoscope
source code using simple tools like grep (ack actually:
http://betterthangrep.com/) and etags. So, while I understand the need
for things like Sphinx for documenting APIs of libraries, I don't
think Pythoscope really needs them, at least not in the context of
functions/methods signatures and docstrings. Feel free to prove me
wrong. ;-)
What I'd like to focus on here instead is the need for better dispatch
methods. I mentioned typecheck already, which isn't a dispatch library
per se, but there is also this thing called peak.rules
(http://peak.telecommunity.com/DevCenter/RulesReadme). Power of those
things lies in decorators, so we would need at least Python 2.4. So,
my question is this: is there a way to use decorators without dropping
support for Python 2.3? Since decorators are basically a syntactic
sugar, it shouldn't be that hard. One way to do that would be to
develop code using decorators and then use a conversion tool that
would change this Python 2.4 code:
@decor
def fun(): pass
into this Python 2.3 code:
def fun(): pass
fun = decor(fun)
I never saw a tool like this, but having some experience with lib2to3
and ast, I don't think it would be to hard to implement. Any other
ideas? Is anyone using Pythoscope with Python 2.3 right now anyway?
[*] Once we establish an API for e.g. editor integration, we can
document that separately, probably not in the soure code itself.
Cheers,
mk
Paul Hildebrandt
On Sat, May 2, 2009 at 9:50 PM, Chuck Esterbrook <chuck.es...@gmail.com> wrote:WingIDE performs your last suggestion, which is that it looks for: assert isinstance(c, Customer) so that when you type "c." it can pop-up a list of Customer attributes and methods. So we know it's doable and if you use the same technique then Wing users don't have use 2 techniques.From what I understood this technique helps Wing to find out types of variables, and while it would probably be possible to extend this to function arguments, it would look much worse than the solution with decorators. "if 0" and all those asserts add too much noise to code for my taste.
Paul Hildebrandt
On Sat, May 2, 2009 at 5:49 PM, Paul Hildebrandt <Paul.Hil...@disneyanimation.com> wrote:Michal and I were chatting back and forth in a Pythoscope code review about documenting function parameters. He also bought up using typecheck (http://oakwinter.com/code/typecheck/) for the times when he really wants to check parameter type.BTW, for those of you that like doing code reviews, please check out http://pythoscope.appspot.com/ . We have a single patch under review right now, but there will be more. You need a google account to participate.That led to a stream of though about possibly having something like typecheck work with typing information in the doc string. I know that both Doxygen and Epydoc support a @param tag for the doc string. Epydoc also supports @type for specifying the parameters type. I am not sure how it would work but it would be cool if when we wanted type checking we could specify it in the doc string and it would just magically work. Maybe what we would have to do is put type checking in the code that the documentation engine would recognize. That seems a little more doable. Anyway I am just sharing an idea, maybe a crazy one, in hopes of feedback.As a way to avoid the whole problem, a question. Do we really need a documentation system?
Pythoscope is a tool, not a library, for the most part[*]. From a coder perspective, having docstrings extracted and published in HTML somewhere doesn't help at all. I already have all of this in the code, and more, including comments and the code itself. OTOH API documentation systems suck for design overviews - the place for those is not in the code anyway.
I understand my point of view can be biased on this one, since I wrote most of the Pythoscope source code, but I can easily navigate my way through Pythoscope source code using simple tools like grep (ack actually: http://betterthangrep.com/) and etags. So, while I understand the need for things like Sphinx for documenting APIs of libraries, I don't think Pythoscope really needs them, at least not in the context of functions/methods signatures and docstrings. Feel free to prove me wrong. ;-)
What I'd like to focus on here instead is the need for better dispatch methods. I mentioned typecheck already, which isn't a dispatch library per se, but there is also this thing called peak.rules (http://peak.telecommunity.com/DevCenter/RulesReadme).
My concern on using anything like that is the complexity it adds to the code. I am a advocate for simple code. Something that doesn't violate the Principle of least astonishment. I am not saying this does but I am wary of adding things like this that would.
Power of those
things lies in decorators, so we would need at least Python 2.4. So,
my question is this: is there a way to use decorators without dropping
support for Python 2.3? Since decorators are basically a syntactic
sugar, it shouldn't be that hard. One way to do that would be to
develop code using decorators and then use a conversion tool that
would change this Python 2.4 code:
@decor
def fun(): pass
into this Python 2.3 code:
def fun(): pass
fun = decor(fun)
I never saw a tool like this, but having some experience with lib2to3
and ast, I don't think it would be to hard to implement. Any other
ideas? Is anyone using Pythoscope with Python 2.3 right now anyway?
[*] Once we establish an API for e.g. editor integration, we can
document that separately, probably not in the soure code itself.
Cheers, mk
Michał Kwiatkowski
<Paul.Hil...@disneyanimation.com> wrote:
> Wrong is such a harsh term, how about not fully correct from my point of
> view. :) Proof would also be tough. I think you do a lot of documentation
> in the code already. I just lean towards standards when doing it so it can
> be harvested by tools such as editors, documenters or something I haven't
> thought about.
What I'm concerned about is that each standard has a cost associated
with it. Until I see benefits of having a standard for what we're
doing right now, I am not willing to pay this cost. So, for example, I
don't see a benefit of using things like @param or @type in docstrings
if we don't intend to use any documenter tool, like Sphinx. OTOH I can
see benefits of using a library like typecheck, so that's why I
proposed it in the first place. If it could be used to generate
documentation somewhere down the line - the better - but that's not
the main motivation.
> > What I'd like to focus on here instead is the need for better dispatch
> > methods. I mentioned typecheck already, which isn't a dispatch library
> > per se, but there is also this thing called peak.rules
> > (http://peak.telecommunity.com/DevCenter/RulesReadme).
>
> That does look interesting and it works under Python 2.3.
Yeah, I haven't notice that before... It does that through the
decorator tools library (http://pypi.python.org/pypi/DecoratorTools),
which is kind of evil (sys.settrace + monkeypatching linecache
module), but works. :-) I'll look into this.
> My concern on using anything like that is the complexity it adds to the
> code. I am a advocate for simple code. Something that doesn't violate the
> Principle of least astonishment. I am not saying this does but I am wary of
> adding things like this that would.
Now that I looked through peak.rules and all of its dependencies it
really *does* look complicated. We probably agree that we should keep
Pythoscope's set of dependencies as small as possible.
And one more comment, because that may not have been clear from my
earlier posts. I am basically for the signatures of function
parameters, but only as long as they are executable. There are two
reasons for that. One, it basically ensures that the code will not get
out of sync with those signatures (since they *are* the code). Two, it
will help me during programming, allowing to understand what's going
on quicker. I am *not* happy with what we have right now (my
pseudo-signatures in comments).
Cheers,
mk
Chuck Esterbrook
>
> On Sat, May 2, 2009 at 9:50 PM, Chuck Esterbrook
> <chuck.es...@gmail.com> wrote:
>> WingIDE performs your last suggestion, which is that it looks for:
>>
>> assert isinstance(c, Customer)
>>
>> so that when you type "c." it can pop-up a list of Customer attributes
>> and methods. So we know it's doable and if you use the same technique
>> then Wing users don't have use 2 techniques.
>
> From what I understood this technique helps Wing to find out types of
> variables, and while it would probably be possible to extend this to
> function arguments, it would look much worse than the solution with
> decorators. "if 0" and all those asserts add too much noise to code
> for my taste.
No, the "if 0" is just for module level vars where a cyclical import
dependency. Since I barely use module level vars, it never even came
up in my code. Also, can you really put decorators on variables in
Python? I haven't checked.
Also, the technique doesn't get "extended" to parameters; it just
works. "assert isinstance(foo, Foo)" works for any kind of arg or
variable.
And the asserts execute at run-time, helping you find bugs earlier
which reduces the time it takes to diagnose them. I was using the
"assert" statement way before I ever took WingIDE for a spin. I think
it's nice they are leveraging the language here.
>> Or you could use Cobra and get all these features like types and unit
>> tests built into the language. ;-)
>
> But then we would have to change the name of the project from
> Pythoscope to Cobrascope. ;-)
Well things like Pythoscope, doctest, etc. tend to be built into the
Cobra language and command line. So we'd just call it "Cobra". ;-)
-Chuck
--
http://cobra-language.com/
Chuck Esterbrook
I appreciate it and I understand. My latest effort is to port it to
the JVM which will expand the community and add another selling point.
Also, the refactorings being made for this will make future back-ends
much faster to implement. Possibilities include Obj-C for native Mac
and iPhone apps, and C++.
-Chuck
--
http://cobra-language.com/
Michał Kwiatkowski
>> From what I understood this technique helps Wing to find out types of
>> variables, and while it would probably be possible to extend this to
>> function arguments, it would look much worse than the solution with
>> decorators. "if 0" and all those asserts add too much noise to code
>> for my taste.
>
> No, the "if 0" is just for module level vars where a cyclical import
> dependency. Since I barely use module level vars, it never even came
Ah, OK, my mistake.
> up in my code. Also, can you really put decorators on variables in
> Python? I haven't checked.
No, but there is no need for that, as you can simply apply the
function to the variable right there.
> Also, the technique doesn't get "extended" to parameters; it just
> works. "assert isinstance(foo, Foo)" works for any kind of arg or
> variable.
OK, cool. I am focusing on the parameters case, because that's the
only usage scenario I'm interested in. :-)
> And the asserts execute at run-time, helping you find bugs earlier
> which reduces the time it takes to diagnose them. I was using the
> "assert" statement way before I ever took WingIDE for a spin. I think
> it's nice they are leveraging the language here.
It's a matter of convenience and conciseness. I prefer this:
@accepts(Foo, Bar)
def fun(foo, bar):
pass
over this:
def fun(foo, bar):
assert isinstance(foo, Foo)
assert isinstance(bar, Bar)
while the functionality of both is basically the same.
Cheers,
mk