[Python-ideas] stdlib crowdsourcing
12 views
Skip to first unread message
anatoly techtonik
May 29, 2012, 1:05:27 AM5/29/12
to python-ideas
The problem with stdlib - it is all damn subjective. There is no
process to add functions and modules if you're not well-behaved and
skilled in public debates and don't have really a lot of time to be a
champion of your module/function. In other words - it is hard (if not
impossible for 80% of Python Earth population). So, many people and
projects decide to opt-out. Take a look at Twisted - a lot of useful
stuff, but not in Python stdlib. So..
Provide a way for people to opt-out from core stuff, but still allow
to share the changes and update code if necessary.
This will require:
- a local stdlib Python path convention
- snippet normalization function and AST hash dumper
- web site with stats
- source code crawler
How it works:
1. Every project maintains its own stdlib directory with functions
that they feel are good to have in standard library
2. Functions are placed so that they are imported as if from standard
library, but this time with stdlib prefix
3. The license for this directory is public domain to remove all legal
barriers (credits are welcome, but optional)
4. Crawler (probably PyPI) scans this stdlib dir, finds functions,
normalizes them, calculates hash and submits to web site
4.1 Normalization is required to find the shared function
copy/pasted across different projects with different
indentation level, docstrings, parameters/variable names etc.
4.2 Hash is calculated upon AST. There are at least three hashes for
each entry:
4.2.1 Full hash - all docstrings and variable names are
preserved, whitespace normalized
4.2.2 Stripped hash - docstrings are stripped, variable names
are normalized
4.2.3 Signature hash - a mark placed in a comment above
function name, either calculated from function
signature or generated randomly, used for manual
tracking of copy/paste e.g. pd:ac546df6b8340a92
5. Web site maintains usage and popularity staff, accepts votes on
inclusion of snippets
User stories:
1. "I want to find if there is a better/updated version of my function
available"
1.1 I enter hash into web site search form
1.2 Site gives me a link to my snippet
1.3 I can see what people proposed to replace this function with
1.4 I can choose the function with most votes
1.5 I can flag the functions I may find irrelevant or
1.5 I can tag the functions that divert in different direction
than I need to filter them
2. "I want to reuse code snippets without additional dependencies on
3rd party projects"
1.1 Just place them into my own stdlib directory
3. "I want to update code snippets when there is an update for them"
1.1 I run scanner, it extracts signature hashes, stripped hashes
and looks if web-site version of signature matches normalized hash
4. "I want to see what people want to include in the next Python version"
1.1 A call for proposals is made
1.2 People place wannabe's into their stdlib dirs
1.3 Crawl generates new functions on a web site
1.4 Functions are categorized
1.5 Optionally included / declined with a short one-liner reason - why
1.6 Optionally provided with more detailed info why
--- feature creep cut ---
5. "I want to see what functions are popular in other languages"
1.1 A separate crawler for Ruby, PHP etc. stdlib converts their
AST into compatible format where possible
1.2 Submit to site stats
6. "I want to download the function in Ruby format"
1.1 AST converter tries to do the job automatically where possible
1.2 If it fails - you are encouraged to fix the converter rules or
write the replacement for this signature manually
Just an idea.
--
anatoly t.
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
process to add functions and modules if you're not well-behaved and
skilled in public debates and don't have really a lot of time to be a
champion of your module/function. In other words - it is hard (if not
impossible for 80% of Python Earth population). So, many people and
projects decide to opt-out. Take a look at Twisted - a lot of useful
stuff, but not in Python stdlib. So..
Provide a way for people to opt-out from core stuff, but still allow
to share the changes and update code if necessary.
This will require:
- a local stdlib Python path convention
- snippet normalization function and AST hash dumper
- web site with stats
- source code crawler
How it works:
1. Every project maintains its own stdlib directory with functions
that they feel are good to have in standard library
2. Functions are placed so that they are imported as if from standard
library, but this time with stdlib prefix
3. The license for this directory is public domain to remove all legal
barriers (credits are welcome, but optional)
4. Crawler (probably PyPI) scans this stdlib dir, finds functions,
normalizes them, calculates hash and submits to web site
4.1 Normalization is required to find the shared function
copy/pasted across different projects with different
indentation level, docstrings, parameters/variable names etc.
4.2 Hash is calculated upon AST. There are at least three hashes for
each entry:
4.2.1 Full hash - all docstrings and variable names are
preserved, whitespace normalized
4.2.2 Stripped hash - docstrings are stripped, variable names
are normalized
4.2.3 Signature hash - a mark placed in a comment above
function name, either calculated from function
signature or generated randomly, used for manual
tracking of copy/paste e.g. pd:ac546df6b8340a92
5. Web site maintains usage and popularity staff, accepts votes on
inclusion of snippets
User stories:
1. "I want to find if there is a better/updated version of my function
available"
1.1 I enter hash into web site search form
1.2 Site gives me a link to my snippet
1.3 I can see what people proposed to replace this function with
1.4 I can choose the function with most votes
1.5 I can flag the functions I may find irrelevant or
1.5 I can tag the functions that divert in different direction
than I need to filter them
2. "I want to reuse code snippets without additional dependencies on
3rd party projects"
1.1 Just place them into my own stdlib directory
3. "I want to update code snippets when there is an update for them"
1.1 I run scanner, it extracts signature hashes, stripped hashes
and looks if web-site version of signature matches normalized hash
4. "I want to see what people want to include in the next Python version"
1.1 A call for proposals is made
1.2 People place wannabe's into their stdlib dirs
1.3 Crawl generates new functions on a web site
1.4 Functions are categorized
1.5 Optionally included / declined with a short one-liner reason - why
1.6 Optionally provided with more detailed info why
--- feature creep cut ---
5. "I want to see what functions are popular in other languages"
1.1 A separate crawler for Ruby, PHP etc. stdlib converts their
AST into compatible format where possible
1.2 Submit to site stats
6. "I want to download the function in Ruby format"
1.1 AST converter tries to do the job automatically where possible
1.2 If it fails - you are encouraged to fix the converter rules or
write the replacement for this signature manually
Just an idea.
--
anatoly t.
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Nick Coghlan
May 29, 2012, 2:02:25 AM5/29/12
to anatoly techtonik, python-ideas
Once again, you're completely ignoring all existing knowledge and
expertise on open collaboration and trying to reinvent the world. It's
*not going to happen*.
The standard library is just the curated core, and *yes*, it's damn
hard to get anything added to it (deliberately so). There's a place
where anyone can post anything they want, and see if others find it
useful: PyPI.
The standard library provides tools to upload to PyPI, and, as of 3.3,
will even include tools to download and install from it.
If you don't like our ecosystem (it's hard to tell whether or not you
do: everything you post is about how utterly awful and unusable
everything is, yet you're still here years later).
If you think the PyPI UI is awful or inadequate, follow the example of
crate.io or pythonpackage.com and *create your own*. There's far more
to the Python universe than just core development, stop trying to
shoehorn everything into a place where it doesn't belong.
Finally-giving-up'ly,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
expertise on open collaboration and trying to reinvent the world. It's
*not going to happen*.
The standard library is just the curated core, and *yes*, it's damn
hard to get anything added to it (deliberately so). There's a place
where anyone can post anything they want, and see if others find it
useful: PyPI.
The standard library provides tools to upload to PyPI, and, as of 3.3,
will even include tools to download and install from it.
If you don't like our ecosystem (it's hard to tell whether or not you
do: everything you post is about how utterly awful and unusable
everything is, yet you're still here years later).
If you think the PyPI UI is awful or inadequate, follow the example of
crate.io or pythonpackage.com and *create your own*. There's far more
to the Python universe than just core development, stop trying to
shoehorn everything into a place where it doesn't belong.
Finally-giving-up'ly,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Alexandre Vassalotti
May 29, 2012, 3:00:32 AM5/29/12
to Nick Coghlan, python-ideas
On Tue, May 29, 2012 at 2:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
If you don't like our ecosystem (it's hard to tell whether or not you
do: everything you post is about how utterly awful and unusable
everything is, yet you're still here years later).
I understand the discouragement with regard to repeating yourself over and over again. But, let's keep the discussion friendly here, okay? This is Python-ideas: crazy proposals are fine. We can simply ignore those and move on.
Georg Brandl
May 30, 2012, 2:47:38 AM5/30/12
to python...@python.org
Am 29.05.2012 09:00, schrieb Alexandre Vassalotti:
> On Tue, May 29, 2012 at 2:02 AM, Nick Coghlan
> <ncog...@gmail.com
> On Tue, May 29, 2012 at 2:02 AM, Nick Coghlan
> <ncog...@gmail.com
matter-of-fact statement...
Georg
anatoly techtonik
Jun 1, 2012, 11:08:21 AM6/1/12
to Nick Coghlan, python-ideas
On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
> Once again, you're completely ignoring all existing knowledge and
> expertise on open collaboration and trying to reinvent the world. It's
> *not going to happen*.
It's too boring to live in a world of existing knowledge and
> Once again, you're completely ignoring all existing knowledge and
> expertise on open collaboration and trying to reinvent the world. It's
> *not going to happen*.
expertise, and yes, I am not aware of any open collaboration stuff
expertise. Any reading recommendations with concentrated knowledge
that can fit my brain?
> The standard library is just the curated core, and *yes*, it's damn
> hard to get anything added to it (deliberately so). There's a place
> where anyone can post anything they want, and see if others find it
> useful: PyPI.
back project compilation from the old days. The biggest Python
advantage at all times was "copy and run" ability.
The drawbacks of PyPI for this proposal are:
1. every function you need will require a separate upload to PyPI
2. you can't upload function with the same stdlib name, but slightly
different implementation as it is used in different projects
3. you can't find functions that people recommend to be included into stdlib
4. it is hard (impossible) to gather feedback on the quality of these proposals
> The standard library provides tools to upload to PyPI, and, as of 3.3,
> will even include tools to download and install from it.
really rock, if the new feature won't be settled in stone right after
release and will gain a few UX iterations with allowed break-ability.
As for PyPI, the major drawback of it is security - DNS attack for a
couple of minutes, and one of your automatically deployed nodes is
trojan ready. I remember PyPI password are stored in clear-text on
developer's machine, but I don't remember if anyone turned off HTTP
basic authorization on PyPI to protect passwords travelling to PyPI
with every upload from intercepting. It would be an interesting
exercise to sniff PyPI passwords over WiFi during next conference
(i.e. https://ep2012.europython.eu/) and match those to the
developer's accounts on *.python.org ;)
> If you don't like our ecosystem (it's hard to tell whether or not you
> do: everything you post is about how utterly awful and unusable
> everything is, yet you're still here years later).
wouldn't stick there. It is like a vintage car - awesome, nice
looking, and there is even this new twisted pyusion engine inside,
but.. well - it's not for youngsters.
> If you think the PyPI UI is awful or inadequate, follow the example of
> crate.io or pythonpackage.com and *create your own*. There's far more
> to the Python universe than just core development, stop trying to
> shoehorn everything into a place where it doesn't belong.
Speaking about PyPI enhancements and ecosystem, instead of reinventing
bicycles I'd rather patch existing one. The only problem is that
patches are not accepted.
https://bitbucket.org/loewis/pypi/pull-request/1/fix-imports-add-logging-to-console-in
Calvin Spealman
Jun 2, 2012, 1:24:05 PM6/2/12
to anatoly techtonik, python-ideas
On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <tech...@gmail.com> wrote:
> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>> Once again, you're completely ignoring all existing knowledge and
>> expertise on open collaboration and trying to reinvent the world. It's
>> *not going to happen*.
>
> It's too boring to live in a world of existing knowledge and
> expertise,
Frankly, this one fragment is enough to stop me reading further. Who
> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>> Once again, you're completely ignoring all existing knowledge and
>> expertise on open collaboration and trying to reinvent the world. It's
>> *not going to happen*.
>
> It's too boring to live in a world of existing knowledge and
> expertise,
wants to learn
from the vast and broad experience when you could simply randomize the rules of
reality through ignorance and stubbornness?
I sound fickle, because I am.
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
anatoly techtonik
Jun 12, 2012, 5:04:58 AM6/12/12
to ironf...@gmail.com, python-ideas
On Sat, Jun 2, 2012 at 8:24 PM, Calvin Spealman <ironf...@gmail.com> wrote:
> On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <tech...@gmail.com> wrote:
>> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>>> Once again, you're completely ignoring all existing knowledge and
>>> expertise on open collaboration and trying to reinvent the world. It's
>>> *not going to happen*.
>>
>> It's too boring to live in a world of existing knowledge and
>> expertise,
>
> Frankly, this one fragment is enough to stop me reading further. Who
> wants to learn
> from the vast and broad experience when you could simply randomize the rules of
> reality through ignorance and stubbornness?
If everybody would think like this, the world will never learn about
> On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <tech...@gmail.com> wrote:
>> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>>> Once again, you're completely ignoring all existing knowledge and
>>> expertise on open collaboration and trying to reinvent the world. It's
>>> *not going to happen*.
>>
>> It's too boring to live in a world of existing knowledge and
>> expertise,
>
> Frankly, this one fragment is enough to stop me reading further. Who
> wants to learn
> from the vast and broad experience when you could simply randomize the rules of
> reality through ignorance and stubbornness?
anti-patterns, and the software craftmanship collapsed in astonishing
agony some years ago. If it doesn't make it clear - it is not
randomizing - it is putting beliefs to the test asking for the current
status.
> I sound fickle, because I am.
the fun to discuss the technical part no matter how long ago it was
invented. If you have a lot of people who ask the same question -
create a FAQ. That's not a vast and broad experience - that's just a
time proven practice from usenet times.
Common guys, what's wrong with you? It is just an idea, not a proposal
or scientific paper. And I am not a scientist - I just want to discuss
the idea, and I am not sending mails to python-dev anymore, because
you asked to. I've spent some time trying to make the idea
interesting. It is fine If you know a scientific paper about the
matter, can explain it in a few words and send a link for more
details. But the replies like "you're stubborn and ignorant, and
nobody should help you" doesn't make you a better person. I am
criticizing, because I lack time, motivation and fantasy to write
stuff about good and bright sides in my life that I just don't see. I
write because I see bad things that can be better, and I am still open
to discuss if it is real or not.
--
anatoly t.
Daniel Fetchinson
Jun 12, 2012, 6:16:06 AM6/12/12
to python-ideas
> I just want to discuss the idea,
Great! You even got a perfectly good answer already: *not going to happen*!
Hey, this seems to be working: you raise a point, the community
discusses it and after careful deliberation comes up with an answer!
This is how things are supposed to be working, aren't they?
> But the replies like "you're stubborn and ignorant, and
> nobody should help you" doesn't make you a better person.
think they came from thin air? People read your post, thought about
it, considered the pros and cons and then put in the time to answer
it, write an email and hit the Send button.
Now, move on, nothing to be seen here, chop, chop, carry on!
Cheers,
Daniel
--
Psss, psss, put it down! - http://www.cafepress.com/putitdown
Yuval Greenfield
Jun 12, 2012, 12:11:11 PM6/12/12
to anatoly techtonik, python-ideas
I think having a separate site "anatloy's std-lib" which somehow implemented an easy install of the top 10-100 most useful/popular/selected packages on pypi could be nice. I considered making such a bundle myself a while ago.
I don't think it really needs to be python.org sanctioned.
Yuval
PS I like how candid the replies you got were, and indeed getting a reply is better than the sound of crickets. Though some of these replies carried the scent of excrement poredom - the author's need to import niceness.
Mike Meyer
Jun 12, 2012, 12:51:47 PM6/12/12
to python...@python.org
On Tue, 12 Jun 2012 12:04:58 +0300
mismatch. He wants to discuss language/platform/environment
ideas. This is valuable work, and he does have some interesting
ideas. It definitely has a place in the world.
It's just that this isn't that place. Python has a set of objectives
for the language that have been around long enough to qualify as
"traditions". As such, it's not a good place to experiment with
arbitrary changes to things, because you keep running afoul of the
traditions.
> Common guys, what's wrong with you? It is just an idea, not a proposal
> or scientific paper.
Yes, but it's an idea that ignores the traditions of the environment
you're proposing it for. If you're serious about discussing ideas
about changing Python, you need to do the groundwork of understanding
those traditions, and try and make sure your ideas don't collide with
them. It doesn't matter whether or not they're good ideas, if they
clash with the traditions, they aren't going to happen. You need to
figure that out yourself, and not ask us to do it for you.
If, on the other hand, you want to talk about
language/platform/environment design ideas without that restriction,
then you need a different forum. Just because you happen to be working
in Python doesn't mean that a Python forum is appropriate for them,
any more than discussing (say) drone control programs would be
appropriate in a Python forum just because I happen to be writing it
in Python.
If you're somewhere in between the two, maybe a PyPy forum would be
more appropriate? I dunno. I'm sorry I can't really recommend a good
forum for you. The last time I was seriously interested in such
things, Python hadn't been released yet.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
anatoly techtonik <tech...@gmail.com> wrote:
> On Sat, Jun 2, 2012 at 8:24 PM, Calvin Spealman <ironf...@gmail.com> wrote:
> > On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <tech...@gmail.com> wrote:
> >> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
> >>> Once again, you're completely ignoring all existing knowledge and
> >>> expertise on open collaboration and trying to reinvent the world. It's
> >>> *not going to happen*.
> >> It's too boring to live in a world of existing knowledge and
> >> expertise,
> > Frankly, this one fragment is enough to stop me reading further. Who
> > wants to learn
> > from the vast and broad experience when you could simply randomize the rules of
> > reality through ignorance and stubbornness?
> If everybody would think like this, the world will never learn about
> anti-patterns, and the software craftmanship collapsed in astonishing
> agony some years ago. If it doesn't make it clear - it is not
> randomizing - it is putting beliefs to the test asking for the current
> status.
Ah, I think I see Anatoly's problem here. It's an impedance
> On Sat, Jun 2, 2012 at 8:24 PM, Calvin Spealman <ironf...@gmail.com> wrote:
> > On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <tech...@gmail.com> wrote:
> >> On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
> >>> Once again, you're completely ignoring all existing knowledge and
> >>> expertise on open collaboration and trying to reinvent the world. It's
> >>> *not going to happen*.
> >> It's too boring to live in a world of existing knowledge and
> >> expertise,
> > Frankly, this one fragment is enough to stop me reading further. Who
> > wants to learn
> > from the vast and broad experience when you could simply randomize the rules of
> > reality through ignorance and stubbornness?
> If everybody would think like this, the world will never learn about
> anti-patterns, and the software craftmanship collapsed in astonishing
> agony some years ago. If it doesn't make it clear - it is not
> randomizing - it is putting beliefs to the test asking for the current
> status.
mismatch. He wants to discuss language/platform/environment
ideas. This is valuable work, and he does have some interesting
ideas. It definitely has a place in the world.
It's just that this isn't that place. Python has a set of objectives
for the language that have been around long enough to qualify as
"traditions". As such, it's not a good place to experiment with
arbitrary changes to things, because you keep running afoul of the
traditions.
> Common guys, what's wrong with you? It is just an idea, not a proposal
> or scientific paper.
you're proposing it for. If you're serious about discussing ideas
about changing Python, you need to do the groundwork of understanding
those traditions, and try and make sure your ideas don't collide with
them. It doesn't matter whether or not they're good ideas, if they
clash with the traditions, they aren't going to happen. You need to
figure that out yourself, and not ask us to do it for you.
If, on the other hand, you want to talk about
language/platform/environment design ideas without that restriction,
then you need a different forum. Just because you happen to be working
in Python doesn't mean that a Python forum is appropriate for them,
any more than discussing (say) drone control programs would be
appropriate in a Python forum just because I happen to be writing it
in Python.
If you're somewhere in between the two, maybe a PyPy forum would be
more appropriate? I dunno. I'm sorry I can't really recommend a good
forum for you. The last time I was seriously interested in such
things, Python hadn't been released yet.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Stephen J. Turnbull
Jun 13, 2012, 1:09:04 AM6/13/12
to anatoly techtonik, python-ideas
> I write because I see bad things that can be better, and I am still
> open to discuss if it is real or not.
place for ideas where the poster is pretty sure it's real *and* has a
concrete proposal (the "idea" in python-ideas) to make it better *and*
has the will to follow up themselves if nobody else grabs the ball.
There's some room for blue-sky ideas (lacking concrete proposals or
personal commitment), but if all you ever offer is blue-sky ideas that
get no uptake, you're just wasting time, yours as well as everybody
else's.
Reply all
Reply to author
Forward
0 new messages