GSoC, Python 3 dependencies, Paste
Juliusz Gonera
I thought that it will be easier if I just post here instead of trying
to catch someone on IRC.
I have quite a few questions regarding GSoC. First of all, I'm not quite
sure if I should propose the specific project or it should be given to
me by one of the mentors. I was thinking about porting the necessary
packages to Python 3, but I guess it could be too much for a single
project (or maybe not?).
Because of that I though about concentrating on Paste issue. I don't
know if any decision has been made (whether to update Paste, rewrite
create and serve or use something else, e.g. Marrow) and I'm almost sure
I'm not the one to make such decisions. However, I have to put something
more detailed in the application form.
Regarding Paste and comments on
https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm:
1. YAML vs INI - is there any decision?
2. Is marrow.script not being argparse-compatible a big issue? Wouldn't
it be easy to fix?
3. The same goes for marrow.server.http being asynchronous. As long as
it was multithreaded too I don't know how this could be a problem (but
maybe I don't know enough). And according to the readme on github
(https://github.com/marrow/marrow.server.http), multi-threading is on
the todo list:
"threaded — Enable multi-threading. This is currently unimplemented
pending support up-stream in marrow.server and defaults to False."
4. "I'm uncomfortable with the direction of these libs personally. They
seem to be more researchy than practical in lots of cases."
Could somebody elaborate? I'm not trying to question anything, I don't
know that much, I would just like to know what is wrong with Marrow.
5. If adapting and improving Marrow is not an option, then what should I
do if I want to work on porting Pyramid to Python 3? On IRC Blaise
Laflamme suggested CherryPy. What do you think?
I thought that this is more development related so I'm posting to devel.
Let me know if I was wrong.
Regards,
--
Juliusz Gonera
http://juliuszgonera.com/
Joe Dallago
I also plan on helping port Pyramid to Python 3 as my GSOC project. You can check out my application on my blog(jayd3e.com) for more details. As far as Paste goes, it has unofficially been decided that we will be going with an entirely separate HTTP server, as opposed to isolating 'paster serve'. I know a few people are in favor of using a wsgi server called WiseGuy(https://github.com/wsgicabal/wiseguy), there a variety of others as well though. 'Paster create' is currently being isolated by someone afaik, forget who.
I personally think that you should choose to port Pyramid as your project, because we could always use more help for that effort.
--
You received this message because you are subscribed to the Google Groups "pylons-devel" group.
To post to this group, send email to pylons...@googlegroups.com.
To unsubscribe from this group, send email to pylons-devel...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Daniel Holth
I hope Ian's Web Application Format proposal gets specified. Then I can just follow the specification and choose between potential Paste Deploy replacements based on some other criteria than 'is a mandatory Pyramid dependency'.
Alice Bevan–McGregor
> https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm:
>
> 1. YAML vs INI - is there any decision?
There isn't any particular reason not to go with both, INI for
backwards compatibility (using similar logic to YAML's parsing of
lists, numbers, etc., and YAML for the future. YAML's native rich
structures are just too handy, and if you know JSON, you can write YAML.
I'll be rolling that into marrow.config.
> 2. Is marrow.script not being argparse-compatible a big issue? Wouldn't
> it be easy to fix?
marrow.script was designed to be explicitly argparse/optparse/getopt
incompatible. Paste Script's wrapping of optparse made things more
complicated, not easier. Unless, of course, someone can come up with
an _excellent_ reason why marrow.script should be rewritten to utilize
argparse.
Marrow.script, OTOH, wraps nothing. It's a direct command line ->
argspec converter and help text generator. The ease of just throwing a
function or class at marrow.script (new or existing!) is highly
desirable.
> 3. The same goes for marrow.server.http being asynchronous. As long as
> it was multithreaded too I don't know how this could be a problem (but
> maybe I don't know enough). And according to the readme on github
> (https://github.com/marrow/marrow.server.http), multi-threading is on
> the todo list:
> "threaded — Enable multi-threading. This is currently unimplemented
> pending support up-stream in marrow.server and defaults to False."
Seems I need to fix that reference. Threading is supported (now that
it's supported in marrow.server) using a Futures-based thread pool.
The underlying server being async has never been a bad thing in any of
the benchmarking I've done, including C10K and 12KR/sec concurrency and
speed benchmarks, even with the thread pool enabled.
> 4. "I'm uncomfortable with the direction of these libs personally. They
> seem to be more researchy than practical in lots of cases."
> Could somebody elaborate? I'm not trying to question anything, I don't
> know that much, I would just like to know what is wrong with Marrow.
Same! :D
Just because the packages take a distinctly different approach to some
things than Paste/WebOb/Django/<pkgname here> doesn't make them "bad",
"unusable", or "unstable". In fact, for PyPi releases, the packages
require 100% unit test coverage, full documentation, and 2.6+/3.1+
polygot compatibility; that already makes them somewhat "better" than
(some of) the alternatives.
I'm already using marrow.tags (and the widgeting it provides),
marrow.script, and marrow.blueprint in a number of projects, all with
great success.
— Alice.
Chris McDonough
I think "researchy" describes some of the marrow stuff pretty well (that
doesn't imply bad, unusable or unstable, it just implies opinionatedness
that is outside the current mainstream). For example, in this message
you've quoted use of a futures-based asynchronous server that says it
implements WSGI 2.0, but WSGI 2.0 does not yet exist in any ratified
form. I don't see WSGI 2.0 draft compatibility as a goal that we need
to explore under GSOC. I think it's great you're pushing the envelope,
but we really just need some very simple things that are well behind any
of the curves you're at the front of.
In the meantime, I don't much care if we get a PasteDeploy replacement
that works under Python 3.X for GSOC. Pyramid apps don't strictly
require PasteDeploy, they can be run without any declarative
configuration. I also don't care much if we don't get a PasteScript
replacement for GSOC either. Having a command line wrapper that does a
bunch of stuff is handy but not required to make things run. We
actually really just need some bits of Paste itself (the FileApp app,
StaticURLParser, some of the auth_tkt bits) to work in order to reach a
reasonable goal for 3.X compat. After that, we can think about how we'd
like to wrap it up nicely for usage. I see the PasteScript/PasteDeploy
stuff as a sort of last-resort project for GSOC, because it's pretty
gratuituous until we get all the other dependencies ported.
Alice Bevan–McGregor
> I think "researchy" describes some of the marrow stuff pretty well
> (that doesn't imply bad, unusable or unstable, it just implies
> opinionatedness that is outside the current mainstream). For example,
> in this message you've quoted use of a futures-based asynchronous
> server that says it implements WSGI 2.0, but WSGI 2.0 does not yet
> exist in any ratified form. I don't see WSGI 2.0 draft compatibility
> as a goal that we need to explore under GSOC. I think it's great
> you're pushing the envelope, but we really just need some very simple
> things that are well behind any of the curves you're at the front of.
I agree with regards to m.s.http; it was originally developed as a
'proof of concept' server to be developed in parallel with my rewritten
PEP 444. However, making it speak PEP 333(3) is relatively trivial.
At the grossest level (callable syntax), you use the following p-code
decorator:
def make_pep444(fn):
def inner(environ):
def start(status, headers):
start.status = status
start.headers = headers
body = fn(environ, start)
return start.status, start.headers, body
return inner
> In the meantime, I don't much care if we get a PasteDeploy replacement
> that works under Python 3.X for GSOC. Pyramid apps don't strictly
> require PasteDeploy, they can be run without any declarative
> configuration. I also don't care much if we don't get a PasteScript
> replacement for GSOC either. Having a command line wrapper that does a
> bunch of stuff is handy but not required to make things run.
They remain nice goals, however +1 on pairing down to the minimum.
— Alice.
Chris McDonough
> Hi,
>
> I thought that it will be easier if I just post here instead of trying
> to catch someone on IRC.
>
> I have quite a few questions regarding GSoC. First of all, I'm not quite
> sure if I should propose the specific project or it should be given to
> me by one of the mentors. I was thinking about porting the necessary
> packages to Python 3, but I guess it could be too much for a single
> project (or maybe not?).
I think asking a single student to port everything is too much. At this
point, I think there's an opportunity to get two students working on the
porting effort.
> Because of that I though about concentrating on Paste issue. I don't
> know if any decision has been made (whether to update Paste, rewrite
> create and serve or use something else, e.g. Marrow) and I'm almost sure
> I'm not the one to make such decisions. However, I have to put something
> more detailed in the application form.
To be honest, I'm not sure we can manage "paste replacement" as a GSOC
project, especially "from scratch". Paste is relatively large, and
encompasses many things. Furthermore, we actually use very little of
Paste in Pyramid. Pyramid will happily run without PasteDeploy or
PasteScript, so porting either is not a "hot" issue for Python 3
compatibility in the short term.
Pyramid won't run without some things from the Paste package proper
(namely its static file application named StaticURLParser, and some
auth_tkt components). These will need to be replaced in order to run on
Python 3. But it's a *lot* easier just to cut and paste and fix these
specific components than it is to port the entirety of Paste, so I'd
suggest we just do that.
I think we should hold off on passing judgment on "replacing Paste"
until after GSOC, when we know that we can run some form of Pyramid
(albeit without PasteDeploy or PasteScript support) successfully.
> Regarding Paste and comments on
> https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm:
>
> 1. YAML vs INI - is there any decision?
>
> 2. Is marrow.script not being argparse-compatible a big issue? Wouldn't
> it be easy to fix?
>
> 3. The same goes for marrow.server.http being asynchronous. As long as
> it was multithreaded too I don't know how this could be a problem (but
> maybe I don't know enough). And according to the readme on github
> (https://github.com/marrow/marrow.server.http), multi-threading is on
> the todo list:
> "threaded — Enable multi-threading. This is currently unimplemented
> pending support up-stream in marrow.server and defaults to False."
>
> 4. "I'm uncomfortable with the direction of these libs personally. They
> seem to be more researchy than practical in lots of cases."
> Could somebody elaborate? I'm not trying to question anything, I don't
> know that much, I would just like to know what is wrong with Marrow.
>
> 5. If adapting and improving Marrow is not an option, then what should I
> do if I want to work on porting Pyramid to Python 3? On IRC Blaise
> Laflamme suggested CherryPy. What do you think?
>
> I thought that this is more development related so I'm posting to devel.
> Let me know if I was wrong.
I think all the above stuff is probably too hard to tackle under GSOC.
I suspect we should just get the bare minimum ported and see where we
are. There is a list of dependencies on the
https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm page that
indicates the packages that need porting. I'd suggest that creating a
proposal to port some of those would be best-received.
- C
Juliusz Gonera
> I also plan on helping port Pyramid to Python 3 as my GSOC project. You
> for more details.
Thanks. You mention that a number of books have been written on the
subject of porting applications. Did you refer to Python 2 -> 3 or
porting in general?
> I personally think that you should choose to port Pyramid as your
> project, because we could always use more help for that effort.
This is what I'll do.
Juliusz Gonera
> IIRC Pyramid doesn't really depend on Paste in any important way. You
> could just try to get it working under CherryPy, uWSGI or wsgiref for
> example, and get the configuration dictionary some other way besides
> having Paste parse .ini into {}.
I'm aware of that. I deployed some of Pylons applications under uWSGI
and I guess this wouldn't be different in case of Pyramid.
Anyway, I think it's good for a framework to have some basic HTTP server
and "scaffolding"/configuration tools integrated.
> I hope Ian's Web Application Format proposal gets specified. Then I can
> just follow the specification and choose between potential Paste Deploy
> replacements based on some other criteria than 'is a mandatory Pyramid
> dependency'.
What's Ian's Web Application Format?
Daniel Holth
The goal would be to be able to copy your application over to a web server and expect it to run without having to worry about configuring the server as much.
Juliusz Gonera
> I think asking a single student to port everything is too much. At this
> point, I think there's an opportunity to get two students working on the
> porting effort.
I see. I didn't know that. I thought that it was most likely that
Pyramid will get only one person.
Anyway, I think that when porting the amount of work will depend on the
port/replace ratio.
> To be honest, I'm not sure we can manage "paste replacement" as a GSOC
> project, especially "from scratch". Paste is relatively large, and
> encompasses many things. Furthermore, we actually use very little of
> Paste in Pyramid. Pyramid will happily run without PasteDeploy or
> PasteScript, so porting either is not a "hot" issue for Python 3
> compatibility in the short term.
I see your point now.
> Pyramid won't run without some things from the Paste package proper
> (namely its static file application named StaticURLParser, and some
> auth_tkt components). These will need to be replaced in order to run on
> Python 3. But it's a *lot* easier just to cut and paste and fix these
> specific components than it is to port the entirety of Paste, so I'd
> suggest we just do that.
> I think we should hold off on passing judgment on "replacing Paste"
> until after GSOC, when we know that we can run some form of Pyramid
> (albeit without PasteDeploy or PasteScript support) successfully.
I didn't think about porting everything from Paste. That's why I was
rather considering possible replacements for it (Marrow, CherryPy).
Anyway, this clears it up a bit for me. Now I understand that having an
integrated server and some automatization/config tools is not essential
at the moment.
As I understand it, those will be eventually necessary for some first
official release of Pyramid for Python 3 (which won't be very soon). As
for now, other dependencies have to be ported and satisfied in order to
be able to start working on next major release of Pyramid.
> I think all the above stuff is probably too hard to tackle under GSOC.
> I suspect we should just get the bare minimum ported and see where we
> are. There is a list of dependencies on the
> https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm page that
> indicates the packages that need porting. I'd suggest that creating a
> proposal to port some of those would be best-received.
Should I choose specific ones or just mention that I would be porting
part of it?
What about porting zope packages? Should we port them or wait for
someone who has already begun to finish it?
Chris McDonough
That sounds right.
> > I think all the above stuff is probably too hard to tackle under GSOC.
> > I suspect we should just get the bare minimum ported and see where we
> > are. There is a list of dependencies on the
> > https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm page that
> > indicates the packages that need porting. I'd suggest that creating a
> > proposal to port some of those would be best-received.
>
> Should I choose specific ones or just mention that I would be porting
> part of it?
Choosing a subset is a good idea for the application (because it's
probably impossible to port all of them), as long as you're willing to
trade around as necessary when there's more than one person that wants
to work on the same packages.
> What about porting zope packages? Should we port them or wait for
> someone who has already begun to finish it?
We should port them.
- C
Alice Bevan–McGregor
> On 2011-04-04 11:31:15 -0700, Chris McDonough said:
>> I think "researchy" describes some of the marrow stuff pretty well >
>> (that doesn't imply bad, unusable or unstable, it just implies >
>> opinionatedness that is outside the current mainstream). For example,
>> > in this message you've quoted use of a futures-based asynchronous >
>> server that says it implements WSGI 2.0, but WSGI 2.0 does not yet >
>> exist in any ratified form. I don't see WSGI 2.0 draft compatibility >
>> as a goal that we need to explore under GSOC. I think it's great >
>> you're pushing the envelope, but we really just need some very simple >
>> things that are well behind any of the curves you're at the front of.
>
> I agree with regards to m.s.http; it was originally developed as a
> 'proof of concept' server to be developed in parallel with my rewritten
> PEP 444. However, making it speak PEP 333(3) is relatively trivial.
> At the grossest level (callable syntax), you use the following p-code
> decorator:
>
> def make_pep444(fn):
> def inner(environ):
> def start(status, headers):
> start.status = status
> start.headers = headers
>
> body = fn(environ, start)
> return start.status, start.headers, body
>
> return inner
Just a quick update: 333->444 adaption code is now in the repo:
WebCore (utilizing WebOb, WebError, etc.) works great. (Interactive
Python shell and stack interrogation work great, as does form handling
within WebCore.)
— Alice.
Juliusz Gonera
>> Should I choose specific ones or just mention that I would be porting
>> part of it?
>
> Choosing a subset is a good idea for the application (because it's
> probably impossible to port all of them), as long as you're willing to
> trade around as necessary when there's more than one person that wants
> to work on the same packages.
OK, it took me some time but I finally wrote the application:
http://blog.juliuszgonera.com/wp-content/uploads/2011/04/gsoc-pyramid.pdf
I'm not sure if I should put something more in milestones. PSF wiki
(http://wiki.python.org/moin/SummerOfCode/Application) suggests listing
week-by-week work plan, but I'm not sure it's suitable for this project.
I guess many things can change, but if a week-by-week work plan is
recommended I can prepare a draft. On the other hand side, their own
application template lists only those 3 milestones (start, midterm, final).
Also, I'd like you to comment on the packages chosen by me to be ported
(too little / too much work?)
Any comments are appreciated. If there are some ridiculous English
mistakes I'd also like to know that ;)
Mike Orr
> OK, it took me some time but I finally wrote the application:
> http://blog.juliuszgonera.com/wp-content/uploads/2011/04/gsoc-pyramid.pdf
>
> I'm not sure if I should put something more in milestones. PSF wiki
> (http://wiki.python.org/moin/SummerOfCode/Application) suggests listing
> week-by-week work plan, but I'm not sure it's suitable for this project. I
> guess many things can change, but if a week-by-week work plan is recommended
> I can prepare a draft. On the other hand side, their own application
> template lists only those 3 milestones (start, midterm, final).
Week by week is probably not realistic because you don't know how long
each part will take, and the tasks probably won't fall on week
boundaries. Still, if you can identify a sequence of steps that must
be taken for each milestone, you can estimate the number of days for
each step. E.g., write tests for this subpackage, that subpackage,
etc. Leave at least the last week of each milestone free for
contingencies.
> Also, I'd like you to comment on the packages chosen by me to be ported (too
> little / too much work?)
My main concern about all the applications is that I'm not sure we can
accurately predict that this far in advance how long each package will
take and their relative importance. (I'm only talking about
applications related to porting to Python 3/unit testing; I'm not sure
if there are any applications outside that.) That partly depends on
what all the GSOC students and other developers do over the summer.
What I'm saying is that while we can make tentative student-to-package
assignments up front, I think we'll need the flexibility to
re-evaluate this midterm-ish. Just to discuss between the
students/mentors/group leader whether the second-half assignments
still make sense or whether a student may want to change packages.
Because the ultimate goal is to run Pyramid on Python 3, and we need
to avoid giving one package too much attention and another package too
little.
For instance, most of the applications are focused heavily on Ian's
packages (Paste, WebOb, WebError, etc.) This may be appropriate for
not; I'm not sure; ChrisM would have a better handle on that. But we
can't have two students working on the same package, or rigidly focus
on a few packages that turn out to be too easy or too marginal (e.g.,
10% of the problem). That's where I think flexibility comes in. It
would reassure me if applications indicate whether students are
willing to consider working on other packages if it seems appropriate
at the time, and perhaps list alternate packages that they'd be most
(or least) interested in working on.
The other thing I'd like to see in all the applications is a more
specific indication of the person's Python programming experience.
E.g., what kinds of packages or programs you've worked on, the kinds
of Python complexity involved in that, etc. Writing unit tests
probably requires just basic Python experience, but porting may
involve more depending on the package. For instance, I have a solid
background with (Mako-style) templating, database models, URL dispatch
and tag builders, but not much with the Zope-style interfaces or
traversal. So I'm fine with anything database-y but if it involves ZCA
or garbage collection or packet tracing, I'd probably want to call in
the experts. Anyway, it helps to know up front what kinds of tasks a
student can jump right into and what kinds would be more difficult.
--
Mike Orr <slugg...@gmail.com>