Pyramid 2 Brainstorm & Marrow
Alice Bevan–McGregor
I've updated the Brainstorm Wiki page [1] to include some
comments-to-the-comments, copied and elaborated upon below. There
seems to be a lot of confusion and/or misinformation being spread about
the Marrow components and I'd like to correct a few points here.
> MO: My main concerns are, marrow.script is not argparse-compatible, and
> marrow.server.http is asynchronous. I don't think we want to make such
> a large leap from multithreaded to asynchronous in the default server.
> CM: I'm uncomfortable with the direction of these libs personally. They
> seem to be more researchy than practical in lots of cases.
I would appreciate elaboration on these points, other than the async
one which is based on an erroneous assumption. "Researchy" isn't even
in my dictionary. ;)
My responses follow. On argparse:
> Avoiding argparse is a feature, not a bug. If you want argparse, what
> benefit does a scripting interface API give you? Effectively none.
> marrow.script is designed to allow you to write new callables (or use
> existing ones!) and have the command line interface Just Work. Even
> more importantly, marrow.script does not provide a single central
> script to run; your scripts aren't magically part of another package.
> Optparse/argparse/getopt are terribly non-Pythonic, even though
> argparse had the opportunity to fix things!
Honestly, if you're highly desirous of using any of the existing stdlib
parsers, you have no apparent need for a command line parsing library
that adapts any function into a command line argspec.
Paster really provides a centralized command (paster) which only adds
headspace confusion, and context-aware scripts which only add
frustration; e.g. I can't open a shell unless I'm sitting in my source
folder… on deployment there isn't a source folder!
Alternatively, why would you want a higher-level API (which defines, if
nothing else, duck-typed class attributes which you "fill out" and some
form of __call__ method) to use argparse?
> Simple is better than complex.
> Complex is better than complicated.
I'd argue that the 'magic' involved in marrow.script (introspection)
falls into the complex category, while the end-use use of the library
is quite simple. Paster… that's pretty complicated. I would be
interested, however, if someone could create a Paster script with the
following argspec: (notes in {} denote typing, unicode unless
specified, default values presented inline)
cmd [-v|--value=None] [-n|--name="world"] [-s|--switch]
[-a|--age=18{int}] <required> [arg[ arg...]] [--arg=value[
--arg=value...]]
Those last two denote unlimited positional arguments and unlimited
name/value pairs respectively. The above is generated from the
following in marrow.script:
def ultima(required, value=None, name="world", switch=False, age=18,
*args, **kw):
print "Hello %s!" % (name, )
On marrow.server.http:
> The underlying communication stack (marrow.io/server) is fundamentally
> async. That particular fact has zero (and I mean it, zero) impact on
> writing WSGI applications that run under the HTTP/1.1 implementation.
> The underlying stack also has full support for a worker thread pool
> (using Futures) and multi-processing. Both of these details are handled
> for you by the web server; if you enable a thread pool, communication
> is still handled using async, but the WSGI application is executed in a
> thread. This allows you to tune a deployment for your requirements.
> Development or io-bound? Run a base server. CPU-bound? Run a threaded
> server. Need to scale? Add multi-processing to either of those.
I'll mention it again, since this point is very important: the
underlying server being async has ZERO impact on your own applications.
It's not like I magically monkeypatched async support in everywhere;
no, instead, only the core reactor for I/O (which makes sense) is
async. The HTTP protocol implementation plugged into marrow.server
either runs the WSGI application inline (which blocks other requests if
not fast enough) or in a thread pool, your choice. (Choice is good! I
have an RPC service running in pure async mode that can handle ~6KRsec;
threading slows it down!)
The entire HTTP/1.1 protocol is ~175 opcodes and is fairly well
organized with some helpful comments sprinkled within.
On build quality:
> Proper releases of marrow packages require 100% unit test coverage,
> documentation, and multiple working examples. This takes the packages
> somewhat out of the realm of 'experimental' or plain 'research'. I've
> also successfully integrated blueprints, scripting, and marrow.tags
> into production, commercial code and managed to get WebCore and Pyramid
> both running under the HTTP server (with a tiny adapter decorator for
> WSGI1).
Admittedly, few of the packages have reached 1.0 yet, and one
(marrow.util, I'm looking at you!) doesn't adhere to the unit test
coverage / documentation rule presented above. marrow.util has existed
a lot longer than the others, though, and will be brought up to spec in
the near future. (It still has 80-90% coverage, though.)
Additionally, all marrow packages support 2.6+ and 3.1+ from initial
release onwards.
One of the easiest ways to examine these libraries are via the example
code [2][3], and through the code for the libraries themselves.
Additionally, you'll have to dig into a fork [4] to see demonstrations
of WSGI1 compatibility in marrow.server.http. The adapter [5] is
simple, and likely incomplete, but functional.
Finally, people shouldn't forget about other components like: TurboMail
3 [6], now marrow.mailer 4 [7], a TurboMail replacement which supports
an extremely large number of delivery back-ends and even 'delivery
tickets' to later verify successful transfer; and alacarte [8], a
generalized templating API with an array of existing engine adapters.
- Alice.
P.s. Actual critique of the code (other than "this feels
researchy/lame" ;) or even bug reports / feature requests would be
appreciated!
[1] https://github.com/Pylons/pyramid/wiki/Pyramid-2-Brainstorm
[2] https://github.com/marrow/marrow.script/tree/master/examples
[3] https://github.com/marrow/marrow.server.http/tree/master/examples
[4] https://github.com/marrow/marrow.server.http/tree/pep3333/examples
[5]
https://github.com/marrow/marrow.server.http/blob/pep3333/marrow/server/http/adapt.py
[6]
https://bitbucket.org/gothalice/turbomail/
[7] https://github.com/marrow/marrow.mailer/
[8] https://github.com/GothAlice/alacarte/
Mike Orr
<al...@gothcandy.com> wrote:
> Howdy!
>
> I've updated the Brainstorm Wiki page [1] to include some
> comments-to-the-comments, copied and elaborated upon below. There seems to
> be a lot of confusion and/or misinformation being spread about the Marrow
> components and I'd like to correct a few points here.
>
>> MO: My main concerns are, marrow.script is not argparse-compatible, and
>> marrow.server.http is asynchronous. I don't think we want to make such a
>> large leap from multithreaded to asynchronous in the default server.
>
>> CM: I'm uncomfortable with the direction of these libs personally. They
>> seem to be more researchy than practical in lots of cases.
>
> I would appreciate elaboration on these points, other than the async one
> which is based on an erroneous assumption. "Researchy" isn't even in my
> dictionary. ;)
Alice, thank you for correcting any misconceptions about Marrow, and
for documenting it on the Brainstorm page for future reference. I
apologize if I've made any mischaracterizations, either from reading
something that was false or interpreting it wrongly.
The Marrow suite looks like a well-thought-out, pythonic collection of
services. As such, it definitely serves as an example of one way
Pyramid could handle things in the future. It'll be several months
before we make any definite decision though. The developers need a
break from the year-long push to finish Pylons 1 and Pyramid 1, and
from building our own applications under fast-changing frameworks.
Pyramid 1 and Pylons 1 are stable now, and several of us would rather
focus on using them rather than changing them for a while. The Python
3 porting is enough work for this summer.
As you know, a few people are advocating various approaches for
replacing Paste*, PasteHTTPServer and the INI file. In the next few
months they'll be releasing them and writing HOWTOs for them. If you
feel like it, a HOWTO for adding Marrow enhancements to a typical
Pyramid or Akhet application would be welcome. After people have
gotten some experience using these approaches as third-party addons,
they'll be more comfortable saying whether some of them should be
integrated into Pyramid 2.
> My responses follow. On argparse:
>
>> Avoiding argparse is a feature, not a bug. If you want argparse, what
>> benefit does a scripting interface API give you? Effectively none.
>> marrow.script is designed to allow you to write new callables (or use
>> existing ones!) and have the command line interface Just Work. Even more
>> importantly, marrow.script does not provide a single central script to run;
>> your scripts aren't magically part of another package.
>> Optparse/argparse/getopt are terribly non-Pythonic, even though argparse had
>> the opportunity to fix things!
>
> Honestly, if you're highly desirous of using any of the existing stdlib
> parsers, you have no apparent need for a command line parsing library that
> adapts any function into a command line argspec.
I took a quick look at the Marrow parser both for Pyramid and for my
standalone utilities. My command-line options tend to have complex
specifications and interactions, and they don't always correspond
directly to function arguments. Perhaps if I thought more about it,
they could. But the "magic" in the decorators is not something I need,
even if others find it desirable. Argparse does what it does
reasonably well; I have not yet gotten sick enough of it to jettison
it.
For Pyramid, the developers seem to prefer conservative innovation.
Make incremental improvements, but not too radical. Obviously, Pyramid
is a radical departure from Pylons 1, but that's an occasional
paradigm shift akin to Python 3; it's not typical for all
Pyramid/Pylons development. And Pyramid did stick with the traditional
Paste* wrappings. Many users are still digesting the recent
innovations in Pyramid, and are converting existing Pylons programs,
or writing new ones from their existing mindset (whether their
background is Pylons, BFG, TurboGears, or another WSGI framework).
When Python makes a language change, the criteria is, "Is it obviously
the best way? Do the advantages significantly outweigh the bother of
changing? Both for the users and for the developers?" It takes time
for everybody to try out new libraries, and for a consensus to form
that one of them is obviously the best way and is a significant
advantage.
> Paster really provides a centralized command (paster) which only adds
> headspace confusion, and context-aware scripts which only add frustration;
> e.g. I can't open a shell unless I'm sitting in my source folder… on
> deployment there isn't a source folder!
Yes, those are a few of the problems with Paste*. But we (Pyramid)
need to stop and think awhile about what we want to replace it with.
> On marrow.server.http:
>
>> The underlying communication stack (marrow.io/server) is fundamentally
>> async. That particular fact has zero (and I mean it, zero) impact on writing
>> WSGI applications that run under the HTTP/1.1 implementation. The underlying
>> stack also has full support for a worker thread pool (using Futures) and
>> multi-processing. Both of these details are handled for you by the web
>> server; if you enable a thread pool, communication is still handled using
>> async, but the WSGI application is executed in a thread. This allows you to
>> tune a deployment for your requirements. Development or io-bound? Run a base
>> server. CPU-bound? Run a threaded server. Need to scale? Add
>> multi-processing to either of those.
>
> I'll mention it again, since this point is very important: the underlying
> server being async has ZERO impact on your own applications. It's not like I
> magically monkeypatched async support in everywhere; no, instead, only the
> core reactor for I/O (which makes sense) is async. The HTTP protocol
> implementation plugged into marrow.server either runs the WSGI application
> inline (which blocks other requests if not fast enough) or in a thread pool,
> your choice. (Choice is good! I have an RPC service running in pure async
> mode that can handle ~6KRsec; threading slows it down!)
If you can put this in a Pyramid/Marrow HOWTO, it would make users
more comfortable about trying it.
Most Pyramid/Pylons users come from a multithreaded background. They
either know or have learned how to make their views thread-safe. Many
have not used async servers and don't know how to make their views
async-safe. Normally it involves using special database libraries,
socket wrappers (if the view itself is invoking web services or doing
RPC etc), and also thinking about what else might be an intolerable
blockage (e.g., parsing or producing a multigigabyte file).
You say that Marrow users don't have to worry about any of these. In
that case, please explain in the HOWTO how to configure Pyramid/Marrow
for the various modes, the actual effect of typical blocking
constructs (SQLAlchemy, web services, etc) on the various modes, and
how Marrow's thread pool compares to PasteHTTPServer's thread pool.
Explaining it in the terminology that Pyramid and Pylons users are
familiar with will go a long way in making users more willing to try
Marrow, and the Pyramid developers more willing to consider Marrow.
>> Proper releases of marrow packages require 100% unit test coverage,
>> documentation, and multiple working examples.
In that case, suitable documentation may already be written. You'd
just need to paste some paragraphs into a HOWTO and reword it a bit
so that users can see how THIS [what I'm familar with in Pyramid]
corresponds to THAT [something new in Marrow]. The correspondence may
seem obvious to you, but not necessarily to users. And users may not
be willing to spend time figuring out what the correspondence is;
they'll just move on to another library.
> One of the easiest ways to examine these libraries are via the example code
> [2][3], and through the code for the libraries themselves. Additionally,
> you'll have to dig into a fork [4] to see demonstrations of WSGI1
> compatibility in marrow.server.http. The adapter [5] is simple, and likely
> incomplete, but functional.
I'm probably sounding redundant here, but only those who are
especially motivated to use Marrow will follow all those references
and figure out the correspondence between those examples and Pyramid.
Others will move on unless there's a Pyramid-specific HOWTO. Of
course, it may not be worth your while to write such a thing,
depending on how important Pyramid is to you. But if you seriously
want to spread Marrow adoption and/or Pyramid is important to you, it
would be worth your while.
--
Mike Orr <slugg...@gmail.com>
Alice Bevan–McGregor
On 2011-05-25 14:50:58 -0700, Mike Orr said:
> Alice, thank you for correcting any misconceptions about Marrow, and
> for documenting it on the Brainstorm page for future reference. I
> apologize if I've made any mischaracterizations, either from reading
> something that was false or interpreting it wrongly.
No worries! Besides, if something isn't documented properly or gives
the wrong impression, that's more my fault than anything else.
> The Marrow suite looks like a well-thought-out, pythonic collection of
> services. As such, it definitely serves as an example of one way
> Pyramid could handle things in the future. It'll be several months
> before we make any definite decision though.
Of course. Following the Zen, *right now* is rarely a good idea. ;)
> As you know, a few people are advocating various approaches for
> replacing Paste*, PasteHTTPServer and the INI file. In the next few
> months they'll be releasing them and writing HOWTOs for them. If you
> feel like it, a HOWTO for adding Marrow enhancements to a typical
> Pyramid or Akhet application would be welcome.
Somewhat difficult to write HOWTO's for a framework I don't currently
use. :P I'll see what can be done, however, as upcoming employment
does make use of Pyramid, which should give me a better 'working grasp'
of how to integrate my Marrow packages.
> I took a quick look at the Marrow parser both for Pyramid and for my
> standalone utilities. My command-line options tend to have complex
> specifications and interactions, and they don't always correspond
> directly to function arguments.
I would love to see some examples of complex interactions. If one of
your requirements are options that are only made available if another
option is enabled/defined, that's on the roadmap. :) Also,
sub-commands (hg/git style) are already supported in head using classes
instead of bare functions.
> Perhaps if I thought more about it, they could. But the "magic" in the
> decorators is not something I need, even if others find it desirable.
The decorators I used are adding a py3k feature to 2.x. Python 3
already includes a method for adding 'hinting' information to
arguments. These decorators are actually the examples given in the
decorator PEP! :D
> Argparse does what it does reasonably well; I have not yet gotten sick
> enough of it to jettison it.
For sure there is no point taking an existing working solution and
throwing it to the wind to re-engineer it under a different parser.
Marrow.script is primarily targeted at people who either have an
existing callable (class/method) they want to expose as a command-line
script, or for those who want to write really, really fast one-off
scripts.
OTOH it could greatly simplify the paster scripts I have thus far come
across in the wild.
> For Pyramid, the developers seem to prefer conservative innovation.
> Make incremental improvements, but not too radical. Obviously, Pyramid
> is a radical departure from Pylons 1, but that's an occasional paradigm
> shift akin to Python 3; it's not typical for all Pyramid/Pylons
> development.
Despite optional ZCML and the use of ZCA, Pyramid isn't, in my mind,
such a drastic departure from existing frameworks. Traversal (pre-
view lookup) is limited object dispatch using __getitem__ instead of
__getattr__. Woot. ;P The Paste components are identical, Beaker and
SA are popular, etc. For people migrating from TurboGears/Pylons or
any other framework derived from Paste, things should be /relatively/
comfortable.
> When Python makes a language change, the criteria is, "Is it obviously
> the best way? Do the advantages significantly outweigh the bother of
> changing? Both for the users and for the developers?" It takes time
> for everybody to try out new libraries, and for a consensus to form
> that one of them is obviously the best way and is a significant
> advantage.
I see it as less of a technological "best solution" war than a
marketing one. WebCore has been API stable for a year and a half and
critical bug-free for a year. Over 50 commercial projects have been
created and launched using it, spread between three primary users in as
many countries. Why hasn't it caught on with other users? For the
most part, no-one knows it exists despite my view that it trounces
other frameworks in simplicity, usability, and performance. (And uses
the exact same underlying Paste, WebOb, Beaker, SA, etc. stack w/ only
~300 lines of glue.)
What one might consider "best" is purely subjective, which is
unfortunate from a "let's make a standard" perspective. Advantage and
disadvantage comparison is also rather subjective. Hell, Django
decided to forego any external reusable dependencies…
> Yes, those are a few of the problems with Paste*. But we (Pyramid) need
> to stop and think awhile about what we want to replace it with.
I'd love to see a wiki page for each of these things with discussion,
requirements exploration, and pros/cons of different approaches.
> You say that Marrow users don't have to worry about any of these.
Not quite. m.s.http (the preferred way to abbreviate these modules,
since Paste isn't just an HTTP server, either ;) users need to choose
the options that best fit their requirements:
:: Base: Development, IO-bound or otherwise speedy execution.
:: Threading: CPU-bound or otherwise slow execution.
:: +Multi-Process: Scaling out instead of up.
If you need to debug things, nothing helps quite like only doing one
thing at a time. Having these options means properly understanding the
consequences of a developer decision: threading means you need to be
thread-safe. A base server means you need to be careful about blocking.
As an aside, that RPC service I mentioned uses its own internally
managed thread pool (Futures again) to defer/parallelize tasks away
from the request/response cycle and avoid blocking. Additionally, you
only get IO-bound benefit if you make use of marrow.io for your
communication, which will re-use/integrate with the marrow.server
reactor. (I avoid monkey-patching /anything/, but in theory it can be
done.)
> In that case, suitable documentation may already be written.
This is the documentation so far, and will be expanded upon in the
future. (E.g. adding a 'design / rationale' section.)
https://github.com/marrow/marrow.server.http/blob/draft/README.textile
> I'm probably sounding redundant here, but only those who are especially
> motivated to use Marrow will follow all those references and figure out
> the correspondence between those examples and Pyramid. Others will
> move on unless there's a Pyramid-specific HOWTO.
I'm posting on the Pyramid/Pylons *developer* list, not the user list,
so I'd expect the "open-mindedness"/curiosity level to be somewhat
higher, as I would also expect the tolerance to non-Pyramid-specific
content to be higher. Unfortunately, the majority of the Marrow
projects that could benefit Pyramid would require core changes, thus a
fork and a number of branches to explore the options, not just add-on
packages or adaptive HOWTOs.
> Of course, it may not be worth your while to write such a thing,
> depending on how important Pyramid is to you.
At the current time, Pyramid as a framework is not important to me. I
both do not use it and do not agree with a number of fundamental design
decisions. This may change in the future depending on how much of my
new job involves working on Pyramid-powered applications.
A lack of interest (or tolerance to reading code when HOWTOs aren't
present) from the Pyramid group on finding and integrating third-party
modules (Marrow or other, now or in the future) isn't really *my*
problem. Getting the Marrow suite operational, released, and polished
for my own projects is! ^_^
— Alice.
Mike Orr
<al...@gothcandy.com> wrote:
>> Argparse does what it does reasonably well; I have not yet gotten sick
>> enough of it to jettison it.
>
> For sure there is no point taking an existing working solution and throwing
> it to the wind to re-engineer it under a different parser. Marrow.script is
> primarily targeted at people who either have an existing callable
> (class/method) they want to expose as a command-line script, or for those
> who want to write really, really fast one-off scripts.
That *is* the issue. Marrow is ideal if you have a function like:
myfunc(a, b, c=1, d=None):
"a is a string; b is a list of strings; c is an int; d is an
optional string"
And you want to publish it without replicating your argument structure
in Argparse. On the other hand, those who are already using Argparse
may find Marrow unnecessary, too magic, and something else to learn.
But Pyramid is a different situation than either of these. The main
function in a pyramid application is:
main(global_conf=None, **local_conf):
"global_conf is an optional dict; local_conf is keyword args,
potentially parsed from a file"
settings = local_conf
init_component_1(settings)
init_component_2(settings)
'main' does not know what all the required arguments are. Instead,
each components finds the arguments it needs in the settings dict.
Theoretically there could be a validator at the beginning of the
function that checks all the settings. But none of this is specific
enough for @argument-like decorators on the function.
What Paste mainly does is to convert an INI file like this:
[app:myapp]
use = entrypoint#main
arg1 = value1
arg2 = value2
into a function call. "entrypoint#main" is an entry point that maps to
some callable. The other keys become arguments to that call. This is
something beyond what Marrow's arg parsing does. So anything that
replaces PasteDeploy would have to do this or the equivalent.
>> I'm probably sounding redundant here, but only those who are especially
>> motivated to use Marrow will follow all those references and figure out the
>> correspondence between those examples and Pyramid. Others will move on
>> unless there's a Pyramid-specific HOWTO.
>
> I'm posting on the Pyramid/Pylons *developer* list, not the user list, so
> I'd expect the "open-mindedness"/curiosity level to be somewhat higher, as I
> would also expect the tolerance to non-Pyramid-specific content to be
> higher. Unfortunately, the majority of the Marrow projects that could
> benefit Pyramid would require core changes, thus a fork and a number of
> branches to explore the options, not just add-on packages or adaptive
> HOWTOs.
I can see your point. It's just that it's not a case of Pyramid
choosing between alternatives it knows it needs. It's a case of you
promoting an alternative that Pyramid isn't sure it needs, so it wants
to make sure it does need it before putting a lot of time into it.
--
Mike Orr <slugg...@gmail.com>
Alice Bevan–McGregor
On 2011-05-26 16:11:45 -0700, Mike Orr said:
> On the other hand, those who are already using Argparse may find Marrow
> unnecessary...
There are always alternate parsers and the option to do it by hand.
Also, please ensure you mention a specific Marrow package rather than
the entire suite. (Like refering to Repoze when you actually mean
repoze.who.)
> ...too magic...
Python's introspection features, while 'advanced', are well known.
> ...and something else to learn.
Actually, the point of m.script is that there isn't anything else to
learn. If you can write a function, you can write a command-line
script. The two lines of boilerplate to tie your function to m.script
are unchanging and documented.
> But Pyramid is a different situation than either of these. The main
> function in a pyramid application is: [snip]
That "main" function relies on upstream configuration loading support.
As such, that function isn't the one you'd wrap in m.script. You'd
wrap the command that loads the configuration.
> main(global_conf=None, **local_conf):
The above becomes:
main(config, log=None, verbose=False, quiet=False, ...)
Where config would be the first required positional argument of the
script, generally the name of an INI file. If you wanted to do
something advanced like Paster, where the configuration can be passed
as STDIN by executing the INI as a shell script, you'd need to add a
few lines of code before the m.script boilerplate to determine if STDIN
is readable.
Replacing the functionality of PasteDeploy (the paster script
configuration parsing) was not the goal of m.script; m.script is just
command-line parsing. An additional component, marrow.config, will
include a m.script-powered command-line script to process YAML
configurations in a similar way to PasteDeploy, but I'm still
meditating on the exact feature set and implementation.
> What Paste mainly does is to convert an INI file like this: [snip]
Yup; m.config is the domain for that, not m.script.
> I can see your point. It's just that it's not a case of Pyramid
> choosing between alternatives it knows it needs. It's a case of you
> promoting an alternative that Pyramid isn't sure it needs, so it wants
> to make sure it does need it before putting a lot of time into it.
Indeed. I'll be using the Marrow suite regardless. ;)
Have a great night,
— Alice.
P.s. apologies if the last bit of my previous response sounded too
snippy; it was not intentional.
Mike Orr
<al...@gothcandy.com> wrote:
> Unfortunately, the majority of the Marrow projects that could
> benefit Pyramid would require core changes, thus a fork and a number of
> branches to explore the options, not just add-on packages or adaptive
> HOWTOs.
In that case, what might be better than a HOWTO is an outline of how
the packages could be integrated. I.e., which specific parts of
Pyramid would interface with which parts of the Marrow suite. If you
need help analyzing the Pyramid side, i may be able to help with that
; i.e., looking under the surface in Pyramid and what design
requirements it has at that particular point.
--
Mike Orr <slugg...@gmail.com>