TurboJSON and PEAK-Rules
Jonathan LaCour
am having an absolute *fit* with TurboJSON after the switch to
PEAK-Rules. I don't want to use any of the default jsonify rules,
and before with the older version of TurboJSON, I was able to call
`jsonify.clear()` so that I wouldn't have any conflicts. It now
appears that this capability is gone, and the documentation for PEAK
Rules is essentially non-existent.
So, now I have a massive set of rules that are all blowing up with
`AmbiguousMethod` exceptions and no way that I can determine a way
to fix them.
Honestly, I don't see any point for TurboJSON to include any default
rules whatsoever, since none of them really save much in terms of
code, and create more headaches than they are worth, especially since
you *can't* override them as far as I can tell.
--
Jonathan LaCour
http://cleverdevil.org
Florent Aide
<jonatha...@cleverdevil.org> wrote:
>
> I am upgrading my application to TurboGears 2.0, and in the process
> am having an absolute *fit* with TurboJSON after the switch to
> PEAK-Rules. I don't want to use any of the default jsonify rules,
> and before with the older version of TurboJSON, I was able to call
> `jsonify.clear()` so that I wouldn't have any conflicts. It now
> appears that this capability is gone, and the documentation for PEAK
> Rules is essentially non-existent.
>
> So, now I have a massive set of rules that are all blowing up with
> `AmbiguousMethod` exceptions and no way that I can determine a way
> to fix them.
I'm on the fence on that one but I must admit I have been bothered by
this too. I had no preexisting rules so just went the __json__ way but
I see your point...
Anyone?
Florent.
Roger Demetrescu
I am +1 on moving those rules into a function that could be called
once by the developer.
(to maintain compatibility).
Cheers,
Roger
Mark Ramm
> once by the developer.
> (to maintain compatibility).
This sounds like a good plan, and we can call that rule in json.py in
the quickstart template.
But this would be a breaking change for some folks, so we should document it.
--Mark
Jonathan LaCour
Great idea. I'd love to see this included in another alpha release
really soon now (along with the `wrap_app` change discussed
earlier). With these two changes, I am pretty sure my app will run
100% under TG 2.0.
Mark Ramm
> really soon now (along with the `wrap_app` change discussed
> earlier). With these two changes, I am pretty sure my app will run
> 100% under TG 2.0.
Well, the wrap_app change had been in since r4823.
So, it's pretty much this TurboJson change you still need.
--Mark Ramm
Alberto Valverde
I think this could be solved in two ways which would allow default rules
to be bundled (after all, TG2 is all about providing sane defaults to make
simple things simple, right?)
1) Implement a custom Method that honors a "priority" (optional, default
to 0) argument to the when/around decorators which will be used to
disambiguate. This is what I had in TW when it used rule dispatch and used
in many projects of mine and worked like a charm. Default rules in
turbojson would have a priority=1000 (for example) so when an ambiguous
methods situation emerges from user code extending jsonify the user would
just need to raise his/hers implementation's priority.
I will probably implement this soon for my gsoc project so if interested I
could make it a standalone egg (unless PJE wants to include it in PEAK
itself :)
2) Add context to jsonify so it can be dispatched on. This can be achieved
by implementing jsonify as a method of GenericEncoder, example:
class GenericEncoder(JSONEncoder):
@generic()
def jsonify(self, value):
....
When generic functions live inside a class peak rules also uses 'self' to
dispatch on giving preference to extensions declared inside the class body
when the instance method is called (I'm not 100% sure though, only 95%.
RuleDispatch did and the little I've played with peak-rules makes me
believe it does too but I haven't any unittest to prove it yet)
Users could then subclass turbojson's encoder and extend jsonify from there:
jsonify = GenericEncoder.jsonify.im_func
class MyEncoder(GenericEncoder):
@jsonify.when("obj is an SA mapped instance")
def _my_sa_jsonifier(self, value):
return something useful
Instances of this encoder will use _my_sa_jsonifier to encode SA instances
overriding any rule declared en GenericEncoder. Rules declared outside of
the class body will still affect all GenericEncoder subclasses (unless one
does isinstance(self, Foo) of course).
I believe 2 is the "correct" way to do it but the change might not be
transparent enough. Besides that, some mechanism should be implemented to
tell TG2 which encoder it should use.
1, however, is 100% backwards compatible (but needs someone to implement
it :), as I've said, I'll probably do it but I first need to study
peak-rules internals in more detail)
To sum up, I strongly believe turbojson should ship with default rules as
it has always done since it saves developer time by providing sane
defaults which is what TG is about. Introspecting SA classes, heck, even
extending generic functions, is not something someone new to python would
like to face too soon IMHO
Alberto
Christoph Zwerschke
> 2) Add context to jsonify so it can be dispatched on. This can be achieved
> by implementing jsonify as a method of GenericEncoder, example:
>
> class GenericEncoder(JSONEncoder):
> @generic()
> def jsonify(self, value):
> ....
I was also thinking about this, because it could help removing another
small wart in TurboJSON, namely the handling of config options.
The GenericEncoder is instantiated twice: When importing jsonify, a
global instance is created, using default options, and when
instantiating JsonSupport (the JSON templating engine), an additional
instance is created, passing the proper config options on to
JSONEncoder. Also, the specialized functions have no way of accessing
these options. That's why turbojson.descent_bases has to be accessed in
an obscure way, needing to import turbogears.config which leads to
circular imports with the Pylons nose plugin. See also:
http://groups.google.de/group/turbogears-trunk/browse_thread/thread/c0f9e8b6c4ca4ca2
If we use methods instead of functions, it would be easier to access
additional configuration options such as turbojson.descent_bases.
-- Christoph
Jonathan LaCour
> I think this could be solved in two ways which would allow default
> rules to be bundled (after all, TG2 is all about providing sane
> defaults to make simple things simple, right?)
TG2 is about making simple things simple, but its not about doing it
at the expense of complex things. Bottom line: both of your options
make it much more difficult to write a simple custom jsonifier, and
don't really solve the problem in a simple way.
I want to have this at the top of my `json.py`:
@jsonify.when((Person,))
def jsonify_person(obj):
...
This is much simpler to write, and is way easier to explain to a new
user
than something like this:
@jsonify.when("isinstance(obj, Person) and crazy_disambiguator(1001)")
def jsonify_person(obj):
...
Imagine having to write this 50-60 times (in my case)!
New users don't want to think about things like "generic function
disambiguation". I think your ideas are both fine ideas, but they
don't eliminate the need to be able to turn the defaults off. I
think the simplest approach here is the best: put the defaults in
a separate module, and have them be imported by default in the
TG2 json.py template in your project. If you want to remove the
defaults, you can simply delete that line from your `json.py` and
you're set!
The simplest solution is the best, IMO :)
Alberto Valverde
> Alberto Valverde wrote:
>
>
>> I think this could be solved in two ways which would allow default
>> rules to be bundled (after all, TG2 is all about providing sane
>> defaults to make simple things simple, right?)
>>
>
> TG2 is about making simple things simple, but its not about doing it
> at the expense of complex things. Bottom line: both of your options
> make it much more difficult to write a simple custom jsonifier, and
> don't really solve the problem in a simple way.
>
simple? (and it might solve another bug for free as Chris Z has pointed out)
> I want to have this at the top of my `json.py`:
>
> @jsonify.when((Person,))
> def jsonify_person(obj):
> ...
>
> This is much simpler to write, and is way easier to explain to a new
> user
> than something like this:
>
> @jsonify.when("isinstance(obj, Person) and crazy_disambiguator(1001)")
> def jsonify_person(obj):
>
Option 1 involves *no* change in user code since TJ's implementations
can have a lower priority than the default so any extension made by the
user automatically overrides the built ins (I thought I had emphasized
this point...). Anyway, the "crazy_disambiguator" is just a simple
optional argument to the decorators, ie:
@jsonify.when((Person,), 1)
------------------------------------------^
> Imagine having to write this 50-60 times (in my case)!
>
The following command could save you some time ;)
$ find . -name '*.py' -exec sed -e -i 's/\(@jsonify.*\))$/\1, 1)/g' {} ';'
> New users don't want to think about things like "generic function
> disambiguation".
New users don't need to understand how the sausage is made in order to
eat it, I don't get your point.
> I think your ideas are both fine ideas, but they
> don't eliminate the need to be able to turn the defaults off.
> I think the simplest approach here is the best: put the defaults in
> a separate module, and have them be imported by default in the
> TG2 json.py template in your project. If you want to remove the
> defaults, you can simply delete that line from your `json.py` and
> you're set!
>
So if you only need to override how Person is jsonified you suddenly
need to copy and paste (or re-implement for the fun of it) the rule to
jsonify all other SA mapped objects from tg since you turned it turned
it off en masse.
What if a some tg extension in the future needs to add rules to jsonify
their objects so they can be reused by user code? Now those rules are
disabled too, need to copy them again... suddenly all the power that
generic functions give framework developers to add hooks so their code
can be easily extended is lost.
Both solutions I proposed allow to *selectively* extend the encoder
while preserving rules any extension might have added or might need to add.
> The simplest solution is the best, IMO :)
>
I fully agree, but I believe I've excelled in my previous at making them
sound much harder than they really are ;)
Alberto
Jonathan LaCour
> Option 2 involves subclassing TJs encoder, in what way isn't that
> simple? (and it might solve another bug for free as Chris Z has
> pointed out)
Maybe I misunderstood, would rules still be able to be functions,
rather than methods? I was under the impression that this method
would involve changing the way that jsonification rules are
written. If that isn't the case, then my objections to this method
are completely withdrawn :)
> Option 1 involves *no* change in user code since TJ's
> implementations can have a lower priority than the default so any
> extension made by the user automatically overrides the built ins
> (I thought I had emphasized this point...).
Yeah, in my exhausted stupor, I must have missed that detail :) As
long as the user's rules *always* override the defaults, I am a
happy man! Objection withdrawn to this solution.
> The following command could save you some time ;)
>
> $ find . -name '*.py' -exec sed -e -i 's/\(@jsonify.*\))$/\1,
> 1)/g' {} ';'
Ah, sed! Incidentally, this is exactly what I would have done :)
> So if you only need to override how Person is jsonified you
> suddenly need to copy and paste (or re-implement for the fun of
> it) the rule to jsonify all other SA mapped objects from tg since
> you turned it turned it off en masse.
Well, this is what I was doing, because I actually don't want every
single field of my SQLAlchemy objects to be serialized. I want to be
able to explicitly control it, which is why I don't like there being
no way to turn off the defaults! Having complete control over the
JSONification of my objects is extremely important to me.
> Both solutions I proposed allow to *selectively* extend the
> encoder while preserving rules any extension might have added or
> might need to add.
Sure, but I still want to be able to turn them off, even if it
does break some "future" extension. The bottom line for me is that
I totally understand (and agree with) your arguments, but I still
think that there needs to be a mechanism for clearing out the
default rules to give the user complete control over them.
Thanks for your hard work, Alberto!
Alberto Valverde
Alberto Valverde wrote:Option 2 involves subclassing TJs encoder, in what way isn't that simple? (and it might solve another bug for free as Chris Z has pointed out)Maybe I misunderstood, would rules still be able to be functions, rather than methods? I was under the impression that this method would involve changing the way that jsonification rules are written. If that isn't the case, then my objections to this method are completely withdrawn :)
jsonify = GenericEncode.jsonify.im_func
Option 1 involves *no* change in user code since TJ's implementations can have a lower priority than the default so any extension made by the user automatically overrides the built ins (I thought I had emphasized this point...).Yeah, in my exhausted stupor, I must have missed that detail :) As long as the user's rules *always* override the defaults, I am a happy man! Objection withdrawn to this solution.The following command could save you some time ;) $ find . -name '*.py' -exec sed -e -i 's/\(@jsonify.*\))$/\1, 1)/g' {} ';'Ah, sed! Incidentally, this is exactly what I would have done :)
So if you only need to override how Person is jsonified you suddenly need to copy and paste (or re-implement for the fun of it) the rule to jsonify all other SA mapped objects from tg since you turned it turned it off en masse.Well, this is what I was doing, because I actually don't want every single field of my SQLAlchemy objects to be serialized. I want to be able to explicitly control it, which is why I don't like there being no way to turn off the defaults! Having complete control over the JSONification of my objects is extremely important to me.
@jsonify.around("instantance(obj, Person)")
def jsonify_person(next_method, obj):
dct = next_method(obj) # call TJ's implementation
del dct['swiss_bank_acc_no']
return dct
Both solutions I proposed allow to *selectively* extend the encoder while preserving rules any extension might have added or might need to add.Sure, but I still want to be able to turn them off, even if it does break some "future" extension. The bottom line for me is that I totally understand (and agree with) your arguments, but I still think that there needs to be a mechanism for clearing out the default rules to give the user complete control over them. Thanks for your hard work, Alberto!
http://toscawidgets.org/trac/rum/browser/rum/prioritized_methods.py
With tests and docs (for a change ;) You can use the prioritized_when decorator
in there *now* to fix your TJ troubles until TJ adopts a definitive solution (I'll probably release it as a standalone egg when it gets more eyeballs).
from turbjson.jsonify import jsonify
from prioritized_methods import prioritized_when
jsonify.when = prioritized_when.__get__(jsonify)
@jsonify.when("isinstance(obj, Person"), prio=1)
def jsonify_person(obj):
....
(don't actually need the prio argument to override ATM since prioritized_when overrides peak.rules.when by default)
Alberto
Alberto Valverde
http://pypi.python.org/pypi/prioritized_methods/
Any interest around here in me getting my dirty hands on TurboJson to
integrate it?
Plan is:
1) Replace jsonify.around and jsonify.when with their prioritized_*
counterparts but with their old name.
2) Add a prio=-1 argument when registering all implementations inside TJ
so user extensions automatically override them if they create an
ambiguous methods situation (if a builtin has a more specific rule it
will still apply, of course)
Changes should be 100% backwards compatible and Jonathan will be happy ;)
Alberto
Alberto Valverde
> Sure, but I still want to be able to turn them off, even if it
> does break some "future" extension. The bottom line for me is that
> I totally understand (and agree with) your arguments, but I still
> think that there needs to be a mechanism for clearing out the
> default rules to give the user complete control over them.
>
>>> from peak.rules import Dispatching, when
>>> def f(n): pass
...
>>> f(6)
>>> when(f, "n>5")(lambda n: n)
<function <lambda> at 0x862325c>
>>> f(6)
6
>>> rule_set = Dispatching(f).rules
>>> rule_set.rules
[Rule(<function f at 0x862341c>, (), None, 121), Rule(<function <lambda>
at 0x862325c>, Test(Comparison(Local('n')), Range((5, 1), (Max, 1))),
None, 122)]
>>> rule_set.remove(rule_set.rules[1])
>>> f(6)
I'm loving this PEAK-rules thing... :)
Alberto
Alberto Valverde
> Jonathan LaCour wrote
>
>> Sure, but I still want to be able to turn them off, even if it
>> does break some "future" extension. The bottom line for me is that
>> I totally understand (and agree with) your arguments, but I still
>> think that there needs to be a mechanism for clearing out the
>> default rules to give the user complete control over them.
>>
>>
> Just found out how you can do this:
>
http://paste.turbogears.org/paste/3141
Alberto
Jonathan LaCour
> http://paste.turbogears.org/paste/3141
I had already discovered this and attempted it myself, and it didn't
fix my problem. I put it at the top of my `json.py` module, and
it doesn't seem to get rid of the default rules, resulting in an
AmbiguousMethods exception.
Nice try, though :)
Jonathan LaCour
> Just a FYI, I've released prioritized_methods at pypi:
> http://pypi.python.org/pypi/prioritized_methods/
>
> Any interest around here in me getting my dirty hands on TurboJson
> to integrate it?
Since I'm the squeaky wheel, I'll give this a shot today. I'll let you
know how it goes :)
Jonathan LaCour
> Since I'm the squeaky wheel, I'll give this a shot today. I'll
> let you know how it goes :)
Well, I tried, and failed :)
Looks like there is a bug somewhere in prioritized_methods. When
I replace the peak.rules `when` with the `prioritized_when`, all
sorts of things go boom. Specifically, all of the default rules in
TurboJSON raise exceptions about not being able to find imported
modules.
I can get a little further by replacing the `isinstance` checks with
concrete type checking, but anything that passes a string-based rule
that depends on something in the local namespace (imported modules
like `datetime`, or helper functions like `is_saobject`), things
blow up...
Any ideas, Alberto?
Alberto Valverde
I think I've fixed this in a recent commit, update:
easy_install -U prioritized_methods==dev
and try again please.
Alberto
Jonathan LaCour
> I think I've fixed this in a recent commit, update:
>
> easy_install -U prioritized_methods==dev
>
> and try again please.
It works now. Patch for TurboJson trunk attached. Let me know if
there is anything else you need. I'd love for this to be included in
the upcoming preview release.
Thanks, Alberto!
Alberto Valverde
I've just created a turbojson branch and integrated prioritized_methods.
with the pm's dev version all TJ tests pass untouched and a new one I've
added passes too.
http://svn.turbogears.org/projects/TurboJson/branches/prioritized/
If this works for you I'll release a fixed prioritized_methods to pypi and
if no one objects I'll merge this TJ branch with the trunk too.
Alberto
Alberto Valverde
> Alberto Valverde wrote:
>
>> I think I've fixed this in a recent commit, update:
>>
>> easy_install -U prioritized_methods==dev
>>
>> and try again please.
>
> It works now. Patch for TurboJson trunk attached. Let me know if
> there is anything else you need. I'd love for this to be included in
> the upcoming preview release.
Seems we got a mid-air collision :) I made an almost identical diff in the
branch I mentioned + a test, if it's ok with you I'll leave the changes in
the branch untouched since they're almost the same and they have a test,
ok?
> Thanks, Alberto!
no problem :)
Alberto
Jonathan LaCour
> If this works for you I'll release a fixed prioritized_methods
> to pypi and if no one objects I'll merge this TJ branch with the
> trunk too.
Works great for me.
Alberto Valverde
> Alberto Valverde wrote:
>
>
>> If this works for you I'll release a fixed prioritized_methods
>> to pypi and if no one objects I'll merge this TJ branch with the
>> trunk too.
>>
>
> Works great for me.
>
I've merged it already. I found a couple minor quirks in
prioritized_methods which I've fixed but don't feel confident enough in
making a release yet (I already made 3 today). Anyway, TJ requires the
dev version which will be pulled out from the mercurial archive whenever
it is is installed so it is not much of a problem.
I'll make a release of both in a few days hopefully when the dust
settles down a bit.
Alberto
Christopher Arndt
> I've merged it already. I found a couple minor quirks in
> prioritized_methods which I've fixed but don't feel confident enough in
> making a release yet (I already made 3 today). Anyway, TJ requires the
> dev version which will be pulled out from the mercurial archive whenever
> it is is installed so it is not much of a problem.
You mean, you need mercurial installed to install TurboJson? IMHO,
that's unacceptable. Please bear in mind that TurboJson is also used by
the TG 1.1 branch, for which we hope to issue a beta release soon. We
have worked hard to ensure that TG 1.0 is installable with just Python
and no other development tools and I think it would be very bad if we
regress from there without need.
So, if you make TJ depend on prioritized_methods, please make sure that
you make a release of priorized_methods available at PyPI, before you do
a TJ release.
Chris
Alberto Valverde
> Alberto Valverde schrieb:
>
>> I've merged it already. I found a couple minor quirks in
>> prioritized_methods which I've fixed but don't feel confident enough in
>> making a release yet (I already made 3 today). Anyway, TJ requires the
>> dev version which will be pulled out from the mercurial archive whenever
>> it is is installed so it is not much of a problem.
>>
>
> You mean, you need mercurial installed to install TurboJson? IMHO,
> that's unacceptable. Please bear in mind that TurboJson is also used by
> the TG 1.1 branch, for which we hope to issue a beta release soon. We
> have worked hard to ensure that TG 1.0 is installable with just Python
> and no other development tools and I think it would be very bad if we
> regress from there without need.
>
Mercurial doesn't have to be installed to "easy_install" mercurial
archives. These are created on the fly from the repository tip,
*zipped*, and served by hgweb. easy_install couldn't care less if the
zip was dynamically generated or if is was a static file gathering dust
in some hard drive platter ;)
Take a look if you want to see for yourself at the links labeled
"download zip" in http://toscawidgets.org/download. This URL is listed
as "download_url" in pypi for prioritized_methods so easy_install looks
there when it sees pypi has no 'dev' version registered there.
However, I would like some feedback from someone who can try the new
version with a TG1 app that makes some good use of custom jsonify rules
before making a release, just in case. Although I'm pretty sure that if
Jonathan's app didn't break with jsonify 50-60 rules we're going to be
ok... :)
Cheers,
Alberto
Christopher Arndt
> Christopher Arndt wrote:
> Mercurial doesn't have to be installed to "easy_install" mercurial
> archives. These are created on the fly from the repository tip,
Ok, I see.
> *zipped*, and served by hgweb. easy_install couldn't care less if the
> zip was dynamically generated or if is was a static file gathering dust
> in some hard drive platter ;)
> Take a look if you want to see for yourself at the links labeled
> "download zip" in http://toscawidgets.org/download. This URL is listed
> as "download_url" in pypi for prioritized_methods so easy_install looks
> there when it sees pypi has no 'dev' version registered there.
I still don't like idea of TJ having a dependency on a development
version of another project. Please make a release and require that.
Chris
Florent Aide
>> Take a look if you want to see for yourself at the links labeled
>> "download zip" in http://toscawidgets.org/download. This URL is listed
>> as "download_url" in pypi for prioritized_methods so easy_install looks
>> there when it sees pypi has no 'dev' version registered there.
>
> I still don't like idea of TJ having a dependency on a development
> version of another project. Please make a release and require that.
+1
Alberto Valverde
Ok, I've just released 0.2 to pypi [1] and updated turbojson's setup.py
accordingly [2].
The only development version TJ still depends on is PEAK-Rules itself, I
have no control over that package though.
Alberto
[1] http://pypi.python.org/pypi/prioritized_methods/0.2
[2] http://trac.turbogears.org/changeset/4920