web2py-py3k

[Mariano Reingart]

I was bored so I started to port web2py to python 3, these are the
initial results:

CLI startup:

http://pic.twitter.com/7Ff85G1r

Welcome app:

http://pic.twitter.com/BR5PqAJ8

DAL has some issues with Sqlite that should be fixed, but my overall
sensation is optimistic ;-)

The changes were a mostly bytes <-> string and a lots of imports where fixed.

ASAP I'll make a repo clone in googlecode.

What do you think?

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Mariano Reingart]

The repo is at googlecode, reingart-web2py clone, py3k branch:

http://code.google.com/r/reingart-web2py/source/list?name=py3k

This is the diff:

http://code.google.com/r/reingart-web2py/source/detail?r=fab104bec8982d095375f9d6430b72f0ef87c6e8&name=py3k

The changes were made against changeset 2455:80b2c74561c0 tagged as R-1.99.2

Status:

- Almost all major web2py gluon modules are "working" (including DAL,
tested with sqlite)
- Admin is "working"
- Appadmin is "working"

"working" means that there are no major/serious issues, but there are
some minor issues with redirections, sessions, forms and ajax, nothing
that cannot be solved with some time, but it is working, i.e., you can
enter and register an user.

Manual changes (not made by 2to3) were very few, I count 46 lines:
* some unicode vs str/byte
* hasattr and raising AttributeError from __getitem__
* auth.__next__???
* relative imports (from . import saraza) and moved modules (email, http, etc.)
* contrib.simplejson is broken (replaced by json)
* has_key
* sort(key=
* hmac, md5, uuid fixes (needs bytes)

web2py apps where modified a little, I count 23 changes.
2to3 should be adapted/revised to understand web2py models,
controllers and views (I didn't tested this a lot so maybe this could
by my fault)

Attached is a diff filtered with the manual changes and applications
changes/diff.

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Massimo Di Pierro]

This is useful. Politically we should have a strategy.

If we are going to post to 3 how about:
- we rebuild web2py on top of bottle
- we store session in cookies (bottle does it)
- we store ticktes in database (if available) as filesystem as failover
- we cleanup request/response and perhaps we use the bottle ones
- we make all actions restful (not sure how, perhaps using decorators)
- we remove crud completely
- we rewrite forms and validator logic (the way widgets, validators and html are interlaced is very messy)

> --
> mail from:GoogleGroups "web2py-developers" mailing list
> make speech: web2py-d...@googlegroups.com
> unsubscribe: web2py-develop...@googlegroups.com
> details : http://groups.google.com/group/web2py-developers
> the project: http://code.google.com/p/web2py/
> official : http://www.web2py.com/

[Anthony]

Are you suggesting doing a web3py now? How would that affect the continued development of web2py?

- we rebuild web2py on top of bottle

Would Bottle be a dependency, or would it be forked? What are the advantages of building on Bottle?

 

- we store session in cookies (bottle does it)

So does Flask. Would the cookie be (at least optionally) encrypted? Also, would it be worth having a server-side option as well -- if the session holds a lot of data, you might not want to keep sending it back and forth (I think there's also a 4KB limit on cookies).

 

What about exec? What about some of these other ideas: https://groups.google.com/d/msg/web2py-developers/55_7Kr3jDjI/0008xQ036uYJ

Anthony

[Mariano Reingart]

Mmm, I was thinking exactly the opposite :(

Are you speaking of web2py or web3py?

Maybe my later posts are not clear, I'll try to clarify.

I think we have something good here:

1. Porting web2py to py3k is really easy, 2to3 does most of the work,
only small and specific parts need manual changes (mainly wsgibase
wrapper).

46 lines needed to be changed for the proof of concept, surely a
hundred will be need to a final production release for py3k.
Let's be pessimistic and lets say a thousand lines have to be changed.
That the 1% of the code.
It think it is a minimal impact, in fact most changes could be made in
advance and we could have web2py compatible with py2 and py3 (after
using 2to3, of course).
Why we will rewrite large parts?

2. If we were using py2.7, web2py applications could be run UNCHANGED
between py2 and py3.
This is largely because web2py transparent "handles" unicode, web2py
helpers abstract python standard libraries that have been changed and
the exec model hides import problems.
I think this is the MOST IMPORTANT point.
The only minor issue that is preventing this is "except XX as x"
syntax, that is mandatory on py3 and is not available until py2.7
Even 2to3 could not be necessary, anyway we could make a web2py-2to3
at runtime to ease the transition to final users, so they don't have
to bother even for the syntax changes.
Which other framework lets you run your app in py2 and py3 without changes?

And that is my point: breaking backward compatibility is a bad signal.
I think it doesn't matter if web3py will break it, we are saying
web2py has no future.

Python3 is 2 to 7 years ahead, if I have to start a big project (in my
case, an ERP that is a big investment and needs a lot of funding and
time), I will not use web3py that doesn't exist yet, nor web2py that
could have a dead end.

If the message were exactly the opposite, i.e., that anyone could use
web2py apps in python2 and python3 with minimal or no changes, I
would not doubt even for a second to use it.

Said that, you could see my opinion can be biased, I'm basing on
web2py for my career (degree thesis) and my work (ERP system, already
started), so sorry if this sounds arrogant or selfish and I want to
stick with web2py, please take your own conclusions.

BTW, breaking forward compatibility is also bad IMHO (specifically
form.process and @auth.requires_signature IIRC), I have many complains
of my students that are having troubles with this.

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Bruno Rocha]

+1 for bottle as base and other things, but why do you want to remove crud?

--

--

Bruno Rocha

[ About me: http://zerp.ly/rochacbruno ]

[ Aprenda a programar: http://CursoDePython.com.br ]

[ O seu aliado nos cuidados com os animais: http://AnimalSystem.com.br ]

[ Consultoria em desenvolvimento web: http://www.blouweb.com ]

[Bruno Rocha]

I also agree with Mariano, I am working on a big project using web2py, I do not want to port it to Py3k but I want to use web2py for a long time.

We have to keep web2py up and being developed, updated even if web3py became real.

for now I am more interested in solving web2py issues, improve things already running in 2py as better forms (html5), reports, scheduller, run in PyPy etc than porting to py3k.

we can start it, but as a side project not as a main goal for now, I like the idea of using bottle as base, but it needs to be a different project, different users list, different website.. other thing.

I dont want we2py to die for py3k.

[Bruno Rocha]

Said that, you could see my opinion can be biased, I'm basing on
web2py for my career (degree thesis) and my work (ERP system, already
started), so sorry if this sounds arrogant or selfish and I want to
stick with web2py, please take your own conclusions.

+1 here.

I left my job on a great company to work exclusivelly with web2py, I am doing it for more than one year and I am having preety much success with this, I have some small systems (CRM, CMS, websites, stores) running in clients. And I am building a startup (already funded) working with web2py too.

Also I teach web2py and I am an advocate here in Brazil, it is not good to say web2py can die because of 3. may be having a fork to run in py3k is better for now.

 

BTW, breaking forward compatibility is also bad IMHO (specifically
form.process and @auth.requires_signature IIRC), I have many complains
of my students that are having troubles with this.

I can't see how this features broke compatibility? 

[Massimo Di Pierro]

I have nothing against this. I think the "official" web2py shlud stay 2.x for now. I have nothing against building a 3.x version. I agree people will see as a positive thing. Yet I think we should take the occasion to consider something better than web2py.

[Massimo Di Pierro]

On Oct 16, 2011, at 12:45 AM, Anthony wrote:

Are you suggesting doing a web3py now? How would that affect the continued development of web2py?

- we rebuild web2py on top of bottle

Would Bottle be a dependency, or would it be forked? What are the advantages of building on Bottle?

I think we can make web2py faster.

 

- we store session in cookies (bottle does it)

So does Flask. Would the cookie be (at least optionally) encrypted?

Yes. I think bottle does it already. If we eliminate the dependency on writing on file system it will avoid some of the problems we are having running on GAE and Heroku.

Also, would it be worth having a server-side option as well -- if the session holds a lot of data, you might not want to keep sending it back and forth (I think there's also a 4KB limit on cookies).

We can have an if statement and optionally store extra data in database.

What about exec? What about some of these other ideas: https://groups.google.com/d/msg/web2py-developers/55_7Kr3jDjI/0008xQ036uYJ

We will have to modify the custom_import and we create a module reload function but perhaps it is possible to get away with exec. Not that we have to but we could since people do not seem to understand that logic, if we work out another way to achieve the same goals we should try.

[Massimo Di Pierro]

I really do not like crud. Crud exists because SQLFORM lacks an abstract representation and only works in HTML.

[Jonathan Lundell]

On Oct 16, 2011, at 6:26 AM, Massimo Di Pierro wrote:

- we store session in cookies (bottle does it)

So does Flask. Would the cookie be (at least optionally) encrypted?

Yes. I think bottle does it already. If we eliminate the dependency on writing on file system it will avoid some of the problems we are having running on GAE and Heroku.

We also depend on the session file for request serialization via file-system locking. 

[Anthony]

 Good point. How do prevent race conditions with sessions stored in cookies?

[Jonathan Lundell]

Or for that matter with sessions stored in the database (GAE-style). As I read the code, there's no serialization (unless GAE locks on a select, which seems highly unlikely).

One of the nice aspects of file-system session locking is that it's granular: locking is at the session level.

If we uncommented this line in Session:

                 # rows[0].update_record(locked=True)

...then with a database adapter like sqlite3, the entire database is locked until the commit at the end of the request (in MySQL the table is locked, I think). That's pretty drastic; the performance hit would be considerable.

I'm not remotely a database expert, and there are some subtleties in the DAL that affect this stuff, at least for some databases. For example, with MySQL we implicitly create InnoDB tables, which determines the transaction mechanism & policy, and by default we turn off autocommit, which also affects transaction handling. 

GAE has a run_in_transaction capability, but web2py doesn't seem to use it.

[Mariano Reingart]

On Sun, Oct 16, 2011 at 1:18 PM, Jonathan Lundell <jlun...@pobox.com> wrote:
> On Oct 16, 2011, at 7:58 AM, Anthony wrote:
>
> On Sunday, October 16, 2011 10:24:09 AM UTC-4, Jonathan Lundell wrote:
>>
>> On Oct 16, 2011, at 6:26 AM, Massimo Di Pierro wrote:
>>>
>>> - we store session in cookies (bottle does it)
>>
>> So does Flask. Would the cookie be (at least optionally) encrypted?
>>
>> Yes. I think bottle does it already. If we eliminate the dependency on
>> writing on file system it will avoid some of the problems we are having
>> running on GAE and Heroku.
>>
>> We also depend on the session file for request serialization via
>> file-system locking.
>
>  Good point. How do prevent race conditions with sessions stored in cookies?
>
> Or for that matter with sessions stored in the database (GAE-style). As I
> read the code, there's no serialization (unless GAE locks on a select, which
> seems highly unlikely).

Database Locking semantics : select_for_update and lock
(django is getting that)

exec is good, reload is VERY BAD, if some people don't like exec, that
is their problem, again, I don't see anyone saying "PostgreSQL has to
remove PlPython stored procedural language because it uses exec-like
approach or global variables ", and believe me, if postgres uses it,
it should be good.
also, reload is removed from py3!!!!
exec has proven to be one of the smartest decisions of web2py
regarding ease of use, ease of maintance and deploy (automatic
"reload"), organized structure, and now, py3k compatibility.
I see feasible only a CodeObject approach, but that is, and exec camouflaged

BTW, is not better the current approach like Martin bootle + dal?, so
anyone can start with web2py and if he really really needs a lot of
performance, he could reuse their models, views and controllers (with
minimal modification) in other frameworks
Why to reinvent the wheel?
IMHO I think that approach will be better viewed by the python
community, and I already it is the interchangeable approach django and
other framework are taking.

Also, I don't think major web2py problems are in external interfaces,
I think they are inside web2py:
* Keyed table vs normal table "duality", migrations execution order
and lack of fixtures
* database locking semantics (as stated before), postgresql returning
and many other tweaks
* Template system issues (syntax errors, pass and }}), I'l interested
in exploring .py pure views
* Lack of "debug" bar

The first two are very serious issues for major systems, the last two
are annoying not only in the teaching room, also can be time-consuming
to debug complex apps.

Finally, I'm not saying web2py has to move to py3k right now, we could
be aware of this and start a long-term planned transition (I think it
is minimal, according my web2py-py3k proof of concept), but we need to
have a statement about this to clarify the roadmap.
If you give me access permissions to create a wiki page, I could start
to write that, I think mailing lists messages are not clear enough.

[Jonathan Lundell]

On Oct 16, 2011, at 9:45 AM, Mariano Reingart wrote:

> Database Locking semantics : select_for_update and lock
> (django is getting that)

Useful, but not very portable and too coarse for session locking, at least in MySQL, where IIRC it locks the whole table.

[Mariano Reingart]

Select for update should lock only affected record (ie, the session
being read), it is almost the same query and the lock can be
READ/WRITE or READONLY.
What locks the whole table are insert/updates/deletes in certain MySQL
storage engines (myisam IIRC)

Anyway, if the database has a poor implementation, it is not a web2py fault ;-)

Also, PostgreSQL has recently added support for 'unlogged' tables that
can speed up sessions a lot.

BTW, I forgot to reply how forward compatibility can be an issue, this
are the steps: create an app in the latest web2py release, then try to
use that app in former releases, it will fail. I don't saying that
APIs must be frozen, but just a couple of "if" statements in the
welcome app could have prevent this nasty issues (sadly I didn't have
time to make such patch).

[massimo....@gmail.com]

You  made excellent points

From my Android phone on T-Mobile. The first nationwide 4G network.

-------- Original message -------- Subject: Re: [w2py-dev] web2py-py3k From: Mariano Reingart To: web2py-d...@googlegroups.com CC:

On Sun, Oct 16, 2011 at 1:18 PM, Jonathan Lundell <jlun...@pobox.com> wrote:
> On Oct 16, 2011, at 7:58 AM, Anthony wrote:
>
> On Sunday, October 16, 2011 10:24:09 AM UTC-4, Jonathan Lundell wrote:
>>
>> On Oct 16, 2011, at 6:26 AM, Massimo Di Pierro wrote:
>>>
>>> - we store session in cookies (bottle does it)
>>
>> So does Flask. Would the cookie be (at least optionally) encrypted?
>>
>> Yes. I think bottle does it already. If we eliminate the dependency on
>> writing on file system it will avoid some of the problems we are having
>> running on GAE and Heroku.
>>
>> We also depend on the session file for request serialization via
>> file-system locking.
>
>  Good point. How do prevent race conditions with sessions stored in cookies?
>
> Or for that matter with sessions stored in the database (GAE-style). As I
> read the code, there's no serialization (unless GAE locks on a select, which
> seems highly unlikely).

Database Locking semantics : select_for_update and lock
(django is getting that)

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Jonathan Lundell]

On Oct 16, 2011, at 10:35 AM, Mariano Reingart wrote:

> On Sun, Oct 16, 2011 at 2:08 PM, Jonathan Lundell <jlun...@pobox.com> wrote:
>> On Oct 16, 2011, at 9:45 AM, Mariano Reingart wrote:
>>
>>> Database Locking semantics : select_for_update and lock
>>> (django is getting that)
>>
>> Useful, but not very portable and too coarse for session locking, at least in MySQL, where IIRC it locks the whole table.
>>
>
> Select for update should lock only affected record (ie, the session
> being read), it is almost the same query and the lock can be
> READ/WRITE or READONLY.
> What locks the whole table are insert/updates/deletes in certain MySQL
> storage engines (myisam IIRC)
>
> Anyway, if the database has a poor implementation, it is not a web2py fault ;-)

No, except for GAE and SQLite, which we have other reasons for wanting to support well.

On the whole, though, I'm in favor of supporting useful features of (major) databases even if we might not be able to hide the differences completely.

[Martín Mulone]

I'm planning to use bottle+dal as a tool to teach web framework to entry level python users. I think bottle is easy to learn python because you can do all in one file, and there is no model pattern that, perhaps lost the new ones. Also DAL is easy to understand than any orm, specially for people that came from php and sql.

My idea is that dal act as a plugin in bottle. A possible roadmap in this order:

+ PiPy (done) easy to install: pip install bottle-web2pydal

+ Document and make some examples

+ Test make some populate and verify data.

+ html.py (Helpers) URL=Web2py dependant

+ validators.py (Validators)

+ html.py FORM .submit(), rewrite process(), NO CLRF but do validation.

+ SQLFORM sqlhtml.py rewrite process(), NO CLRF but do validation and insertion.

+ Rewrite some session using dal.

+ SQLFORM sqlhtml.py rewrite process()

Returning to web2py ...

I think we need to start thinking in a future. First step before python3 we need to make the step on pep8, and deprecation warnings, in the followings modules:

dal.py (all we agree this module is important an means that our data access persist in the future), html.py, validators.py, sqlhtml.py, storage.py, utils.py, widgets.py, etc.

I can start making pep8, but we need to be synchronized in some way, perhaps we need to wait to github.                                     

2011/10/16 Mariano Reingart <rein...@gmail.com>

--
mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details    : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official    : http://www.web2py.com/

--

 http://martin.tecnodoc.com.ar

[Massimo Di Pierro]

I am sorry I have not yet the time to deploy on github. Will do it next week.

[DenesL]

Hi Mariano,

could you expand on point 1 table "duality" and point 3 template
issues?.

Denes.

[Mariano Reingart]

Duality: normal tables are not compatible with keyed tables (i.e.
references), IIRC there is duplicate code to maintain for both
approach (and not only on dal, look for hasattr("_privatekey") in
sqlhtml, appadmin, etc).
In some cases, using surrogate keys (id field) is not a good practice,
and collides with many existing databases (for performance, conceptual
design, normalization, etc. it is not always recomended using
autonumeric fields)

I think dal could be further improved (separating table creation and
contraint in migrations, more expressiviness and custom database
feature support) but having to maintain two approach is blocking, and
I think this is one of the most important issue that may prevent
enterprise-level adoption.

Template issues:

* You could not write large python code blocks without using pass, I
don't see the point on having views only for HTML or text oriented
contents, for binary data they are clunky (I had to make a pie char
image as an alternate view, and I had to make other controller, a .png
view didn't work)
* Detecting opening and closing {{ and }} marks maybe tricky or
confusing, i.e., closing two dicts, you have to left an space, is is
easy to forget to put a { or } (or put an additional one) and mess up
all the view
* Finally, the most important one: views are difficult to debug as
there is no direct relation of error messages / line numbers with
actual .html contents

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[DenesL]

On Oct 21, 10:54 am, Mariano Reingart <reing...@gmail.com> wrote:
> Duality: normal tables are not compatible with keyed tables (i.e.
> references),

keyed tables need to know which field you are referring to among all
the fields of the table (since there is no id field usually), if I
remember correctly I could not find a way around that.

> IIRC there is duplicate code to maintain for both
> approach (and not only on dal, look for hasattr("_privatekey") in
> sqlhtml, appadmin, etc).

"_primarykey" is used to tell apart keyed tables and since they are
different they require different handling, no way around this.

> In some cases, using surrogate keys (id field) is not a good practice,
> and collides with many existing databases (for performance, conceptual
> design, normalization, etc. it is not always recomended using
> autonumeric fields)

which is why keyed tables were added.

> I think dal could be further improved (separating table creation and
> contraint in migrations, more expressiviness and custom database
> feature support) but having to maintain two approach is blocking, and
> I think this is one of the most important issue that may prevent
> enterprise-level adoption.

improvement is always welcome but IMO the duality is what gave web2py
access to legacy enterprise data expanding its potential use base.

> Template issues:
>
> * You could not write large python code blocks without using pass, I
> don't see the point on having views only for HTML or text oriented
> contents, for binary data they are clunky (I had to make a pie char
> image as an alternate view, and I had to make other controller, a .png
> view didn't work)

a while back Massimo was against having large code blocks in views
(proper MVC) and I tend to agree, do that in the controller which is
pure python,

I would like other view types too, something configurable/pluggable.

> * Detecting opening and closing {{ and }} marks maybe tricky or
> confusing, i.e., closing two dicts, you have to left an space, is is
> easy to forget to put a { or } (or put an additional one) and mess up
> all the view
> * Finally, the most important one: views are difficult to debug as
> there is no direct relation of error messages / line numbers with
> actual .html contents

it could show the python code that is being exec-ed as an alternative.

Denes.

[Mariano Reingart]

On Fri, Oct 21, 2011 at 5:02 PM, DenesL <dene...@yahoo.ca> wrote:
>
>
> On Oct 21, 10:54 am, Mariano Reingart <reing...@gmail.com> wrote:
>> Duality: normal tables are not compatible with keyed tables (i.e.
>> references),
>
> keyed tables need to know which field you are referring to among all
> the fields of the table (since there is no id field usually), if I
> remember correctly I could not find a way around that.

that should be in the _primarykey

>> IIRC there is duplicate code to maintain for both
>> approach (and not only on dal, look for hasattr("_privatekey") in
>> sqlhtml, appadmin, etc).
>
> "_primarykey" is used to tell apart keyed tables and since they are
> different they require different handling, no way around this.

there is a workaround: convert 'id' fields to _primarykey

I'm not wanting to remove keyed tables, just I think they can be
unified with "normal" id tables, to avoid different handling

>> In some cases, using surrogate keys (id field) is not a good practice,
>> and collides with many existing databases (for performance, conceptual
>> design, normalization, etc. it is not always recomended using
>> autonumeric fields)
>
> which is why keyed tables were added.
>
>> I think dal could be further improved (separating table creation and
>> contraint in migrations, more expressiviness and custom database
>> feature support) but having to maintain two approach is blocking, and
>> I think this is one of the most important issue that may prevent
>> enterprise-level adoption.
>
> improvement is always welcome but IMO the duality is what gave web2py
> access to legacy enterprise data expanding its potential use base.

yes, but there is some code not very compatible with keyed tables, and
having different approach I think it doesn't help to test and improve
dal and the other modules

I'd some issues with this, but I think a bit of cleanup and
simplification could be good, I'll make a proposal patch when I get
some time.

>> Template issues:
>>
>>  * You could not write large python code blocks without using pass, I
>> don't see the point on having views only for HTML or text oriented
>> contents, for binary data they are clunky (I had to make a pie char
>> image as an alternate view, and I had to make other controller, a .png
>> view didn't work)
>
> a while back Massimo was against having large code blocks in views
> (proper MVC) and I tend to agree, do that in the controller which is
> pure python,
>
> I would like other view types too, something configurable/pluggable.

Me too, I don't want more code in the html views, I just would love to
bypass .html in certain scenarios. and write .py directly.

>
>>  * Detecting opening and closing {{ and }} marks maybe tricky or
>> confusing, i.e., closing two dicts, you have to left an space, is is
>> easy to forget to put a { or } (or put an additional one) and mess up
>> all the view
>>  * Finally, the most important one: views are difficult to debug as
>> there is no direct relation of error messages / line numbers with
>> actual .html contents
>
> it could show the python code that is being exec-ed as an alternative.

IIRC it shows the code somewhere but I've to do a quick and dirty hack
in py3k because it was showing html as python code in the traceback
(not useful at all, the html and py lines don't even match)

I would like something more friendly, ie, showing a comment with the
original line number of the html file, or some kind of metadata to
show where the error is in the original file (having a syntax error
checker on save would be great!)

Again, when I found some time I'll send a patch with this proposals.

[DenesL]

On Oct 21, 6:45 pm, Mariano Reingart <reing...@gmail.com> wrote:

> On Fri, Oct 21, 2011 at 5:02 PM, DenesL <denes1...@yahoo.ca> wrote:
>
> > On Oct 21, 10:54 am, Mariano Reingart <reing...@gmail.com> wrote:
> >> Duality: normal tables are not compatible with keyed tables (i.e.
> >> references),
>
> > keyed tables need to know which field you are referring to among all
> > the fields of the table (since there is no id field usually), if I
> > remember correctly I could not find a way around that.
>
> that should be in the _primarykey
>
> >> IIRC there is duplicate code to maintain for both
> >> approach (and not only on dal, look for hasattr("_privatekey") in
> >> sqlhtml, appadmin, etc).
>
> > "_primarykey" is used to tell apart keyed tables and since they are
> > different they require different handling, no way around this.
>
> there is a workaround: convert 'id' fields to _primarykey
>
> I'm not wanting to remove keyed tables, just I think they can be
> unified with "normal" id tables, to avoid different handling

it maybe doable but I did not plan on changing Massimo's original
table design

maybe Massimo can give us a hint on where this should be added in.
.pdf .py .haml ... <drooling>