tgext.admin so slow that it is practically unusable

[Juraj Variny]

Hi,

does here anybody actually use admin extension with meaningful amounts of data? When I started having 1000+ records in test database, just listing them took several seconds. I have looked with debugger what it is doing and it seems that:

* Regardless of paging, all records in the table are fetched and for every one record extra select query is done
* If there is one-to many relationship, also all records from related table are fetched and for every one record extra select query is done

Or tgext.admin is meant to be this way and I have it badly configured? This happened both with sqlite and postgres.

[Alessandro Molina]

Currently tgext.admin has not been optimized at all for performances
and as you noticed it retrieves every related object instantly.
I tend never to use it with more than a few hundred records, when I
need to manage big collections I tend to write custom get_all methods.

There is for sure an huge space for optimizations in sprox on that
topic right now.
I'll try to give a look as soon as possible but I cannot guarantee you
to roll out a new release in a short time as that is a part of sprox
that I didn't write myself.

> --
> You received this message because you are subscribed to the Google Groups
> "TurboGears" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/turbogears/-/hGZLwVYcVDEJ.
> To post to this group, send email to turbo...@googlegroups.com.
> To unsubscribe from this group, send email to
> turbogears+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/turbogears?hl=en.

[Juraj Variny]

Thanks for reply! At first glance most of the problem could be fixed by passing correct limit and offset parameters from paginator all the way down to provider.query(), I was surprised the support seems to be there but parameters are not passed. I don't have time right now to dig deeper, must finish and ship the site soon. Fortunately tgext.admin was planned only for backend usage and can be avoided/fixed later.

Juraj

Dňa štvrtok, 6. septembra 2012 15:31:33 UTC+2 Alessandro Molina napísal(-a):

[Alessandro Molina]

I came up with a solution for pagination issue that seems not to break
compatibility with past code or CrudRestController subclasses. The
real issue was not retrieving the paginated set of data, as you said
that is just a matter of passing limit and offset to sprox, the issue
was to make the paginator behave correctly with that set of data.

I'll see if there is space for a possible tgext.crud and tgext.admin
release with the other TG guys so that your issue can be fixed with a
simple upgrade.

> https://groups.google.com/d/msg/turbogears/-/i4Tr_veGbfEJ.

[Michael Pedersen]

Alessandro: If you have the fix ready to go, make sure it's available
on github, and I'll get it out this week.

Michael J. Pedersen
My Online Resume: http://www.icelus.org/ -- Google+ http://plus.ly/pedersen
Google Talk: m.ped...@icelus.org -- Twitter: pedersentg

[Alessandro Molina]

the fix is already on github, as there is some work going on to add
jinja2 support to admin and crud I would tag a release at:

tgext.crud -> 19ea2947e1973ca1e31835d6c7e290cd59c35031
tgext.admin -> eef2553b8a35fc79c804b827a8417263be928459

So that we don't include jinja support which still is not complete yet
(I have to check and merge another pull request)

[Michael Pedersen]

I'm sorry. I lost track of this task. It's done, and tgext.admin and
tgext.crud v.0.5.4 are both out, and at the same revision that
Alessandro specified below.