Re: Digest for django-developers@googlegroups.com - 16 Messages in 5 Topics

[Simon Lambert]

On 6 Jul 2010, at 04:45, django-devel...@googlegroups.com wrote:

  Today's Topic Summary

Group: http://groups.google.com/group/django-developers/topics

• Spam attack on the Wiki [2 Updates]
• Django Developers needed for Start-up.... Long Island, NY [1 Update]
• MySQL index hints [7 Updates]
• documentation re-factor for 1.3? [3 Updates]
• Django and the Eyjafjallajökull eruption [3 Updates]

 Topic: Spam attack on the Wiki

Christopher Petrilli <petr...@amber.org> Jul 05 12:18PM -0400 ^

 
Just an FYI, but there's been hundreds of edits to add spam links in
the last few hours. Might be worth locking it and rewinding the
database.
 
Chris
--
| Chris Petrilli
| petr...@amber.org


 

Alex Gaynor <alex....@gmail.com> Jul 05 12:13PM -0500 ^

 
I just went through and deleted all of his edits.
 
Alex
 
On Mon, Jul 5, 2010 at 11:18 AM, Christopher Petrilli
 
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me


 

 Topic: Django Developers needed for Start-up.... Long Island, NY

Norm Levy <norm...@mediastreetgroup.com> Jul 05 11:29AM -0400 ^

 
Do you know any Djangonaughts that are in the Long Island or New York area
that may be interested in Part Time or Full Time work?
 
I am in need of some Django developers for my upstart.... we created a few
web platforms using Django.....
 
http://ShoutOmatic.com - audible Tweet
http://LifeGoRound.com - social-photo aggregator that sends your online
content to your digital photo frame.
 
Thank you for any guidance ... or suggestions....
 
.. Working Remotely may be OK as well.... we need a minimum of 4 to 5 hours
a day.....
 
Equity-stake in the new Start-up is also part of the dicsussion
 
Norm Levy
 
---
 
http://MediaStreetGroup.com


 

 Topic: MySQL index hints

hinnack <henrik....@miadi.net> Jul 05 12:08AM -0700 ^

 
Thats interesting.
Can you explain, how the search keyword made it into the source?
Entry.objects.filter(headline__search="+Django -jazz Python")
SELECT ... WHERE MATCH(tablename, headline) AGAINST (+Django -jazz
Python IN BOOLEAN MODE);
Seems to be very MySQL specific...
 
regards
 
Henrik
 
 


 

Simon Litchfield <si...@slicmedia.com> Jul 05 12:59AM -0700 ^

 
> interested. 1 and 2 are fairly trivial; I can think of some obvious
> answers for 3, but 4 is the big problem, and will require some serious
> research and consideration.
 
Well, I'm glad you like the with_hints() approach. Items 1-3 are easy.
Re 4 though, every db does it differently. In practice, most don't
need hints like MySQL does <sigh>, because their query optimisers do a
much better job.
 
I'd be happy to use raw(); but then you lose len(), slicing,
pagination, filter chaining, etc. My problem came about because admin
change lists were unusably slow on large tables. With_hints allowed a
simple monkey patch to admin, dropping 2s per page down to ~0.01s.
 
MySQL is still the most used backend AFAIK, and it's the one that
really needs the hints (poor old thing it is). Adding with_hints()
will only serve to encourage support from other backends too, where
possible.
 
Maybe we can make the with_hints() syntax more generic with both
common and backend-specific kwargs --
 
.with_hints(index='my_index') (string implies index on queryset base
model)
.with_hints(index={Model:'my_index', RelatedModel:'index2'})
(dictionary allows defining index on a per-model basis)
 
So the Oracle backend, for example, could implement --
 
.with_hints(hash=True, index={Model:'my_index',
RelatedModel:'index2'}) (
 
On Oracle there are dozens of possible hints, so I'd say unsupported
kwargs could simply be ignored. This would ensure seamless database
portability.


 

Simon Litchfield <si...@slicmedia.com> Jul 05 01:01AM -0700 ^

 
> SELECT ... WHERE MATCH(tablename, headline) AGAINST (+Django -jazz
> Python IN BOOLEAN MODE);
> Seems to be very MySQL specific...
 
Yes Henrik, it's true- this is a classic example of a *very useful*
MySQL-specific feature already in the ORM.


 

Luke Plant <L.Pla...@cantab.net> Jul 05 01:11PM +0100 ^

 
On Mon, 2010-07-05 at 00:59 -0700, Simon Litchfield wrote:
 
> pagination, filter chaining, etc. My problem came about because admin
> change lists were unusably slow on large tables. With_hints allowed a
> simple monkey patch to admin, dropping 2s per page down to ~0.01s.
 
So the underlying problem is: how to rewrite a query (whether generated
by my code, someone else's code, or a mixture) so that it performs
better on my database. There will always be third party code which
generates queries that are really bad for performance, depending on your
project or your database. This also applies to tickets like
http://code.djangoproject.com/ticket/11604
 
This makes me think that what we need is a mechanism to do an equivalent
to your monkey patching that is easier and more general. Some kind of
"replace the auto-generated SQL with this SQL" method could be really
useful.
 
Could this be solved by writing your own cursor wrapper which checks the
SQL against a list and rewrites as necessary?
 
(You could set it up quickly like this:
http://chris-lamb.co.uk/2010/06/01/appending-request-url-sql-statements-django/
 
That still involves monkey patching, but it is possible to do the same
thing by writing a database backend).
 
The major drawback is fragility - doing a replacement at this level
could easily break, for example if you added a field to a model.
Adequate testing could catch this if your cursor wrapper had a debug
mode that could report what was replaced.
 
If we tried to do the replacement at a higher level, then you run into
the problem of how to recognise a query and replace it, which could be a
much more expensive operation if we are trying to compare objects of
type django.db.models.sql.query.Query, for example, and you also might
have the problem of not having full access to SQL statements.
 
However, if this method does work, if would be good to document how to
do it, and provide some better builtin hooks if necessary.
 
Luke
 
--
"We may not return the affection of those who like us, but we
always respect their good judgement." -- Libbie Fudim
 
Luke Plant || http://lukeplant.me.uk/


 

Russell Keith-Magee <rus...@keith-magee.com> Jul 05 08:16PM +0800 ^

 
> SELECT ... WHERE MATCH(tablename, headline) AGAINST (+Django -jazz
> Python IN BOOLEAN MODE);
> Seems to be very MySQL specific...
 
Yes; As implemented, __search is MySQL specific, and the API is
covered by our backwards compatibility policy, so we will continue to
support it. However, that doesn't mean it should be used as an example
or excuse for other database-specific features in the ORM.
 
The only explanation I can give for this particular feature is age.
The __search keyword was added over 4 years ago (Revision r3073,
ticket #573). Django was still in a period of rapid growth at that
time. Some of the features that were added at that time might not
survive if they were proposed today (either because of hindsight, or
because of changes in the perceived role of the ORM over time).
 
If a fulltext search operator were proposed today, it might still be
accepted -- after all, a query of text data using a rich full-text
search syntax is something that transcends SQL and could be considered
a primitive operation on object data. However, before I would accept
such a proposal, I would need to be convinced that the proposed syntax
would be supported on other backends. Postgres 8.3, for instance, has
fulltext search indexing capabilities. I would also need to be
convinced that the proposed syntax was rich enough to potentially
cover the range of features provided by MySQL. As it stands, we only
support queries "IN BOOLEAN MODE", not "IN NATURAL LANGUAGE MODE" or
"WITH QUERY EXPANSION".
 
Yours,
Russ Magee %-)


 

Russell Keith-Magee <rus...@keith-magee.com> Jul 05 08:38PM +0800 ^

 
> simple monkey patch to admin, dropping 2s per page down to ~0.01s.
 
> MySQL is still the most used backend AFAIK, and it's the one that
> really needs the hints (poor old thing it is).
 
I would like to know how you're validating your assertion that MySQL
is the most used backend. It doesn't match my experience or
observation.
 
The fact that this is a MySQL-specific issue is perhaps the biggest
impediment to my enthusiasm about this specific feature (as proposed).
I've spent enough time in the past covering for the inadequacies of
MySQL. I don't particularly relish the thought of adding more time to
that particular ledger. :-)
 
 
> On Oracle there are dozens of possible hints, so I'd say unsupported
> kwargs could simply be ignored. This would ensure seamless database
> portability.
 
Having an implementation ignore kwargs is the obvious approach to
providing this feature cross platform. The issue for me is what
exactly the kwargs should be, and how they would be interpreted. It
isn't clear to be how 'index="my_index"' maps to the application of
USE INDEX, IGNORE INDEX or FORCE INDEX in a query, or how the
dictionary syntax would map to the situation where you have two
appearances of a given table in a query.
 
Yours,
Russ Magee %-)


 

Simon Litchfield <si...@slicmedia.com> Jul 05 08:19AM -0700 ^

 
> I would like to know how you're validating your assertion that MySQL
> is the most used backend. It doesn't match my experience or
> observation.
 
Nobody knows for sure. I'd put my money on it though.
 
> I've spent enough time in the past covering for the inadequacies of
> MySQL. I don't particularly relish the thought of adding more time to
> that particular ledger. :-)
 
I know, I've seen you pulling your hair out :-)
 
> USE INDEX, IGNORE INDEX or FORCE INDEX in a query, or how the
> dictionary syntax would map to the situation where you have two
> appearances of a given table in a query.
 
Great, so lets define the syntax then and away we go.
 
1) I'd say index= would map to MySQL's USE INDEX and Oracle's index().
The others you mention there would be MySQL-specific kwargs. Or, more
simply, ignore and force could be '-index' and '+index'. Bit like we
use + and - for order_by.
 
2) The multiple table issue is edge case but worth supporting if it
doesn't complicate the syntax too much. Rather than use models as keys
we could use the filter/exclude name; eg {'relatedmodel__field':
'index'} would apply 'index' specifically to the 'relatedmodel__field'
join.


 

 Topic: documentation re-factor for 1.3?

Peter2108 <pe...@monicol.co.uk> Jul 05 01:43AM -0700 ^

 
As an 'intermediate' django user from time to time I need to look at
the Django code to fill out gaps in the documentation. I think this is
fine: "use the sourse, Luke" has always been a Python thing. But it
would be nice if developers could describe each parameter to the
function/method. This description should include the type of object(s)
the parameter is bound to.
 
Cheers,
 
Peter


 

Luke Plant <L.Pla...@cantab.net> Jul 05 12:30PM +0100 ^

 
On Sun, 2010-07-04 at 18:41 -0700, dffdgsdfgsdfhjhtre wrote:
 
 
> So? Matplotlib has similar documentation to PHP's reference, and it's
> fairly well regarded. The one thing that matplotlib doesn't do well is
> the topical stuff which django does do well.
 
Namespaces mean that the same name can refer to different things - like
'Field', all the different 'utils' modules etc. So you can't have a
shortcut that finds a single thing unless you also know which module it
belongs to, but that seemed to be the major problem you were
highlighting.
 
> > RegexValidator at the moment because of the lack of module directive).
 
> I did not know this... Maybe we should have documentation for the
> documentation....
 
Well, the front page says this in a box right at the top:
 
Looking for specific information? Try the Index, Module Index or the
detailed table of contents.
 
and it's the Index you need in this case.
 
> That takes 2 seconds max. For django, if you wanted to know the order
> of parameters the reverse() function takes, it will take you much
> longer.
 
That speed is because PHP has no namespaces (and makes little use of
classes). In this case, when you say "reverse" do you mean
django.core.urlresolvers.reverse or django.db.models.QuerySet.reverse?
 
So in Django there *has* to be at least one extra step. Equivalent in
Django docs:
 
1) type 'reverse' in the search box. Click on first hit, look at
'Contents' pane down the right, click 'reverse'.
 
2) Go to the General Index, click 'R', scroll to 'reverse' and choose
the one you want. (or, go to General Index and use your browser search
functionality e.g. type /reverse in Firefox.)
 
I guess what you are really asking for is a search box that returns only
exact matches from the index.
 
> we reorganize what we already have into two supersections; reference,
> and topical, since they are aimed at separate audiences and serve
> different purposes.
 
We *already* have this - the Module Index and Index (clearly linked from
the front page) are the reference sections, and most of the rest is
topical, with links to the reference sections thrown in.
 
> > (It already has some things, related to auto-escaping, that might make
> > it behave 'funny' depending on what exactly you pass it).
 
> Then that should all should be documented!
 
The fact that only documented functions are considered public *is*
documented: http://docs.djangoproject.com/en/dev/misc/api-stability/
(This is part of the meta-documentation section)
 
But the details of how the slugify function works should *not* be
documented. It is a *private* implementation detail (at the moment at
least). People who want to use it do so at their own risk, and should
read the source. If you want to propose that these functions be
considered public *as functions*, that is another matter. There are
always various things that ought to become public if they are
sufficiently useful outside of Django internals.
 
I don't happen to think django.template.defaultfilters is in that
category as it stands - the way those functions work is full of things
that are very specific to the template system.
 
> wrong with encouraging people to do this as long as you make it clear
> that these functions can move and that there may be side effects
> from auto-escaping and such.
 
But the point of the internal implementation details is that they can
*change* - if we say "You can use these provided you are aware of points
1), 2) + 3)", we now have to *keep* points 1), 2) and 3), and we can't
add points 4), 5) and 6). Without clairvoyance it is impossible to add
the caveats that we need to add. And we already have the
meta-documentation that says that only publicly documented code is
considered stable (and programmers should in general be able to infer
this anyway).
 
Regards,
 
Luke
 
--
"We may not return the affection of those who like us, but we
always respect their good judgement." -- Libbie Fudim
 
Luke Plant || http://lukeplant.me.uk/


 

Ramiro Morales <cra...@gmail.com> Jul 05 11:37AM -0300 ^

 
> (articles explaining all about one specific object such as slugify or
> the Feed class), and "topical" documentation (articles explaining how
> to do stuff like write templates).
 
On Sun, Jul 4, 2010 at 9:06 PM, dffdgsdfgsdfhjhtre <nbv...@gmail.com> wrote:
> people use. They already know what a bit of code does, what is
> important to them, is how the code worlks. Why a certain under-the-
> hood metaclass does what it does.
 
It is surprising how what you propose in line with something was already
discussed a couple of years ago and has been the general guideline
in the documentation front work since then:
 
http://groups.google.com/group/django-developers/browse_thread/thread/dcfaa00df58b6d3b
 
As Luke says, what is still missing is to complete the addition/separation
of reference material using the tools Sphinx give us, ideally reaching
100% of the
public API reference. This is also documented in a TODO list:
 
http://docs.djangoproject.com/en/1.2/internals/documentation/#todo
 
(that would be the "documentation documentation" document you
talk about)
 
I'd suggest to go forward with you showcasing idea, choose one module that
currently has no reference material, apply the strategy and share your results
with this list.
 
Regards,
 
--
Ramiro Morales | http://rmorales.net


 

 Topic: Django and the Eyjafjallajökull eruption

Mario Briggs <mario....@in.ibm.com> Jul 04 10:26PM -0700 ^

 
Hi Helgi,
 
that's an awesome story. We share your appreciation.
 
Rahul and Ambrish would be really satisfied and motivated if their
ibm_db_django adapter was used in this.
 
thanks
Mario
 


 

Massimiliano della Rovere <massimiliano...@gmail.com> Jul 05 08:44AM +0200 ^

 
Could a poll on the mailing lists (and djangopeople.net) be a way to
know how important for the django programmers View permission in admin
will be?
 
 
On Sat, Jul 3, 2010 at 09:36, Russell Keith-Magee


 

Helgi Borg <helg...@gmail.com> Jul 05 04:07AM -0700 ^

 
Thanks Russell, Josh, Gregor and others for helpful replies.
 
Josh, yes Django seems to be the Python framework of choice here in
Iceland.
 
Mario, yes we are using the ibm_db_django adapter, works fine, thanks.
 
Best regards,
Helgi Borg


 

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.