Re: [Django] #3254: [patch] experimental fulltext search support for postgres, oracle and mssql
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Stefano Crosta <stefano@…>):
can this be useful? http://barryp.org/blog/entries/postgresql-full-text-
search-django/
There's a special "search" method for MySQL but still nothing for
PostgreSQL... any specific reason? This interesting ticket seems just
abandoned!
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:24>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: jpic (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:25>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: robinchew@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:26>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: niwi@… (added)
Comment:
If in the future, are going to implement support for postgresql text
search, you can have this as a reference implementation:
https://github.com/niwibe/djorm-ext-
pgfulltext/blob/master/djorm_pgfulltext/models.py
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:27>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: slav0nic@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:28>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: mpessas@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:29>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by mpessas):
This is definitely something I would like to see in Django and could help
with (see https://github.com/mpessas/django-pg-extensions for another
package that supports FTS in PostgreSQL).
Here is a suggestion as how this could be implmented for PostgreSQL:
- Add two extra options in TextField, use_fts=True and fts_config (see
http://www.postgresql.org/docs/9.3/static/textsearch-intro.html
#TEXTSEARCH-INTRO-CONFIGURATIONS), with 'simple' being the default value.
- When the use_fts option is set, create an index (see
http://www.postgresql.org/docs/9.3/static/textsearch-tables.html
#TEXTSEARCH-TABLES-INDEX). The main reasons to prefer an index to a column
for the tsvector representation are that this is simpler to set up (no
need for columns that are not exposed in Django and no need for triggers
or methods to update the column) and that, even if it is a bit slower,
most people would not mind; if they care for performance, they can do it
manually or use Solr etc. See http://www.postgresql.org/docs/9.3/static
/textsearch-tables.html for the details of the two approaches.
- The user can do a fts query by using the `search` field lookup.
I haven't looked into fts for some time now, so I might be forgetting
something. In any case, how does this proposal sound?
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:30>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => assigned
* owner: nobody => mjtamlyn
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:31>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
search | Patch needs improvement: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by MalikRumi):
I was just reading in the documentation that full text search is only
available for MySQL, and came over here to find out why. What can I do to
help light a fire under this project?
What are people doing in the meantime? Solr? Haystack?
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:32>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle fulltext | Needs documentation: 1
Has patch: 1 | UI/UX: 0
Needs tests: 1 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* keywords: oracle fulltext search => oracle fulltext search postgresql
postgres mysql
Comment:
I'd be happy to work on this ticket. I just implemented a very flexible
FTS integration with custom fields, and built in support for migrations to
automatically create the indices. If it's ok, I'd like to take the reigns
on this if no one else has time to.
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:33>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:34>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: tomi.kyostila@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:35>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by mjtamlyn):
The first parts of a FTS implementation in contrib.postgres are available
for review at https://github.com/django/django/pull/4726, as is my plan of
attack for the extent of features I wish to implement.
The approach is different to that used by previous implementations of full
text search with postgres, staying fairly close to the SQL structures and
making heavy use of the new expressions and lookups. As yet I have not
researched the potential for similar functionality in other databases, but
I expect at least parts of the contrib.postgres implementation to be
highly postgres specific. It may be there is a nice generic search API
with a cross database set of features which could be implemented. If this
is the case though, it will likely make heavy use of the code written for
postgres on that backend at least. I don't intend to work on a generic
solution at this point.
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:36>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: mjtamlyn
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Marc Tamlyn <marc.tamlyn@…>):
In [changeset:"2d877da85526bad0dad7fd6b1d56b1f924c0116a" 2d877da8]:
{{{
#!CommitTicketReference repository=""
revision="2d877da85526bad0dad7fd6b1d56b1f924c0116a"
Refs #3254 -- Added full text search to contrib.postgres.
Adds a reasonably feature complete implementation of full text search
using the built in PostgreSQL engine. It uses public APIs from
Expression and Lookup.
With thanks to Tim Graham, Simon Charettes, Josh Smeaton, Mikey Ariel
and many others for their advice and review. Particular thanks also go
to the supporters of the contrib.postgres kickstarter.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:37>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => new
* needs_better_patch: 1 => 0
* needs_tests: 1 => 0
* owner: mjtamlyn =>
* needs_docs: 1 => 0
* has_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:38>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by seenureddy):
Replying to [comment:34 Tim Graham]: Is anyone working on the full text
search support for oracle?
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:39>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"3d62ddb02695cfafc847951b45e6e1ec5dc0b149" 3d62ddb0]:
{{{
#!CommitTicketReference repository=""
revision="3d62ddb02695cfafc847951b45e6e1ec5dc0b149"
Refs #3254 -- Removed unnecessary truth check in
SearchVectorExact.as_sql().
Direct usage of the @@ operator is perfectly allowed.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:40>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Claude Paroz):
Is this ticket still open just for Oracle full-text search support?
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:41>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Carlton Gibson):
> Is this ticket still open just for Oracle full-text search support?
* I'm not sure if the SQLite criticisms in the original description still
stand — I see various projects making good use of FTS on SQLite.
* Wagtail just added a Search backend for everything **except** Oracle as
part of GSoC 2021 https://wagtail.io/blog/gsoc-2021-wrapup/ — Perhaps
there are things to learn from there.
* It would still be good to have `SearchVector` &co or similar available
on the other DBs than PostgreSQL.
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:42>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Matt Goldman (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:43>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Just as a note, SQLite supports FTS5 [https://sqlite.org/fts5.html docs]
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:44>
Django
-------------------------------------+-------------------------------------
Reporter: Ronny Pfannschmidt | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Alexander Nestorov (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:45>
Django
-------------------------------------+-------------------------------------
| Kumar Sen
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle fulltext | Triage Stage: Accepted
search postgresql postgres mysql |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => Akash Kumar Sen
Ticket URL: <https://code.djangoproject.com/ticket/3254#comment:46>