Re: change NIP-01 to only allow a single filter per `REQ` (PR #1645)

[William Casarin]

On Fri, May 30, 2025 at 06:07:44PM -0700, fiatjaf_ wrote:
>fiatjaf left a comment (nostr-protocol/nips#1645)
>
>> Each filter is just a large OR clause and we know from SQL that those
>> are some of the easiest to optimize since they tend to reuse the same
>> indexes multiple times.
>
>What do you mean? Are they the easiest to optimize in comparison to
>what? I may be wrong but I'm pretty sure any database, when faced with
>a `"ids":["a", "b", "c"]` (or, equivalently, `WHERE id = "a" OR id =
>"b" OR id = "c"`) will have to do 3 index lookups on an "id" index.
>This is what strfry, my eventstore implementations,
>[pocket](https://github.com/mikedilger/pocket) and probably nostrdb do
>and I don't think there is another way.

nostrdb is pretty bad at handling lots of different kinds and tags
mashed into one filter, in fact it's probably broken. one reason I like
this PR is that it allows for simpler queries, which allows for faster
and more correct results (at least on the nostrdb side since my code
sucks);

>> I don't think we have a single dev that knows how to analyze and
>> create different execution plans based on the incoming filters.

yeah

>I'm curious about what you mean. The query planning in all the
>dedicated Nostr event stores I looked is hardcoded based on the
>existing indexes and the attributes in the filter. Something like:
>
>```
>if filter.ids: use "ids" index
>if filter.kinds and filter.authors: use "pubkey-kind-created_at" index
>if filter.tags["e"] or filter.tags["E"]: use "tags-created_at" index
>if filter.kinds: use "kinds-created_at" index
>if filter.authors: use "pubkey-created_at" index
>if filter.tags: use "tags-created_at" index
>else: use "created_at" index
>```
>
>But still a single filter is broken in multiple lookups in each of
>these indexes if the filter has more than one value in the attribute
>that will be consulted.

here's ours:

// this is rougly similar to the heuristic in strfry's dbscan
if (search) {
return NDB_PLAN_SEARCH;
} else if (ids) {
return NDB_PLAN_IDS;
} else if (relays && kinds && !authors) {
return NDB_PLAN_RELAY_KINDS;
} else if (kinds && authors && authors->count <= 10) {
return NDB_PLAN_AUTHOR_KINDS;
} else if (authors && authors->count <= 10) {
return NDB_PLAN_AUTHORS;
} else if (tags && tags->count <= 10) {
return NDB_PLAN_TAGS;
} else if (kinds) {
return NDB_PLAN_KINDS;
}

It works best when the filter is simple.

Cheers,
jb55


--
Reply to this email directly or view it on GitHub:
https://github.com/nostr-protocol/nips/pull/1645#issuecomment-2923876647

Message ID: <nostr-protocol/nips/pull/1645/c2923...@github.com>