RavenSession.Advanced.AsyncLuceneQuery with spatial and multiple query "factors"

[Chad Thiele]

Here's my index:

docs.Places.Select(p => new {
    Query = new object[] {
        p.__document_id,
        p.translatedName,
        p.name,
        p.nickNames,
        p.categories.Select(x => x.name),
        p.location.city,
        p.location.state,
        p.location.address
    },
    Tags = p.tags,
    Categories_Id = p.categories.Select(x => x.id),
    _ = SpatialIndex.Generate(((double ? ) p.location.lat), ((double ? ) p.location.lng))
})

I'm trying to find all places in a given category, with multiple tags, a keyword, within 25km of a coordinate... asynchronously.

I've attempted something like this:

var query = RavenSession.Advanced.AsyncLuceneQuery<Place>("PlacesByQuery")
    .Statistics(out stats)
    .WithinRadiusOf(distance, lat, lng);

    foreach (var tag in tagList) {
        query.AndAlso().Search("Tags", tag);
    }

   query.AndAlso().OpenSubclause().Search("Query", keyword).CloseSubclause();

   query.AndAlso().WhereIn("Categories_Id", categoryList);

   query.Skip(0).Take(25);
   var places = await query.ToListAsync();

If I use a category (categoryList contains only one category id in this scenario), 2 tags, a keyword, and lat/lng I know should return a value... I get 0 results. 

I've tried to use only spatial and tags. Still 0 results.

var query = RavenSession.Advanced.AsyncLuceneQuery<Place>("PlacesByQuery")
    .Statistics(out stats)
    .WithinRadiusOf(distance, lat, lng);

    foreach (var tag in tagList) {
        query.AndAlso().Search("Tags", tag);
    }

   query.Skip(0).Take(25);
   var places = await query.ToListAsync();

I essentially want to get to the point where a user can pick a category, any number of tags, and enter a keyword to search within a predefined coordinate. Seems like everything works until I add the tags code (query.AndAlso().Search("Tags", tag);). 

Perhaps .Search() isn't the appropriate method? I know this must be a horribly ugly mess, there must be a better way.

Thanks!

[Chad Thiele]

Changed the tags part of the query to this:

query.AndAlso().WhereIn("Tags", tagList);

and it works a bit better, but I want to return places that are tagged with every tag passed to the query.

For example: a query with three tags supplied ("kid friendly", "smoke free", and "accepts credit card") should only return places that have those three tags, not a collection of all places that have any combination of those tags.

[Oren Eini (Ayende Rahien)]

Call ToString() on your query, and you'll likely see the issue.

You are saying: Give me a place that match ALL the tags in this distance AND matches the query AND is in this categories.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

Can you create a failing test?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--

[Chad Thiele]

Yea... doing .toString() shows this:

@in<Tags>:("kid friendly","date spot")  AND ( Query:(demode)) AND @in<Categories_Id>:(4bf58dd8d48988d14e941735)

It's clear that @in<Tags> would be all inclusive. If you look at the index, you'll see Tags = p.tag. On the place class, tags are a List<string>.

though it would probably work, I'm assuming it's not good practice to do something like:

@in<Tags>:("kid friendly") AND @in<Tags>:("date spot")  AND ( Query:(demode)) AND @in<Categories_Id>:(4bf58dd8d48988d14e941735)

or is that appropriate here?

[Oren Eini (Ayende Rahien)]

Note that IN is for _any_, not _all_.

So that makes sense in the results you get.

Can you show a failing test? 

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

[Chad Thiele]

There's no failing code, just 0 results. 

Here's the model:

string Place {
  string Id;
  string name;
  List<string> tags;
  List<Category> categories;
  Location location;
  ...
}

Here's the index as copied from RavenStudio:

docs.Places.Select(p => new {
    Query = new object[] {

        p.name,
        p.categories.Select(x => x.name)

    },
   Tags = p.tags,
   Categories_Id = p.categories.Select(x => x.id),
    _ = SpatialIndex.Generate(((double ? ) p.location.lat), ((double ? ) p.location.lng))
})

I'm trying to build a UI where the user can pick a category, any number of tags, and a keyword they enter. The query should return only places that match the category, has every tag they selected, and the keyword they entered.

I've tried to modify the query a bit, to split the tags into their own AND parts, like so:

@in<Tags>:("kid friendly") AND @in<Tags>:("smoke free") AND ( Query:(cheese*)) AND @in<Categories_Id>:(4bf58dd8d48988d1c5941735)

This one finds a place named "Cheese & Olive." So it would seem it works, because "Cheese & Olive" is indeed in that category, has both "kid friendly" and "smoke free" tags, and has "cheese*" in the name.

However,

@in<Tags>:("kid friendly") AND ( Query:(cheese*)) AND @in<Categories_Id>:(4bf58dd8d48988d1c5941735)

returns 0 results. Which seems strange, "Cheese & Olive" does match this criteria.

[Oren Eini (Ayende Rahien)]

Please create a test as shown here: http://ravendb.net/docs/2.5/samples/raven-tests/createraventests

I want to see that it doesn't return the results, but it should.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

[Chad Thiele]

I shall try.

[Chad Thiele]

Trying to add RavenDB Test Helpers package throws an error:

"Updating 'System.Spatial 5.2.0' to 'System.Spatial 5.0.2' failed. Unable to find a version of 'RavenDB.Database' that is compatible with 'System.Spatial 5.0.2'."

On Thursday, July 24, 2014 7:17:02 PM UTC+9, Oren Eini wrote:

[Oren Eini (Ayende Rahien)]

--DependencyVersion Highest

[Chad Thiele]

I don't know what to do now... looks like the test app works.

One thing I noticed is that when I do .toString() on the query in the test app, it shows the spatial part of the query:

@in<Tags>:(kidFriendly)  AND @in<Tags>:(englishFriendly)  AND @in<Categories_Id>:(4bf58dd8d48988d1c5941735) SpatialField: __spatial QueryShape: Circle(139.349905 35.745064 d=25.000000) Relation: Within

The one in the app I'm having problems with looks like this:

@in<Categories_Id>:(4bf58dd8d48988d1bd941735)  AND @in<Tags>:(kidFriendly)  AND @in<Tags>:(englishFriendly)

The way I build that query is this:

var query = RavenSession.Advanced.AsyncLuceneQuery<Place>("PlacesByQuery").WithinRadiusOf(distance, lat, lng);

query.AndAlso().WhereIn("Categories_Id", new List<string> { cat });

foreach (var tag in tagList) {

    query.AndAlso().WhereIn("Tags", new List<string> { tag });

}

var places = await query.ToListAsync();

The only difference I can spot from the test app is that my main app is trying to use AsyncLuceneQuery and .ToListAsync().

One other note, the app's raven database is hosted on RavenHQ.

[Oren Eini (Ayende Rahien)]

Are you using the same version of the client?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

[Chad Thiele]

Hmm, in the main app it's 2.5.2879. In the test app it's 2.5.2907.

I updated the main app to 2.5.2907, however it did not resolve the issue.

I went to RavenHQ, logged into my RavenStudio, and performed the query from the studio.

I used this query:

@in<Categories_Id>:(4bf58dd8d48988d1c5941735)  AND @in<Tags>:(englishFriendly)  AND @in<Tags>:(kidFriendly)

SpatialField: __spatial QueryShape: Circle(139.349905 35.745064 d=25.000000) Relation: Within

You can see it got back 2 results. In the image you can see that one of the places has both tags, one does not. I've also attached a screenshot of the index as it appears inside RavenStudio.

[Oren Eini (Ayende Rahien)]

Note that you cannot issue spatial queries like that in the studio, you have to use the UI (check the sptail query).

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--

[Chad Thiele]

Ah, didn't know that. Re-ran the query using the UI, but got the same results.

When checking the spatial checkbox, the fields cover up the main part of the query... so I included the bottom of the studio where it shows the http code of the query.

[Oren Eini (Ayende Rahien)]

Can you send me the link on the bottom?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

On Thu, Jul 24, 2014 at 5:08 PM, Chad Thiele <chad....@gmail.com> wrote:

Ah, didn't know that. Re-ran the query using the UI, but got the same results.

When checking the spatial checkbox, the fields cover up the main part of the query... so I included the bottom of the studio where it shows the http code of the query.

--

[Chad Thiele]

Sure: https://diver.ravenhq.com/databases/Chaddeus-AMBvNext/indexes/PlacesByQuery?&query=%40in%3CCategories_Id%3E%3A(4bf58dd8d48988d1c5941735)%20%20AND%20%40in%3CTags%3E%3A(englishFriendly)%20%20AND%20%40in%3CTags%3E%3A(kidFriendly)%20%0D%0ASpatialField%3A%20__spatial%20QueryShape%3A%20Circle(139.349905%2035.745064%20d%3D25.000000)%20Relation%3A%20Within&queryShape=Circle(139.349905%2035.745064%20d%3D25.000000)&spatialRelation=Within&spatialField=__spatial&distErrPrc=0.025&spatialUnits=Kilometers&pageSize=128

Here's the same url decoded:

https://diver.ravenhq.com/databases/Chaddeus-AMBvNext/indexes/PlacesByQuery?&query=@in<Categories_Id>:(4bf58dd8d48988d1c5941735) AND @in<Tags>:(englishFriendly) AND @in<Tags>:(kidFriendly) SpatialField: __spatial QueryShape: Circle(139.349905 35.745064 d=25.000000) Relation: Within&queryShape=Circle(139.349905 35.745064 d=25.000000)&spatialRelation=Within&spatialField=__spatial&distErrPrc=0.025&spatialUnits=Kilometers&pageSize=128

[Chad Thiele]

Wait, I think that was when I had the spatial query in the textarea. Here's the correct one (with spatial query in the UI)

https://diver.ravenhq.com/databases/Chaddeus-AMBvNext/indexes/PlacesByQuery?&query=%40in%3CCategories_Id%3E%3A(4bf58dd8d48988d1c5941735)%20%20AND%20%40in%3CTags%3E%3A(englishFriendly)%20%20AND%20%40in%3CTags%3E%3A(kidFriendly)&queryShape=Circle(139.349905%2035.745064%20d%3D25.000000)&spatialRelation=Within&spatialField=__spatial&distErrPrc=0.025&spatialUnits=Kilometers&pageSize=128

and decoded:

https://diver.ravenhq.com/databases/Chaddeus-AMBvNext/indexes/PlacesByQuery?&query=@in<Categories_Id>:(4bf58dd8d48988d1c5941735) AND @in<Tags>:(englishFriendly) AND @in<Tags>:(kidFriendly)&queryShape=Circle(139.349905 35.745064 d=25.000000)&spatialRelation=Within&spatialField=__spatial&distErrPrc=0.025&spatialUnits=Kilometers&pageSize=128

[Chad Thiele]

Possibly an issue with RavenDB on RavenHQ?

[Oren Eini (Ayende Rahien)]

That is unlikely.

Would it be possible for you to either send me credentials for your RavenHQ instance or to send me a db dump?

It _looks_ fine.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--

[Chad Thiele]

I can send you the credentials. How would you like to receive them?

[Chad Thiele]

Actually here's a dump of the DB. It's small... not much in there, dev only.

[Oren Eini (Ayende Rahien)]

Okay, so the sptial thing is a red herring, The problem is in your data.

See here:

There is one location that has tags and categories, but it doesn't matches.

This is document 4b5c0443f964a5204f2129e3.

And it has category: 4bf58dd8d48988d14e941735

But you are searching for: 4bf58dd8d48988d1c5941735

See:

Q) 4bf58dd8d48988d1c5941735

D) 4bf58dd8d48988d14e941735

I strongly recommend that you'll avoid using guids for identifiers, exactly because of those reasons.

Also, may I use this in a blog post?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

On Fri, Jul 25, 2014 at 10:04 AM, Chad Thiele <chad....@gmail.com> wrote:

Actually here's a dump of the DB. It's small... not much in there, dev only.

--

[Chad Thiele]

I'm so sorry... just before you asked for the database dump I had reset it. I've gone back and tagged 2 places appropriately, and attached a new dump that has the correct tags.

A search for category_id: 4bf58dd8d48988d1c5941735 and both tags: "kidFriendly", "smokeFree" should only return one result (but it returns two). I've attached a photo of that query too, underlining the tags on the places found (again, can't see the main body of the query because the spatial UI covers it).

About the guids, these are the category id's as pulled directly from the Foursquare API, otherwise I would use RavenDB generated ids.

Thank you so much for your time and effort on this issue. Of course, you may use any of this material on your blog.

[Oren Eini (Ayende Rahien)]

Hi,

Note that the query you actually used before was from englishFriendly, not smokeFree

I'm unable to reproduce your issue, see:

Note that just beneath the spatial there is a hidden gutter that you can use to expand / contract the query text area.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--

[Chad Thiele]

That's strange... here's a screenshot of my results.

Judging from that query, I should expect to only get back places that have both the englishFriendly and the smokeFree tag, correct?

[Oren Eini (Ayende Rahien)]

Can you try resetting the index?

is the index stale?

[Chad Thiele]

Just tried that, also tried deleting and recreating the index. 

Now I'm confused, you can run the query and get the result I'm looking for, yet when I run the same query I get extra places in the result. 

[Chad Thiele]

I went back, tagged up another place. It appears the query is simple including any place that has any one of the tags in the query, rather than only showing places that has every tag in the query.

Here is a screenshot of a query for 3 tags, with three results. What I'm looking for is to have this query return only the first result (the place with all three tags queried for).

[Oren Eini (Ayende Rahien)]

Can you go on skype?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

[Chad Thiele]

If you have time, sure. I have the code, and RavenStudio, up now.

[Oren Eini (Ayende Rahien)]

Note to the list, resolved on the call.

An issue with a resolved bug and using in for single item queries.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

On Sun, Jul 27, 2014 at 10:22 AM, Chad Thiele <chad....@gmail.com> wrote:

If you have time, sure. I have the code, and RavenStudio, up now.

--

[Chad Thiele]

Thank you so much Ayende. Works great, and your support, as always, was tremendous.