Difficulty getting the closest venues on venues/search when using 'limit' or 'categoryId'

[David D.]

Hi, I've been having trouble getting the venues/search endpoint to
truly give me the nearest venues by distance relative to a specific
categoryId.

Assume: venues/search?ll=40.727535,-73.991443

You should get something like:
-Foursquare HQ
-B Bar & Grill
-And 28 others

Now add: &limit=2

We happily still get:
-Foursquare HQ
-B Bar & Grill

Now, add: &categoryId=4bf58dd8d48988d121941735 (the Lounge category,
which is B Bar's first category)

Now we unhappily get:
-230 Fifth Rooftop Lounge (distance: 1872)
-B Bar & Grill (distance: 55)

This is not so good. To me, I should be getting venues with the lounge
category sorted by distance...so at a bare minimum B Bar should be at
the top of the list. Obviously, there's a popularity contest going on
here and the results are not returned solely by proximity. But, surely
there is another Lounge closer than 230 Fifth. We need a way to return
the closest and only the closest venues for a specific category.

Let's make this problem worse by choosing a more general category,
add: &categoryId=4d4b7105d754a06376d81259 (Nightlife)

Now we get:
-The Standard Biergarten
-13th Step
Neither of which are very close or at least certainly not the closest.
B Bar does not appear until we up the limit to 15. Obviously we
usually have a limit of 30 so I am exacerbating the issue here. But
the point is that I can't figure out a way to get category-specific
results returned only in order of distance. In my scenario, B Bar
should always be at the top.

If I am building on the Foursquare API and I want to show my users the
closest 10 bars (regardless of popularity of check-ins), how exactly
can we do that? Perhaps the functionality already exists, but changing
'intent' does not have the right effect and I cannot find another way
to get venue search results sorted purely by distance and category.

Any help or advice you might have to help me get just the closest
venues would be greatly appreciated.

Thanks and keep up the good work. The Foursquare API truly is one of
the best out there IMO.

[Alice Lee]

David,

It seems strange that this occurs when you specify the venue
categoryid, but things are fine when you're running a search with
nothing specified. I'll check with with venues engineering team to
better understand this and get back to you.

Alice

[David D.]

I did find a way to solve this problem but it was not clear from the
documentation.

On this page: https://developer.foursquare.com/docs/venues/search.html

Under 'categoryId', it says "It does take a radius specified in
meters, up to a limit of a couple thousand. If specifying a top-level
category, all sub-categories will also match the query."

So if you add to your querystring '&radius=500' then the results are
much better. However it was not clear from the documentation how to
use this radius (i.e., just pass it in as another querystring
parameter). I think 'radius' should be added to the table of available
parameters.

I still feel that there is a philosophical question here for this
endpoint. If I'm looking for a specific category, and a small number
of venues (<10), do you want to serve the closest or the most
popular / most likely to be checked into? Personally I think
closest...this is a Location-Based Service afterall not a Popularity-
Based Service. But the best bet is just to make it really clear to
developers how to do one or the other. For now, making 'radius' more
obvious would be a big help.

Thanks!

[Akshay Patil]

Hi David,

The "strictly closest" vs "most likely to check-in" question comes up here time to time.

So far, we've yet to see a use case (other than people trying to scrape our database) where "strictly closest" is what the developer actually wants. When developing, it might be appealing to have results sorted by distance because it seemingly increases the quality of the results, but the reality is that deployed apps often have a coarse or inaccurate fix on a user location and a strict distance sorting fails the user in a big way.

The ability to specify a search radius is a request we hear often and are very much planning to do soon. As you may have read, we just switched over our venue search database to a more sophisticated system. We're still sorting out some hiccups, but once it's settle down, we'll be able to release a radius-search option (plus some other stuff developers have been asking for)

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

--
To post to this group, send email to foursqu...@googlegroups.com

http://groups.google.com/group/foursquare-api

[David D.]

Hi Akshay,

Thanks for your feedback. Is it possible that I could follow up with
you or someone else offline about this, perhaps via phone to discuss
the needs of our application? (I'm @deremer on twitter)

Basically, today it seems like foursquare modified the venues/search
endpoint. Specifically, 'radius' no longer works and 'categoryId'
requires a 'query'. These changes broke our application, and they
significantly impact our ability to use foursquare as our venue
provider if we want to deliver the best experience to our users.

Since these changes, our search results now include MTA Buses, random
people's apartments, gyms, and venues that are completely out of place
in our application. We only want to show the user specific
categoryId's that are relevant to our use case. With respect to
'categoryId' now requiring a 'query', to me that renders the
'categoryId' parameter completely useless. If the user knows something
about the name of the venue, what's the point of restricting by
categoryId? However, this parameter without a query is essential for
creating apps that are focused on a specific type of place (e.g.,
bars, clothing, etc).

With respect to "strickly closest", we found 'radius' to be invaluable
because in our testing we found that we could not rely on the
completeness of foursquare results without it...places we know are
nearby are missing and places that are far away (but popular on
foursquare) were in the results (see my initial example in this
thread). We don't want to show users what is popular on foursquare, we
want to show them what is actually around them (although I understand
that popularity is a proxy for relevant). Our app is focused on venue
discovery rather than check-ins, so this sensitivity is heightened for
us. Moreover the radius parameter is the only way we found that could
give the user some control over the granularity of the search and
subsequently the results. Regardless, I'm glad to here that radius-
search is on its way and I hope it comes soon, especially since it was
there and now appears to be gone.

Anyway, the changes to this endpoint today have majorly impacted our
application. I'd really like to sort out some of these issues, so
please let me know if there's someone at foursquare that I can chat
with. Thanks very much for your time and help.

And please forgive me if this reads snappy, I really like using the
foursquare API, except I had to deal with a call from our dev team
asking what happened to the foursquare endpoint today :-)

Cheers,
David

[Akshay Patil]

Hi David,

I'm talking with the engineer on the venue search team to better understand what changed and what the best actions should be.

Re: 'radius' -- are you using /venues/search or /venues/explore? search doesn't support a radius parameter and never has, to the best of my knowledge.

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[Akshay Patil]

Oh, nevermind, Old categoriId field mentioned a supported "radius" field -- so it was documented and working, just not listed as an actual param. Sorry about the confusion on my part.

Still sorting things out with the engineering team, will get back to you soon, I hope

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[Fábio Borges]

Akshay Patil,
you are the MAN. 

2011/9/29 Akshay Patil <a...@foursquare.com>

--
Cordialmente,

Fábio José Borges Fonseca

Analista de Sistemas Júnior

http://estou.la

celular: +55(31) 9236-4006

Esta mensagem é destinada exclusivamente ao(s) destinatário(s) identificado(s) acima. Se você não é um desses destinatários, não está autorizado a utilizar nenhuma informação contida nesta mensagem, para nenhum fim.

[David D.]

Hi Akshay,

I was using the /venues/search endpoint, and I was passing
&categoryId=XXXX &radius=###

Please let me know what the best course of action is. Our application
absolutely needs to be able to get only specific categories of venues
because as I mentioned MTA buses, gyms, etc. are completely out of
place.

Thanks for your help, hopefully this is a quick fix because otherwise
we have to look at completely reworking the venue aspect of our
application.

Cheers,
David

[Akshay Patil]

Hi David (and everyone else with this problem)

As you might know, we've been transitioning our search infrastructure over from mongo to solr and this was one of the unintended artifacts of that transition. Sadly due to some time pressures there wasn't good communication between me and the engineer in charge which is how the behavior changed so dramatically without notice or versioning control.

We've identified a good plan of action to restore behavior that more closely fits with the old system and we should have the patch pushed soon (a day or two). Apologies for the breakage, but it should be fixed soon.

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[Akshay Patil]

In the meantime, if you route the requests via your server or run a webapp, you can always do the filtering on your client side by looking at the categoryId of the returned result and removing anything you're not interested. Certainly not something we would demand of you in the long run, but a viable short term patch if this is affecting your service

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[David D.]

Hi Akshay,

Thanks for researching this and working hard on helping us. It was labeled "Experimental" so we knew this could change, but it was sudden and we were getting used to the pattern we had established. I'm grateful for the team working to push out a patch! Where should I look to see when the patch is out? The documentation page for the endpoint? Thanks again for being so responsive!

While I've got your attention, another great feature would be to allow us to query for multiple categoryId's at the same time, perhaps as a comma-delimited list. Right now to do this, we have to hit the API once per category (which impacts our call limits), merge the results, remove duplicates, and re-sort. Unfortunately the best sort we can do is by distance, which (despite all my clamoring for "strictly nearest") hides the popularity-based ranking of foursquare. But I'm sure that's just another feature to add to the long list of feature requests.

At any rate, thanks again!

[Akshay Patil]

One of the things on our list =)

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

--

[David D.]

I figured as much. I'll keep an eye out for the patch. Thanks again, as Fabio mentioned you're the man.

[FaceySpacey.com]

yea, i noticed that the search by categoryId wasn't working either, and figured it was API changes. looking forward to the patch ;)

[Joël Cheuoua]

David, Akshay,

Thanks a lot for this thread, it helped me to understand the changes as well. Looking fwd to the patch eagerly. 

This is my first and only bump with the 4square API, and as David pointed out, it's true that this was experimental, so kudos to the 4square dev team for delivering!

[Akshay Patil]

FWIW, the patch didn't deploy yesterday, so we're now targeting Monday for the updated / restored behavior

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

--

[Akshay Patil]

The fix has been pushed and should emulate the past behavior. Due to a small oversight, the docs have not yet been updated, but they will be tomorrow.

When the docs are updated, you'll be happy to note that one of our biggest requests is also now pushed: you can now specify a radius for query searches

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[David D.]

And there was much rejoicing! Thanks Akshay! Seems to be working fine with no modification to our existing code. Thanks to you and the team for getting the patch through quickly.

[Joël Cheuoua]

Akshay,

It still returns mixed results for me.

For example, I've tried using the console with:

https://developer.foursquare.com/docs/explore.html#req=venues/search%3Fll%3D40.7,-74%26categoryId%3D4d4b7105d754a06374d81259

That category is supposed to be "Food" but it return among others:

{
id: "3fd66200f964a520daf11ee3"
name: "South Street Seaport"
contact: {
phone: "2127327678"
formattedPhone: "(212) 732-7678"
twitter: "TheSeaport"
}
location: {
address: "South St."
crossStreet: "Pier 17"
lat: 40.70566047104496
lng: -74.00287628173828
distance: 675
postalCode: "10038"
city: "New York"
state: "NY"
country: "USA"
}
categories: [
{
id: "4bf58dd8d48988d1e0941735"
name: "Harbor or Marina"
pluralName: "Harbors or Marinas"
shortName: "Harbor / Marina"
icon: {
prefix: "https://foursquare.com/img/categories/parks_outdoors/harbor_"
sizes: [
32
64
256
]
name: ".png"
}
primary: true
}
]
verified: true
stats: {
checkinsCount: 18211
usersCount: 12395
tipCount: 62
}
hereNow: {
count: 1

}
}

On Oct 3, 4:02 pm, Akshay Patil <a...@foursquare.com> wrote:
> The fix has been pushed and should emulate the past behavior. Due to a small
> oversight, the docs have not yet been updated, but they will be tomorrow.
>
> When the docs are updated, you'll be happy to note that one of our biggest
> requests is also now pushed: you can now specify a radius for query searches
>
> ~ak
> --
> Akshay Patil / Foursquare Platform Evangelist
> @akdotcom / @foursquareapi
>
>
>
>
>
>
>
> On Sat, Oct 1, 2011 at 12:04 PM, Akshay Patil <a...@foursquare.com> wrote:
> > FWIW, the patch didn't deploy yesterday, so we're now targeting Monday for
> > the updated / restored behavior
>
> > ~ak
> > --
> > Akshay Patil / Foursquare Platform Evangelist
> > @akdotcom / @foursquareapi
>

[Akshay Patil]

Hi Joel,

We're currently counting "secondary categories" when we do category restricts. In this case, South Street Seaport has a secondary category of "Food": https://foursquare.com/v/south-street-seaport/3fd66200f964a520daf11ee3

I went ahead and nixed the Food category -- it's a testament to our User and SuperUser communities that our data is usually so well curated, but it's not always perfect

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

[equinox]

Akhsay,

I sometimes is still getting mixed response, sometimes it responds with a 500 error. 
When will this be fixed 100%?

Aditya

[Akshay Patil]

Consistent with my previous response or something new? Please give concrete examples, otherwise it's hard to understand what the issue you're reporting is....

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

--

[equinox]

Akhsay,

Here's a sample request I made that sometimes returns a 500 response error:

/venues/search?ll=%f,%f&v=20110829&categoryId=4d4b7105d754a06376d81259&client_id=MYCLIENTID&client_secret=MYCLIENTSECRET

Some more with different category ID:

categoryId=4d4b7105d754a06378d81259

categoryId=4d4b7105d754a06372d81259

categoryId=4d4b7105d754a06379d81259

categoryId=4bf58dd8d48988d11d941735

categoryId=4d4b7104d754a06370d81259

[Akshay Patil]

Our new search infrastructure is experiencing some performance hiccups, which is leading to higher latency and occasional 500s. We're trying to calm things down, but you'll probably see elevated error rates in the mean time =/.

~ak
--
Akshay Patil / Foursquare Platform Evangelist
@akdotcom / @foursquareapi

--