While paging through /find/groups responds unpredictably

[Alexis Rondeau]

Hi there!

I'm having an issue with making /find/groups requests as follows (and don't know if I'm doing it wrong!)

The problem is: 

I'm issuing api-key "signed" server-sided GET requests to /find/groups with a page length of 100, location Berlin, Germany and increasing offset.

The offset/pages I can iterate through before receiving an empty response vary greatly every time I run the same query. 

Once in a while I can page through all 26 pages of 100 Meetups each for all Meetups in Berlin.

More often though, I only get to page/offset 4 or 9 or 10 or 14 or anything BUT the full 26 pages which means I can't really do the calculation I need to do (Which is compare growth of all Meetups in Berlin).

I always get 200 OK, but the empty response ([ ]) comes far earlier than expected.

A few more observations: 

This happens on my local machine as well as on my server, so I figure it can't be an AWS IP address issue.

I've experimented with different sleep-times between requests, same, unpredictable results.

I've attached my logs where you can see that once I've gotten all 26 pages back (2683 Meetups fetched) and from there it deteriorates.

Thanks so much for any pointers on what I'm missing here or doing wrong!

Alexis

PS: Code-wise, this is what I'm doing (pseudo-code)

// start at page/offset 0
offset = 0

while(continue) do

    // make my request
    response = get("https://api.meetup.com/find/groups?photo-host=public&location=Berlin&page=#{page_length}&country=DE&sign=true&key=MY-MEETUP-API-KEY&offset=#{offset}")

    // stop the loop if the response is empty
    continue = false if response.body.empty?

    //... process response content otherwise
    ...

    // move on to the next page by increasing the offset
    offset += 1
   
    // sleep for 5 seconds
    sleep(5)
end