"No more than 10 simultaneous connections allowed" error for campaignEcommAddOrder

[jpettett]

About an hour and a half ago, I started getting the error "(-50) No more than 10 simultaneous connections allowed." when calling campaignEcommAddOrder.  However,

- I am testing on a dedicated development computer, and I am the only one generating connections.  No event-generated connections are being made, and I am only making one at a time.
- netstat shows only one open connection to 184.173.175.194 (us1.api.mailchimp.com)
- Your API call log shows that calls have been several minutes apart and are not concurrent.
- Calling other methods, such as listSubscribe and listMemberInfo, with the same code still works.

Why is the connection limit error returned?

[jesse]

You were probably making other calls that were so broken they caused us to 500 and not be able to decrement counters (or log anything, etc), in which case that issue should be reported. We'd also need to know the datacenter you're hitting or something to at least vaguely identify your account.


jesse

[jpettett]

Sorry, our account is "ledet.com".  We're only using us1.api.mailchimp.com.  I'm not getting the error today, but it would be nice to what went wrong, since I wasn't getting the error yesterday morning either.  I'll let you know if it breaks again.

[jpettett]

I'm getting the error again.  I didn't count exactly, but I was able to make about 10 successful calls before I started receiving this error.  Again, netstat shows no other connections open, and other methods, such as listMemberInfo, still work without error.  Why does the server think other connections are still open when campaignEcommAddOrder is called?

[jesse]

My best guess is that there's a bug in the Ruby XML-RPC client you're using (don't use XML or XML-RPC if possible) which isn't closing connections. Or...

While you're here, a couple other things I notice from your call history:
* why are you calling listInterestGroupAdd around 140 times a day? (they are all failing). These also appear to be paired with several listInterestGroupings calls.
* why are you calling listMemberInfo directly after listSubscribe? (that makes no sense)
* why are you exporting lists so often?

You have bunches of calls coming from several different IPs and it kind of generally looks/sounds like you're not quite aware of what's happening, so you may want to start by taking inventory of the code hitting us.


jesse

[jpettett]

What's wrong with the XML-RPC?  If it shouldn't be used, why is it listed as the first option in the API docs?  And how could a bug in the client app cause the connections to stay open on the server end, if they are closed on my end?  (We are talking about TCP connections, right?)  Also, I get the same results with RESTClient for Firefox, so can you recommend a client that does not have the bug that you suspect?

I'll admit that my code might need refactoring for efficiency purposes, but in order to effectively test and debug it, I'll need to know what's going on in the server.  Also, is there a way to export the API call log?  It would be a lot more useful in a spreadsheet format.

[jesse]

It's bloated, slower, much harder to debug, and has silly character set issues. It's also not listed first, it's listed last:
http://apidocs.mailchimp.com/gettingstarted/connectivity.php

Anything happening on the server is generating some sort of response - the details of all of those is what needs to be focused on, not a vague conversation where an error is finally noticed. Nothing in those logs is going to be helpful. And, as I said, you've got calls coming from all over the place, so taking inventory of what's really happening would also be a good start.


jesse

[jpettett]

I meant that XML-RPC is listed first in the actual list of API endpoints <http://apidocs.mailchimp.com/api/1.3/>.  But it really doesn't matter for this specific issue; it turns out that the problem is with version 1.2 of the API, regardless of whether XML-RPC or JSON is used.  And XML-RPC with v1.3 doesn't have this problem.

Unfortunately, the cost of porting all of our existing stable code from v1.2 to v1.3 may be prohibitive for this ecommerce-tracking project.  And since we built our existing implementation on a third-party library, mixing API versions doesn't sound like a workable solution either.

So, I know v1.2 is deprecated now, but if there's any chance of fixing this bug, it'd be appreciated.  Otherwise, I think you should put a note in the documentation warning other users of this issue who might be trying to make what should have been a small modification to existing, stable code.

Regarding the number of IP addresses we use, we have two servers (production and testing) in separate locations, so those each use an IP address.  We have a development system, which would normally only use one address, but since running into this issue, I've alternated between two addresses in a vain attempt to continue development.  Also, we use some third-party hosted integrations, such as Twitter, and of course, we can't control what IP addresses those use.  Is this an unusual number of IP addresses for this setup?  If so, what do you recommend that we change?

[jesse]

There's no bug that I'm aware of in 1.2.

See what I've said previously about the only possible issue on our side that could be causing this and then provide supporting details to allow us to track it down if it is legitimately not your systems causing connection issues.


jesse

[jpettett]

I'm not sure which "only possible issue" you're referring to (XML-RPC?), and you didn't answer my question about IP addresses, but here is a test case demonstrating the bug.  When I last tried this test case from an IP address that hadn't accessed the API for at least hours, if not days, I was able to call it successfully only 5 times before receiving the "No more than 10 simultaneous connections allowed" error.  Of course, replace <eid>, <cid>, and <key> with relevant values, and increment the item "id" value on each call.  I've been using the campaign named "Yet Another TEST (copy 01)" from July 9th for testing, if you need to get those values from there. 

URL: https://us1.api.mailchimp.com/1.2/?output=json&method=campaignEcommAddOrder

POSTDATA:

{"order":{"email_id":"<eid>","store_id":1,"campaign_id":"<cid>","store_name":"Our Store","total":42,"items":[{"qty":1,"category_id":1,"product_name":"Unicorns","category_name":"Animals","cost":42,"product_id":1}],"id":"test1"},"apikey":"<key>"}

[jpettett]

I also just noticed that while I received a "true" response for the first 5 calls, and the API call log indicates 5 successful calls to campaignEcommAddOrder, the ecommerce report for the campaign only shows 4 transactions from that set of attempts.

[jesse]

I was referring to the possibility that we 500'd on a request which could cause us to miscount connections. Details of that call would be what I'd need to see. If the request succeeds (and you can see it in the logs), that won't happen. If you're seeing calls "succeed" without corresponding log entries or data being correctly updated those are likely the failing calls and also likely indicates the code interpreting responses is busted.


jesse

[jpettett]

Wait a minute, are you suggesting that your server is returning an HTTP 500 error that I am missing?

Look, I ran another set of test calls (based on the test case from my post on Friday), from a fresh IP address, all using the same TCP connection, and the first 5 show as successful in the API log and the raw HTTP response for all (except for expected variations in the timestamp) was:

HTTP/1.1 200 OK
Server: nginx/1.2.0
Date: Mon, 30 Jul 2012 15:51:02 GMT
Content-Type: application/json
Content-Length: 4
Connection: keep-alive
X-Powered-By: PHP/5.3.3
Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/

true

But of these 5, the ecommerce report for the campaign I am using for testing (described in my post on Friday), only shows the first 4 transactions from this set of 5.  Then the next call, on the same single TCP connection as the previous five, got this response from your server:

HTTP/1.1 200 OK
Server: nginx/1.2.0
Date: Mon, 30 Jul 2012 15:51:06 GMT
Content-Type: application/json
Content-Length: 72
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.3
Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/
X-MailChimp-API-Error-Code: -50

{"error":"No more than 10 simultaneous connections allowed.","code":-50}

So, that's 6 calls on 1 connection from an IP address that hasn't accessed your server in at least 24 hours, if not several days.  There isn't 10 of anything here.  Also, since working with the specific test case from Friday, this pattern has consistently emerged: The first 5 calls receive a "true" response and show as successful in the API log, but only the first 4 are reflected in the ecommerce data for the campaign, then the next call fails with the "no more than 10 simultaneous connections allowed" error.

I have described a specific test case and detailed the exact responses and problems I am seeing.  If you are still convinced that I am doing something wrong, please provide me with a test case that doesn't exhibit these symptoms.

[jesse]

Yes, that's the only way that would happen. A test case is wholly unnecessary - I can look at the other 15 million or so calls our platform is handling correctly daily to know we're not constantly flubbing connection tracking. Alternately, I can also run any of the example code in a serial loop w/o issue or run it in tight parallel loops to force the issue.


jesse

[jpettett]

But it's not, as my test case proves.  If your server is experiencing an HTTP 500 error internally, though I am getting a HTTP 200 response (with a payload of "true" no less!), what does that mean?  Do you have some sort of proxy that is completely reinterpreting the response before sending to the client?  That would still be a bug in your system.

Also, I never claimed that your platform is constantly flubbing connection tracking; all I know is that it's failing to work as expected for my particular user story and that the response from your server is consistently different from what you claim it has to be.  The purpose of my test case is to demonstrate that.

Could you refer me to the specific example code which you've tested?

[jesse]

If we 500, we'll return a 500.

I happened to use the PHP example for ping() listed in the docs, but the method doesn't matter. The explanation provided only shows that either there are 500 errors occurring you aren't catching/reporting/seeing/etc that I also haven't found or there is stuff going on on your end causing that to legitimately happen for successful calls.


jesse

[jpettett]

You've been testing ping()?  This issue is specific to campaignEcommAddOrder() (and possibly other untested methods) on v1.2 of the API.  I thought we had already established that.  (See the 4th bullet point of my original post, and my post on Jul 25.)  The method does matter, because this issue is method-specific!  In fact, after I begin receiving the simultaneous connection limit error for campaignEcommAddOrder(), I can continue to make error-free calls to ping(), lists(), listMemberInfo(), etc.  No matter how I intersperse them, once campaignEcommAddOrder() goes haywire, it consistently returns the connection limit error while calls to other methods don't experience such problems.  The point is that the error message "No more than 10 simultaneous connections allowed" is itself erroneous in this case, because whatever the problem is, it is not caused by the number of simultaneous TCP connections from the client.

If I make six consecutive calls to the API from a "clean" IP address (i.e. one that has not recently connected to your servers), and all six appear in the API log, and I get six HTTP 200 responses (which I have clearly documented), and you "also haven't found" any evidence of 500 errors, are you willing to consider the possibility that some other type of bug exists?

FYI, I'm going out-of-town and may not be able to respond for about a week.

[jpettett]

I'm back.  Any progress on this?