[slscripters] Urgent problem - all http failing with 499 since 2:50am PDT

[Sasun Steinbeck]

As of 2:50am PDT this morning, all HTTP requests to any of my domains from SL on my web host are failing with a 499. Every single one, not one is getting through. The IIS logs show all requests from scripts stopped about then. The web server is up, I can contact it just fine, from my browser, everything is working there. I can make other http requests to other domains like google.com just fine. It is only failing from SL scripted objects to any of my domains hosted on the same server.

This very much looks like some kind of blocking is going on.  Has anyone ever experienced this before, and how did you solve it? I'm about to start a chat session with LL support but i"m not holding my breath.

In the mean time I have a bunch of customers completely dead in the water since most of my products are web-integrated and are failing completely. Gaaaaahhhhhhhhh....

[Sasun Steinbeck]

This is absolutely insane. After 5 days of waiting for the courtesy of a reply of this critical issue for my business, here is what I got back:

"Unfortunately, the issue you mentioned is not specific to only yourself. This is a bug that we cannot help you with directly in support.

The place to make just reports or requests is the Issue Tracker."

Holy freaking crap, are you serious LL? Is anyone else having their http requests blocked, or has anyone heard of this? Apparently this is a known problem? I wish they would have supplied some details. I have no idea whether switching to another IP address will fix this problem or not...

It looks like this is not a problem they can fix any time soon, which makes NO sense to me, since obviously not ALL IP addresses for outgoing http requests are blocked. In the mean time I have somewhere around 7000 scripted objects all across the grid that will not work until they fix this. Which at this point sounds like it could be a long time. Which means, I'm out of business, and leaving SL as well. This is absolutely incredble, my mind just boggles.

[Kora Zenovka]

If LL awknoledge that this is not an isolated bug, which Jira bug are they referrring to?
499 - for me - happens only if my http server times out, which is quite uncommon.
But I never got a http request blocked (except once when there were problem in the whole grid) and I have an intense heavy traffic between SL and my regions.
I wonder if your IP address has been blocked for some reasons, but it is unlikely, is it? They have a cap on the number of http requests you can have for each region, but in that case, some of them would still get through.

Maybe it is best to submit a bug in the Jira. It is a good way to find out if other people are having the same issue.

_______________________________________________
Click here to unsubscribe or manage your list subscription:
https://lists.secondlife.com/cgi-bin/mailman/listinfo/secondlifescripters

[Roy Riggs]

[Sasun Steinbeck]

I just bought a fixed IP address and it is blocked by SL as well. It is in the same address block as my web host, but I've tried some other random IP addresses in the same block and they work fine. So... I may have just gotten unlucky with the IP address my hosting provider just picked for me, I don't know. But changing my IP didn't work, it is blocked as well. I understand getting an occasional 499 but according to my IIS logs not a single request from an LSL script has hit my server in the past 5 days despite them being sent at an aggregate rate of about 1/second or so from thousands of sims across the grid.

The new fixed IP address works fine from avination, as well as the old one as well (no problems there). This may become a product I'll have to move to other grids and it just won't be available on SL for reasons I'll be happy to make very public. Which screws all my existing customers royally :(

Yes it seems unlikely, it seems crazy, it seems insane. But it is really happening and if it can happen to me, it can happen to you too. I'd think long and hard before starting any new product development that makes http calls to an external site. I know for sure this is the end of any new such products that I create, unless it's for avination or another grid.

[Sasun Steinbeck]

And they didn't give me the courtesy of telling me which jira. Maybe it's not in the database, but they apparently know all about it. I really wish they would have given me some details since I'm running around trying random things (like the fixed IP address, and now looking for a new web host) and have NO idea whether that's going to work or not. Since there's no way to re-open a case, I can't just open another case asking for details and wait 5 more days for an initial reply :(

SL is simply not ready for serious development when it takes that long for an initial reply to a critical incident that breaks content for thousands of residents. And the reply is "file a jira". Sigh.

So far my testing with my new fixed IP address is having the same problem - it's blocked as well.

On Sat, Oct 22, 2011 at 2:13 PM, Kora Zenovka <kora.z...@gmail.com> wrote:

[Kora Zenovka]

Just a few things I wonder:
a) is it possible that they actually blocked your domain name, instead of ip address? (in that case switching ip address won't do much good).
b) if not sure, did they block any requests to the domain name, or just specific path? Can you do a llHttpRequest on a different page of the website and see if you get any http response with 200 status code?
c) do you feel confident that one of your clients didn't abuse of your product in a way to generate a huge amount of http traffic? I don't know how you throttle your http requests around the grid in a way to keep a max of 1/sec, but is there a way that it may accidentally exceed that?

From what you said, it seems that LL is aware of this, but they also advise you to create a new Jira. If you do that, let us know what is the number, and I will vote for it.

[Fred Beckhusen]

Is there a space in the URL? Or another unescaped character that has
somehow gotten in? That will yield an error 499.

You say you can get a response from Google. You should have gotten back
a 404, not a 200 OK.

Otherwise you might have changed something that could have led to a
space or other character (if there is one) being deleted, thereby
passing the test by omitting the character.

Ferd Frederix

[Nexii Malthus]

I'm not getting any issues here, but if your curious, I'm using rackspace cloud servers for my own systems successfully.

- Nexii Malthus

[Sasun Steinbeck]

One thing I should make clear, when I say "block" I mean both intentional and non-intentional. It's all the same to me, no communication. It could be a bug due to the OS upgrade for all I know. They said to file a bug report and Kelly said that they very rarely block IPs or domains for outgoing requests. So it sounds like a bug introduced with the OS changes, if I had to guess. Happened right during that whole process.

Neither my domain name nor my web host shared IP address, nor my brand new static IP address work. You can do an http request directly on the IP which skips the DNS lookup, obviously, and it fails. Furthermore I have a brand new domain I created last month that is not public, and that is blocked as well - it is the same shared IP address on the same web host server.

Requests to any path on any of the websites at any of my domains hosted there fail. No pages work at all, and furthermore, even if I typed in total garbage for a page name, at least I'd see an incoming request record and a 404 reply in my IIS log, which I don't, which means those requests are not hitting my website whatsoever.

c) my IIS logs don't show any change in http request traffic this month. If someone was abusing, and it's certainly possible it could happen (off the top of my head, drop in a script that resets the other scripts which causes a few http requests, and do that every few seconds and replicate that in 1000 objects) but I'd see a huge spike in LSL request traffic. My scripted objects are in use by a lot of people around the grid, all of them doing HTTP calls, and since they are throttled on a per person + per object basis, that means potentially I could get a lot of requests per second to the web app (the more the better! :), so throttling doesn't come into play here. I am not throttling the entire set of requests in any way, the 1/s was just a guess based on current load. I looked at a typical day and I am actually getting 1.4/s from LSL scripts alone. Not much at all from normal browser requests, it's a tiny portion of that. Those requests are coming from upwards of maybe 6500 -7000 objects scattered all over SL. Typically one of my kiosks makes 2 http requests to my domain in a row every 2 hours and receives one http request directly from an in-world object so it's not much at all per object but it adds up with thousands of them around.

Here's the JIRA, please vote for it! Trust me you don't want this happening to you! https://jira.secondlife.com/browse/SCR-226

[Sasun Steinbeck]

I definitey do not get a 404 from google.com since that would be the end of western civilization as we know it :) That works fine. Why would I get a 404 from a GET on google.com? It certainly exists :)

Check the JIRA for a simple test script that you can run that shows the problem. Then load that same URL into your browser and see that it works fine.

Nope, definitely no spaces in the test script. I didn't change anything at 2:50am on monday when the problem started so it wasn't something I screwed up.

[Sasun Steinbeck]

I'm waiting to hear back from reliablesite.net and softsys.com that I've narrowed my choice down to, I need to make sure they allow outgoing http requests on any port, since SL URLs use all kinds of crazy port numbers. My former web host (godaddy.com) did not, which sucked. I'll sign up for a month with one of them and upload my website and see if it works. I'm kinda surprised my new static IP didn't work today though, that's weird. Why would that not work... very weird. It can't be in some kind of IP filter somewhere, it's never been used (afaik) as an IP address for any kind of SL communication...

I've read great things about rackspace.com but it's a bit more than I really need, glad to hear it's still working for you.

Someone mentioned that one of the popular in-world vendor machines ran into a problem with SSL not being compatible with LL's servers or something along those lines... I wonder if it's something like that, though I'm not using SSL in my web app at all.

[Fred Beckhusen]

Your fixed IP as currently used isn't a solution, as the host firm is
using 302 redirect, which is not supported by LSL. A fixed IP is a must
though. In my experience, the Lindens servers will not move quickly to
a new IP address, because the server appears to cache the IP address for
longer than allowed. The SL servers take as long as forever to check
DNS again. Apparently they ignore the time to live field in DNS. It
took weeks, until a rolling restart, for some boxes of mine to check
back into my server after a failover server was put into place and
switched to by DNS. I finally moved the server physically to another
datacenter so it could have the same IP address.

I hope your move goes smoothly, I would love to know how quickly the
servers switch after the DNS change.

There appear to be multiple problems here, both with DNS and the blocked
IP. I commented the Jira on both.

Your host company appears to be using 302 redirect, which is not
supported by Linden Script Language. The IP address being reported now
by Nslookup is returning a 302 redirect. You have to manually code
around this. But this is not causing any problems with the timeout or
the DNS lookup.

Do a nslookup from a command prompt and type in the host name and you
get the following

C:\>nslookup
Default Server: google-public-dns-a.google.com
Address: 8.8.8.8

> sasun.info
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Non-authoritative answer:
Name: sasun.info
Address: 174.120.213.242

Then do a:

telnet 174.120.213.242 80 <ret>
GET / HTTP/1.0<ret><ret>

and you get

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=utf-8
Location: /Default.aspx?pid=Login&ReturnUrl=%2f
Server: Microsoft-IIS/7.0
X-AspNet-Version: 4.0.30319
Set-Cookie: WebsitePanel=; expires=Mon, 24-Oct-2011 01:35:40 GMT; path=/
X-Powered-By: ASP.NET
Date: Sun, 23 Oct 2011 01:35:40 GMT
Connection: close
Content-Length: 158

<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a
href="/Default.aspx?pid=Login&amp;ReturnUrl=%2f">here</a>
.</h2>
</body></html>

This will not work as it is a 302

I tried the minimal code you posted in the JIRA and I certainly believe
that you are being blocked by a bug.

Also, if you Vote for an issue, it won't count. The Lindens announced
some time ago they look at the Watches. So I voted and watched just to
be ornery.

It is time to vote with your feet and move to another host firm and hope
the DNS cache issue clears up for you quickly.

Ferd Frederix

[Sasun Steinbeck]

Hm interesting, I am getting different results from the telnet command. I'm not getting a 302, I'm getting a 200 OK followed by all the HTML from my default page. Looks good to me. Very weird that you are getting a 302. I'm not doing the redirect myself, though I am doing a 301 redirect on one old retired page, but not my default.aspx page or anywhere else.

Last time I changed domains (without setting my TTL lower) it took a day or two, possibly three, for all the sims to update. On some sims it updated in a day, others the next day, etc.

You are right I may have caused a red herring with the static IP change. I thought it would be safe since I did ask my web host to set my TTL to 15 min to avoid this problem. Either I didn't give that change time to propagate, or they just didn't do it. Or, the LL servers are ignoring it :)

So your DNS is be current since you are getting the new IP address. I assume nslookup is just querying your DNS server? Mine updated very quickly, so maybe the TTL setting did get to my DNS server (I use opendns, not my ISP's DNS server). I had to flush my pc's DNS cache with ipconfig -flushdns and after that a ping showed the correct new IP.

That redirect looks funny. The 302 is returning the same URL, which makes no sense. I tried http://www.internetofficer.com/seo-tool/redirect-check/ on sasun.info and it didn't report any redirects... assuming that tool works.

Wow didn't know about ignoring the votes. That should be removed from the page, ugh. Thank you, I had no idea.

[Sasun Steinbeck]

Followup on this, I moved the entire database and web app to another web host. No problems connecting there. So at least I'm back up and running and customers are connecting again. So there's some big problem that developed suddenly last week somewhere along the line between LL and my old web host right when LL was doing the OS upgrades.

We're going on 48 hours now after the DNS change and about half the sims (out of many thousands) have udpated their DNS. Looks like it might take another day. Sometimes rebooting a sim helps, sometimes it doesn't. I'm guessing there's a bunch of intermediate servers that needs a DNS update then maybe the individual sims cache DNS as well since rebooting one can suddenly make it work. When it doesn't it's probably because the intermediate servers aren't updated yet so rebooting won't update anything.