[slscripters] Problem with SL http_response errors

[AnnM...@slfbi.com]

I'm plagued with an http_response error 500 problem, "internal server error", with my vehicles that seems to be getting worse.  The problem first showed about 6 months ago and is now getting critical.
Error frequency seems to be independent of how many vehicles I'm running. (Not database server congestion).
It seems to get worse in the evenings and hardly occurs off peak hours .  (Internet traffic related??)
It  runs in batches lasting from 10 minutes to an hour, then goes away for a while.
A batch of errors will come in from random vehicles distributed all over SL and all start getting failures at the same time.
The http cap limit should not be a problem.  Only 1 or fewer requests per sim per minute.
On first or second  http time-out I wait a minute and re-submit.
On a third failure I shut the vehicle down.

It is so bad tonight I'm down to only 25 vehicles running, each making an http access once every minute or two and I'm still getting many re-tries yet earlier I was running over 125 with no problems.

**HOWEVER**  tonight I wrote a test program running outside SL accessing the same database once every 10 seconds and it is not getting any errors after running a couple of hours.
Previously I've assumed the error 500 "internal server error" was originating from my data base but with tonight's test I'm wondering if the Second Life http handlers could be introducing the problem.

Question.
Can the error 500 be originating in SL or must it be coming from my database server?
If they are being generated in the SL http interface, any suggestions?
Has anyone else been seeing this problem?

AnnMarie Otoole

[AnnM...@slfbi.com]

Good suggestion, Ed.

I don't have a parser for the logs but looking through the raw data I do
see the 500 results going out to the Second Life requests. I don't know
why my external tester is not getting them but it is obviously an
indication of where to start looking.

My knowledge of the MYSQL internal operations is minimal.
Could randomly occurring, rapid (simultaneous) requests be causing file
locking or is there only one thread at a time using the data access?
If there is file locking, would this cause 500 error responses to
blocked requests or do they just wait?
Would duplicate data files and splitting requests between them help?
(Most of the data is static).

On 10/6/2011 3:05 AM, Edmund Edgar wrote:

> On 6 October 2011 15:26, AnnM...@SLFBI.com<AnnM...@slfbi.com> wrote:
>> **HOWEVER** tonight I wrote a test program running outside SL accessing the
>> same database once every 10 seconds and it is not getting any errors after
>> running a couple of hours.
>> Previously I've assumed the error 500 "internal server error" was
>> originating from my data base but with tonight's test I'm wondering if the
>> Second Life http handlers could be introducing the problem.

> Have you checked your server logs to see if your server is sending
> HTTP 500 responses?
>
_______________________________________________
Click here to unsubscribe or manage your list subscription:
https://lists.secondlife.com/cgi-bin/mailman/listinfo/secondlifescripters

[Ima Mechanique]

> Good suggestion, Ed.
>
> I don't have a parser for the logs but looking through the raw data I do
> see the 500 results going out to the Second Life requests. I don't know
> why my external tester is not getting them but it is obviously an
> indication of where to start looking.
>
> My knowledge of the MYSQL internal operations is minimal.
> Could randomly occurring, rapid (simultaneous) requests be causing file
> locking or is there only one thread at a time using the data access?
> If there is file locking, would this cause 500 error responses to
> blocked requests or do they just wait?
> Would duplicate data files and splitting requests between them help?
> (Most of the data is static).

Presumably you connect to a scripting language (like PHP) to access the
MySQL db. Check its error logs to find more specific errors, like db
connection timeouts, that might be behind the 500 status code.

> On 10/6/2011 3:05 AM, Edmund Edgar wrote:
> > On 6 October 2011 15:26, AnnM...@SLFBI.com<AnnM...@slfbi.com> wrote:
> >> **HOWEVER** tonight I wrote a test program running outside SL accessing the
> >> same database once every 10 seconds and it is not getting any errors after
> >> running a couple of hours.
> >> Previously I've assumed the error 500 "internal server error" was
> >> originating from my data base but with tonight's test I'm wondering if the
> >> Second Life http handlers could be introducing the problem.
> > Have you checked your server logs to see if your server is sending
> > HTTP 500 responses?
> >
> _______________________________________________
> Click here to unsubscribe or manage your list subscription:
> https://lists.secondlife.com/cgi-bin/mailman/listinfo/secondlifescripters

Ima Mechanique
Check List
1) Check the documentation.
2) Drink coffee.
3) Check the documentation again, to see what I missed first time.
4) If I still can't solve it, ask if anyone else has the answer.

ima.mechanique(at)blueyonder.co.uk

[AnnM...@slfbi.com]

Thanks for your suggestions, I think I've solved the problem, if anyone
is interested.

I suspected it might be due to file locking when incoming queries
synchronized randomly and suddenly bombarded the data base with 100
queries in less than a second. It doesn't explain why that should occur
more from 6PM to Midnight unless the MYSQL engine is just running slower
under peak hour load.

95% of the queries access one data file that is "Read Frequently, write
rarely". The file is about 750K and about 2,000 records.

So I made two mirrors of the file which are accessed at random and kept
the original as a master copy. This reduced file locking time 50%.
Write functions are just applied to the master and periodically, or on
demand, the master is copied to the mirrors.

A simple solution that took 5 minutes to set up and so far it has been
running without a single http failure. It would be very easy to expand
to more mirrors later if needed but so far 2 data sources seems to be
working.

Ann-Marie Otoole.

[AnnM...@slfbi.com]

Oh that would be great but its a hosted service.  I've not checked but I don't think that would be available.  I'm lucky to be doing what I am on a service that is only $6.50 a month.  When running my vehicles at full capacity in SL my website approaches 500,000 queries a day.

On 10/7/2011 3:31 AM, Nexii Malthus wrote:

Can't you move away from HDD and onto memory?

Read up on the MySQL 'MEMORY' engine for a temporary table that holds things more frequently accessed.

Then copy to HDD infrequently.

- Nexii Malthus 

[Nexii Malthus]

Huh? If you have MySQL it should most deifnitely be available, you can set any table to a specific engine. I think some common frameworks even use it, I swear I had seen it on one table of phpBB or wordpress or wikipedia.

Yeah, wikipedia has got MEMORY engine set on the hitcounter table.

- Nexii Malthus

[Eloise Pasteur]

It's certainly worth a try - I'm on a cheap hosting site and can do it. 

I don't use it much - have to be careful about the size of the table too, and I no longer do much that hammers my server particularly hard, but it is doable.

If you can't, depending on how rarely you are writing to your table, you could consider a static table as .csv, xml or similar and read from that into the PHP. That normally works more smoothly because you don't fight through the MySQL engine to get there.

El.

http://educationaldesigns.eloisepasteur.net/

http://eloisepasteur.net/blog/