recent assets.osgrid.org issues
Dave Coyle
regarding issues some (but not all) people have been having connecting
to assets.osgrid.org. Different people have had different
experiences, so this thread is an attempt to make sure we all have the
same information. The cause is still unknown.
Some of the ways this problem manifests itself:
* console hangs on region startup after:
[MAPTILE]: STORING MAPTILE IMAGE
* you see the following on the console:
[ASSET CONNECTOR]: Unable to send asset <UUID> to asset server.
Reason: The request timed out
* region start which used to take seconds now takes several minuntes
* you see the following on the console:
[ASYNC REQUEST]: Request GET <asset_url> failed with exception
<...> System.Net.WebException: The operation has timed out.
Some people with the above problems report packet loss when pinging
assets.osgrid.org, but others exeriencing the above observe zero
packet loss. daTwitch reported in IRC that after changing
"assets.osgrid.org" to its IP address, "66.240.232.122", the problem
on his maching went away. I've observed that I have similar timeout
issues accessing http://assets.osgrid.org/stats/ as I do retrieving
OpenSim assets. OSGrid admins reported on June 10th
[ http://twitter.com/osgrid ] that a faulty switch had been replaced,
but people are still experiencing problems.
This may be a single or multiple problems, and people may be
experiencing the same problem in different ways.
Are you having additional or different experiences with this issue?
Can anyone report similar success as daTwitch in using IP addresses
vs. FQDNs? It's not clear to me why that change would make a
difference, but so far it's the only reported success I'm aware of in
beating this so it would be worth trying out.
-coyled
FredoChaplin
to discover this group:
assets server (mantis #3782) and sim startup time
====================================
coyled closed during the week-end the mantis #3782 in OpenSim.
He's certainly right doing that, as far as it appears to be an
internal Osgrid issue.
But the problem is nonetheless not resolved closing the mantis, and
the discussion must anyway continue elsewhere.
We still have connexion timeouts on the asset server and, related or
not, my sim needs currently (linux, mono 2.5, 4 regions) more than 15
min to startup.
Translating DNS into fixed IP addresses in .ini files doesn't help
that much.
What can be done, how can we help to finally solve this (long lasting)
issue ?
It really begin to be kinda boring.
2009-06-15 12:07:12,477 INFO -
OpenSim.Region.CoreModules.World.WorldMap.TexturedMapTileRenderer
[MAPTILE]: Generating Maptile Step 1: Terrain
2009-06-15 12:08:52,694 INFO -
OpenSim.Region.CoreModules.World.WorldMap.TexturedMapTileRenderer
[MAPTILE]: Generating Maptile Step 1: Done in 100217 ms
2009-06-15 12:08:52,734 DEBUG -
OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [MAPTILE]:
STORING MAPTILE IMAGE
2009-06-15 12:12:24,306 INFO -
OpenSim.Framework.Servers.BaseOpenSimServer [STARTUP]: Startup took
15m 28s
2009-06-15 12:31:48,998 ERROR -
OpenSim.Framework.Servers.HttpServer.AsynchronousRestObjectRequester
[ASYNC REQUEST]: Request GET http://66.240.232.122:8003/assets/f3538
... cb5c360de2 failed with exception System.Net.WebException: Error:
ConnectFailure
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)
[0x00000]
at OpenSim.Framework.Servers.HttpServer.AsynchronousRestObjectRequester
+<MakeRequest>c__AnonStorey0`2
[System.Int32,OpenSim.Framework.AssetBase].<>m__1 (IAsyncResult res2)
[0x00000]
Hinamori
Dave Coyle
> What can be done, how can we help to finally solve this (long lasting)
> issue ?
As a temporary solution I set up a non-caching reverse proxy (pound)
pointing to assets.osgrid.org on a box which isn't having these
retrieval problems, and configured OpenSim to use that proxy. My
problems seem to have gone away. Here's my pound config:
User "www-data"
Group "www-data"
LogFacility local0
LogLevel 1
Alive 30
Control "/var/run/pound/poundctl.socket"
ListenHTTP
Address prox0.knifejaw.com
Port 10003
xHTTP 0
Service
BackEnd
Address assets.osgrid.org
Port 8003
End
End
End
You're welcome to try using my proxy, too, to see if fixes your
problems, with a few caveats. 1) No guarantee it works. 2) If you use
my proxy you're basically trusting me to not do bad things to your data.
I'm not; I don't have any desire or interest (or time!) to do so, and I
opened the proxy up solely because there may be people who don't have
the resources to set up their own. But by using it you still have to
decide if you trust me with your data. 3) As soon as these problems go
away I'll be shutting the proxy down; I'll send email notification to
this mailing list before I do so.
All that said, if you still want to try it out, replace
"http://assets.osgrid.org:8003" with "http://prox0.knifejaw.com:10003"
We obviously need to have the underlying issue fixed, but until that
happens this is one way around the problem.
-coyled
FredoChaplin
Thanks alot for providing that workaround proxy.
It was a nice and useful idea.
It seems that, at least from the location of my server, results are
really improving hour by hour.
I currently see no more connexion failures, and execptions in my logs.
And the sim appears to be running smoothly.
Just waiting one or two day to shout "hourayyy", twisting the fingers.
Fredo
FredoChaplin
Crossing fingers was not enough, i continue to have a varying number
of assets server misconnexion.
Therefore i switched to your proxy server, and it effectively solved
99% of my troubles. Many thanks for that, it's very useful.
But i'm afraid i'm not the only one to abuse from your kindness, and i
wonder how long the bandwith will resist...
And, last but not least, the question keeps staying opened: "and now ?
What else ?"
As far as i'm not Georges Clooney, and no stakeholder of Nespresso, i
really wonder what is currently done by OSGrid team technically
speaking to go through this, and how we can imagine to normalize that
situation. And i wonder also how many peoples do really get impacted
by this network problem.
Fredo
Dave Coyle
> xHTTP 0
Sorry, that should be "xHTTP 1".
-coyled
Dave Coyle
> Therefore i switched to your proxy server, and it effectively solved
> 99% of my troubles. Many thanks for that, it's very useful.
Glad it helped.
> But i'm afraid i'm not the only one to abuse from your kindness, and i
> wonder how long the bandwith will resist...
I'm not worried about bandwidth. :)
-coyled
Dave Coyle
> Hello. There's been a lot of chat in IRC over the past few days
> regarding issues some (but not all) people have been having connecting
> to assets.osgrid.org.
Nebadon announced that last night around 21:00 EDT (UTC-4) the IP
address for assets.osgrid.org was changed (to 71.6.158.150). The
problems retrieving assets I had seen before seem to have gone away.
I'd encourage everyone who was having trouble to give
http://assets.osgrid.org:8003 another go.
-coyled