A test of moving market data to public crest

[Regner Blok-Andersen]

Hey people, hope all goes well. Before I get into a change we are making I felt I should talk about why. Feel free to skip to the next paragraph if you just want to know the change. CREST uses a custom load balancing in our NGINX machines; it's a form of stick sessions. We do this so that all requests for a given character go to the same proxy machine as the proxy will hold the cluster wide session. This avoids us having 15 different sessions from you hitting 15 different proxies. When we released the real time market data we put it behind authed CREST because we had a few concerns and wanted to be able to easily turn applications off if bad things happened. Well... bad things didn't happen from anyone using the resources but on our end. With all the requests coming in we would max out the CPU on any given proxy. This usually happened on the proxy that EMDR landed on. :P The resukt of this is usually a gateway unavailable error and then it switches to a new proxy.

As a result of this we have had a review of how we do things. After making a few changes on our end to give us more control over banning people from the API we are adding the market data to public CREST. Along with that we are upping the rate limit on public CREST to 150 req/s. This should mean your requests round robin through the proxies. I was hoping we could arrange a time for you guys to hit public CREST while we monitor the proxies.

Let me know if you can help. :)

[Regner Blok-Andersen]

http://public-crest-sisi.testeveonline.com/ is Sisi.

[croakroach]

I wrote a quick CREST -> EMDR bridge last week and had it running for the last few days or so, feeding CREST data into the relay under EveData.Org. I am pulling around 25 request per second but could easily bump this up.

What is the current public request limit?

Hopefully once I get the site completely rewritten it will be pulling more than market data.

[Gregory Taylor]

There is no rate limit right now.

If you're willing to package this up and toss it on Github, we could look at it as the foundation for the eventual official feeder. In the future, we may end up disabling user uploads entirely in favor of the official feeder. I just haven't found the time to get this thing written myself.

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at http://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[croakroach]

Unfortunately I built it into the new engine for the site. I could pull bits of it out and throw it on github though.

On the rate limit, let me play around with the throttle and see how fast it will go before their servers bleed.

[croakroach]

That didn't take long.... uncapped it peaked at 1016 rps, dropping down to 350~430 rps followed by a storm of socket drops and hangs (likely hit max open file descriptors on the proxies... sorry Regner!!!).

It seems reliable around 90 requests a second. However now it looks like the upload is the bottleneck and i may have just overloaded the upload server.

Maybe it would be better to inject this directly into the zmq stream?

[Gregory Taylor]

Maybe it would be better to inject this directly into the zmq stream?

That's the current plan. I just haven't got off my tail and hammered out the scraper to feed it yet. Would definitely love to see what anyone has came up with as a foundation to build on. 

[croakroach]

I started looking at how i can separate this from the site and then recalled the reason i merged it into two to begin with.

The UUDIF is not directly compatible with CREST because CREST does not share solarSystemID which means that when the translation occurs you need a list of stations and systemIDs in a map to translate station to systemID. Right now since i update stations on the site from the API i do this with a database query, along with separate ones for a list of typeID's and regionIDs.

http://pastebin.com/Hw0KLS8f is what i have so far if it helps. Excuse the mess; still learning Go.

[Yann Ramin]

I've seen latency being the primary limit (CREST responses are small and paginated), as well as TLS handshaking (no session resumption support).

I've had good luck on maximum throughput from AWS eu-west-1. Once I scuttle the current scraper in the EVEC API and move it to entirely reactive support it will likely be hosted out of there.

--