ANN: It looks like we're done here

[Gregory Taylor]

Hello all,

It looks like our upload traffic has fallen off very suddenly, so I'm going to assume that the long-anticipated EVEMon upgrade that strips the uploading may have happened. As a consequence, we're pretty much dead in the water now. This is not EVEMon's fault, as they gave us plenty of notice, I just haven't found the time to write a CREST -> EMDR scraper that would replace our current upload gateway.

I'll put out a final last-ditch call to see if anyone is interested in working on such a thing and reviving EMDR. There's still a place for EMDR for those who don't want to run a bunch of heavy polling processes, but I'm probably not going to be motivated enough to tackle this.

In any case, a huge thanks goes out to everyone who made this possible. I like to think that the combination of EMDR and EVEMon helped liberate market data for the greater community, and contributed to the eventual exposing of in-game market data in CREST. We had humble beginnings, but featured quite a following at our peak!

I'll give this thread a week or so. If nobody ends up being interested, I'll take down the announce VMs and put up "Out of Business" signs on the Github repo and documentation. 

Thanks again, all!

[zweiz...@element-43.com]

Hey Greg,

thank yo for your efforts! Short answer: I won't let this happen :) I basically got the bridge lying around here, I just need to tear out the oAuth stuff as market data is on public CREST now. More tomorrow.

Sebastian

[James Muscat]

Sebastian, is yours the "Element43 CREST EMDR Bridge v0.4" that's just started running?

If so then you're outputting a value of 65535 for range="region"; the EVE cache and client always used to return 32767 (e.g. max signed short rather than unsigned). I notice because it doesn't fit into my database schema ;-) It's not against the spec, just different.

I had a go late last night at pulling together a Python app to do the same (identifying itself as CRESTMarketTrawler): https://github.com/jamesremuscat/CRESTMarketTrawler

For something written between 00:00 and 02:00, it seems to work reasonably well...

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at https://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

[zweiz...@element-43.com]

Yup, that was me ;) Changed it back to 32767 for now. At the moment I'm only scraping The Forge at reduced speed to verify general stability. The gateway does not seem to like high volumes from a single IP and rejects connections, also CREST's rate limiting features are behaving 'interesting' to say the least...

IMHO it could not hurt to have multiple scrapers out there, so we eliminate single points of failures. My scraper grew a little complex, as I prioritize certain regions (e.g. The Forge) while others receive less frequent updates. Markets without orders are updated at a slower pace to reduce the overall amount of requests. I don't know if the architecture I went with was that smart in the end, because I'm literally having a dedicated timer for each region/type combination - once that timer expires, a market update process is launched to fetch latest data and reschedule itself. Problem is: CREST's performance seems to fluctuate quite a bit and at times response times seem to be so slow, that requests just keep piling up until the buffers are filled.

Probably a "stupid" best-effort queue-based scraper would have been better.

[Otaci]

I havent used this system in years (or played Eve), but it really was a fantastic system.

I see that others may be keeping it alive with a bridge, which is great news. I still think this technology is great for distributing this data; far superior to everyone scraping CREST.

Thank you Greg, for designing this and supporting it all these years.

(But now I'm finally going to unsubscribe from this group :-))

[Jimi]

We have not yet released the version that removes the EMUU. Probably CCP done something with the EVE cache. Never the less, we are planning to release EVEMon v3 hopefully in a month (I need to do some ironing to the not few changed I have made to the app), which will see the end of EMUU. But I assume that some people will not upgrade to v3 in favor of the EMUU.

[evemark...@gmail.com]

I've recently been in touch with a few of you in regards to the drop in upload traffic and then the nice work of James with his CRESTMarketTrawler, but as you can see by my email I'm the new owner of Eve-MarketData.com. In all honesty, I'm not much of a coder and took over the site because I make use of it a lot and didn't want to see it get shutdown after the previous guy found other interests. So for what it's worth, I'm happy to lend any support that's needed to keep EMDR alive and kicking, be it infrastructure support or running a copy of the trawler myself on specific regions to maintain more current market data.

Mike

[zweiz...@element-43.com]

I and my logfiles would be very grateful if you could provide proper solar system IDs with your market data as specified in the UUDIF. I know it is not included in CREST but if you take the SDE and XML API, you can easily map the values.

Am Mittwoch, 9. März 2016 16:21:59 UTC+1 schrieb James Muscat:

[Jimi]

Just found out why EVEMon stopped uploading data to EMDR. Tranquility IP changed from 87.237.38.200 to 87.237.34.200 when they changed Data Center (you know Tech III devblog). The IP is hardcoded in EVECacheParser which EVEMon uses so there you have it.

I'm not sure if there is any point in updating EVECacheParser to include the new IP and then release a new EVEMon version.

Thoughts?

[Jimi]

BTW SiSi IP also changed from 87.237.38.50 to 87.237.33.2.

[Gregory Taylor]

Since there is now active work to move away from user uploads entirely, I'd probably assign minimal value to updating it.

We'll be announcing more on this in the future as progress happens. It's going to move at the speed of hobby time projects, but have comfort knowing that this is not an effort that I'm heading up (because I've been crazy slow). The person leading this will post on here when he or she is ready to share.

--

You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at https://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com