ANN: Future of EMDR

[zweiz...@element-43.com]

We'll be announcing more on this in the future as progress happens. It's going to move at the speed of hobby time projects, but have comfort knowing that this is not an effort that I'm heading up (because I've been crazy slow). The person leading this will post on here when he or she is ready to share.

Hey there,

sorry for being silent for such a long time and thank you for providing market data with your custom scrapers! While the rumors of EMDR's death are greatly exaggerated, it is true that development moved at glacial speeds for quite some time and I am sorry for that. The core infrastructure kept running stable as ever, however as we lost the steady influx of crowd-sourced data (EVEMon), the newer CREST-based community-run uploaders proved to be less stable than anticipated, leading to the false impression of EMDR being down completely. Currently we have the standard core infrastructure being supplied with data by various independent CREST scrapers run by the community.

I would like to propose the following steps to update the project to the state of the art of EVE market data (CREST):

• If you are currently feeding market data to EMDR, please document your scraper's behaviour here, we will add it to the official documentation (bulk feeding every x minutes, only diffs, historic data - see below...) and please check that your scraper uploads correctly formatted UUDIF messages 
• In other posts it was mentioned that we're lacking a proper monitoring solution for individual scraper's uploads - does anybody have such a solution ready or do we want to build a generic one for the entire EMDR (think public Kibana dashboard with one series per uploader/relay), do we even need one?
• It would also probably be a good idea to make some changes to our upload-gateway infrastructure, IMHO we could just ditch the entire HTTP POST endpoint altogether and just give you a more direct path via ZeroMQ
• Which leads to the next point: we no longer have the problem of having to rely on local cache scrapers on thousands of machines - now we only have a few, trusted different uploaders, so we could move to an IP-whitelist-based approach for the upload endpoint, thus reducing the risk of distributing fake data

If you have comments or proposals of your own, please leave a comment, I'd love to hear your thoughts on this :)

Personally I'm running the 

CREST EMDR Bridge (I know, very creative)

What it does: it checks each region every 5 minutes and only uploads those type's markets which have changed (e.g. Tritanium in Jita) in messages grouped by region (so rather large messages for busy regions with lots of changes in the last 5 minutes)

What it does not do: it does not upload historic data, only orders

GitHub: https://github.com/zweizeichen/market_scraper

Caveats: Due to the nature of CREST having paginated output, there can be the case in which CREST's cache gets updated while I'm walking the market's pages, thus resulting in potentially inconsistent data across the 'edges' of pages. So far I don't have a solution for this (except for nagging CCP to provide an entire region's market in a single page).

Have fun and fly safe,

Sebastian / zweizeichen

[Gregory Taylor]

Thanks for stepping up for this, Sebastian!

I'll be around to contribute here and there, but Sebastian is the Man In Charge now. I've spread myself a bit too thin, and EMDR hasn't been getting the attention it needs.

The project is in great hands going forward! 

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at https://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[James Muscat]

Many thanks for taking this on, Sebastian!

My effort:

*CRESTMarketTrawler*

What it does: Constantly cycles through each region in turn in a random but fixed order, and uploads the entire market to EMDR in chunks of 30,000 orders (to avoid breaking the upload size ceiling). (Also dumps the data into a local database, which is the limiting factor in its speed.)

What it does not: Historic data, nor lookups to populate the solarSystemID field (which is left null).

GitHub: https://github.com/jamesremuscat/CRESTMarketTrawler

Caveats: Possible inconsistency between CREST pages; doesn't (yet) chunk per type so the prior invariant of a single message containing all orders for a (typeID, regionID) pair is invalidated.

I want to expand a little on that last: with the cache-scraping approach, then you were guaranteed to have all market orders for a given type in a given region in a single message (assuming you weren't doing something crazy with your uploads). I suspect that a lot of consumers will do something like:

onOrdersReceived(typeID, regionID):

    database.exec("DELETE FROM orders WHERE typeID=%s AND regionID=%s", typeID, regionID)

    ...etc

Without that invariant holding, I don't see how one can know to expire older (fulfilled, not expired) orders from the database. I'm planning to change my trawler to maintain this invariant in the future.

[zweiz...@element-43.com]

Thank you! By the way: you can also reach us on the tweetfleet Slack channel #emdr, get your invite here or here.

[Suzee Sasen]

Hello Sebastian,

Glad to see a new face finally coming up to take the reins. I recently came into ownership of http://eve-marketdata.com and hope to see great things from this project. If you want, you can use me as a guinea pig tester for any new implementations, as I need to keep the site up to scrap for consumption. I have a limited knowledge and skillset of various programming languages (still learning every day) so hopefully even with that limitation I can help out.

With that said, if someone can help to keep me in the loop about updates that would change the way I need to parse data and whatnot, even show me what I would need to implement, I'd be greatly appreciative of it.

Best Regards,

Suzee Sasen/Jonathan, owner of eve-marketdata.com