Discussing the future of EMDR

[Gregory Taylor]

It was recently announced that market order data will be making its way to the CREST API soon. I wanted to briefly get some opinions on how we should proceed with EMDR. Ultimately, the decision will be made based on whether a sufficient number of market sites would still find our service useful.

With the new CREST market data APIs, anyone who signs a developer agreement with CCP will have API access. Unfortunately, it appears that you are going to need an active subscription to do that. You'd also need to do the API polling on your own, which may be a plus or minus depending on your available infrastructure and tooling. Polling is still a bit more code and resource-heavy than our ZeroMQ feed.

To offer value on both fronts, here's my current thinking for what to do with EMDR:

• We stop accepting player uploads. The API delay is 5 minutes max, making the scraped uploads unnecessary. The risk of malicious/bad payloads far outweighs the 5 minute delay, in my opinion.
• The upload gateways are replaced with processes that continuously hit the market data APIs, converting the responses into Unified Uploader Data Interchange Format.
• There is no interruption for any sites currently using EMDR. Business goes on as usual, with the benefit of there no longer being malicious/bad payloads to worry about.

Assuming that we take this path going forward, I'd like to take a brief hand count on who would still be using EMDR and who would ultimately switch to pulling directly from the CREST API for market orders. I don't only want to hear from people who would stick with EMDR, I'd like to get a feel for the staying/gone percentage split. If there is still a demand for EMDR going forward, we'll re-work ourselves into something like what I described above.

If you'd prefer to email me directly instead of through this list to let me know what you'll be doing, you can reach me at gtaylor (at) gc-taylor (dot) com. Otherwise, replying to the list is perfectly fine.

[Yann Ramin]

As for EVE-Central, the answer will be “both” for some period of time, especially as the CREST rate limits and such are unproven still. 

I think there is still a value in federating the CREST data back over EMDR.

- Yann

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at http://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

[zweiz...@element-43.com]

All I heard from CCP is that rate limits on private CREST will be really high. And even if not, I assume that we would find a solution (e.g. whitelist) - in the end they would benefit from reduced traffic.

[Jimi]

Here is some extra info I received from Regner (CCP FoxFour) today. The EVE client has stopped, for a while now, its own cache usage. They only kept the cache for us (EMDR and friends) and they would like us to switch asap to a non cache scraping procedure. Regner also asked me to provide him with a list of any data missing from CREST API, that we currently get from the cache, in order to include it in the CREST API. So whatever the decision taken here, EVEMon will stop cache scrapping as soon as CCP gives the word or EMDR have switched CREST.

[Gregory Taylor]

On Fri, Nov 21, 2014 at 11:57 AM, <zweiz...@element-43.com> wrote:

All I heard from CCP is that rate limits on private CREST will be really high. And even if not, I assume that we would find a solution (e.g. whitelist) - in the end they would benefit from reduced traffic.

I'm not incredibly concerned about the technical part of this. As you say, we'll be able to make it work.

I'm more concerned about making sure EMDR is still something that more than a few sites would find value in. Right now, overhead is pretty minimal for me ($5/month) and the relay volunteers. With the new CREST API, I'd need to maintain an active subscription at all times, so we'd be looking at $20/month base, which is a bit more for me.

I don't mind taking on the cost, but I want to make sure that there's still a good bit of utility to justify said cost. Unless CCP would cut an EMDR-specific API key that isn't tied to a paid subscription, then this is more of a no-brainer.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[Jimi]

Repeated original post with grammar corrections.

"There is always the solution of PLEX donation via the client. Sites that will use EMDR can do a one time donation (depending on how many will use it, like if those sites are 12, it will be a per year donation)."

[Regner Blok-Andersen]

To give you an idea of the rate limit for authed CREST, it's in the multiple thousands a second. So it should be plenty high. I actually don't know if your account has to be active to create an application on the developers site, but we are also working on removing the restriction on only active accounts signing in with the SSO.

The result should hopefully be that you don't actually have to currently be paying us money, only that you have at some point in the past, and you should have more than enough of a rate limit.

If you have any specific questions or concerns please let me know. I will try and respond ASAP. Want to make sure this works as best as possible for as many people as possible.

[Gregory Taylor]

I wasn't able to sign in and get set up with the new API due to the inactive account issue. That's pretty much a non-starter to being able to experiment. I'm glad to hear there are plans to fix this, but the current reality prevents progress on that front.

That's good news on the API rate limits. We'd need to discuss how fast our developers want the data to flow, but I'd doubt there'd ever be a situation where we'd need to do multiple thousands a second.

--

You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at http://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

[Tom Neyland]

According to some napkin math you would need in the ballpark of 2,500 requests a second to poll all items in all market regions every 5 minutes.

We may not need to poll data that frequently, but I absolutely plan to given the opportunity.

[Tom Neyland]

To be a little less vague, I have a hobby project which builds ~1mb (gziped) regional market summary json files. These are consumed and visualized by a webapp. Right now there is relatively large variation in the recency of order data, being able to refine the that to a 5 minute window would be nice. It'd be be more than nice really, it'd enable additional types of analysis that are currently impractical. 

[Gregory Taylor]

Before we can get too far into the technical details, I need more information on who would still find EMDR useful.

Does this mean that you'd rather poll for yourself rather than use EMDR? That's the question I'm mostly looking for answers to right now. It's OK either way, I'm not going to get offended. Just trying to figure out what most are going to do.

[Tom Neyland]

At the end of the day my goal is this: to have as up-to-date data as possible, for every item, in every region. This does mean I can see myself shifting to polling CREST if the rate limits are as generous as currently proposed.

I'd be willing to bet that my usecase represents only a portion of current EMDR consumers, and that others would still prefer the stream.

If you find that most people do shift over to CREST for order data, but are still interested in running a service, are you willing to consider shifting the role of EMDR to something new? For example, emitting aggregate statistics per item per region, rather than order-level data. Aggregate statistics are just one of a few different interesting possibilities.

[Kenneth Jorgensen]

That's great news about the market data becoming available directly from the source. I think switching EMDR off player uploads and onto the official API is a good idea. The current format would also be preferable as legacy code wouldn't have to be changed. It would require some form of coordination between the source servers doing the actual pulling from the API. If they could be made to work together to provide the best possible coverage over all regions and items, delivering regular updates for each, EMDR would have value over CREST. I know you'd rather not get into technical details, but the API rate calculations are crucial in determining the future value of EMDR.

If it would require 2,500 req/s to cover everything at 5 minute intervals and the rate limitation will be in the thousands, it may be feasible for any one person to get full 5-minute granularity. If EMDR could offer blanket coverage with say, a <1 minute granularity by having multiple clients working together and sharing the load, I definitely see a value in it. Additionally if EMDR filters out duplicate updates (or a new record type is introduced to say "no change", so we we can just update timestamps), that adds value as well.

Given the above, I would personally use EMDR over the CREST API, and only use the CREST API for historical data.

[Nolen Cadmar]

I would prefer the new method. I work with Google Docs to provide spreadsheets for a multitude of clients. The new CREST data will be incredibly difficult if not impossible to pull into Google Docs and/or Excel. I would love to be able to pull the CREST data from a source that does all the manipulation for me. I had a lot of clients excited about being able to pull data directly from CCP and not worrying about holes or old data coming from EVE-Central. These clients were quite dissapointed to discover that the data would not likely be accessible in their application, but I ensured them I would search for third party services able to provide the data in a format that I could use.

Needless to say, I'm glad you're considering using the CREST data. I'd be willing to donate a few PLEX to keep your account going if that's what needed to be done.

~Nolen Cadmar

[Michael Cummings]

If it would require 2,500 req/s to cover everything at 5 minute intervals and the rate limitation will be in the thousands, it may be feasible for any one person to get full 5-minute granularity. If EMDR could offer blanket coverage with say, a <1 minute granularity by having multiple clients working together and sharing the load, I definitely see a value in it.

Just want to say because the data is cached at CCP having multiple account polling the same data does get you less than 1 min granularity, you just end up with dups and it would be a waste of time.In other words the first request CCP actually calculates the data and all other requests after that they just return the same data until the cache expires.

Where I see EMDR still being useful is in providing cross market info and other add on services.

[Galattico Bontius]

I'm currently developing a site that uses the EMDR for its data and I've found it to work very well.  I would definitely see value in keeping the EMDR going as you explained in the OP.  One obvious advantage is the data format.  Of course, legacy code continue to work would be a nice bonus as well.  I'm sure CCP would appreciate the load being distributed to the EMDR network rather than everyone polling CREST too.  In addition, the work of getting the data is kept to the "professionals" (EMDR) and I can focus on what's most important, the functionality of my site, knowing that the data aspect has been handled properly by the EMDR.

If there is financial need for this project I may be willing to assist, just let me know.  I would hate to see the EMDR die due to financial reasons.  I haven't contributed yet, as I only started using it, but would be happy to setup a monthly contribution.

[James Muscat]

I'd like to see EMDR continue, but being fed by perhaps a shared net
of CREST pollers, rather than cache scrapers.

On 24 November 2014 at 04:41, Galattico Bontius

[Ryan Holmes]

I would be interested on EMDR continuing. Doing these market pulls on the fly for my webapp would be too resource-heavy (in the page load way, not bandwidth). Or I can pull them during set intervals, but I try to keep as up to date as possible. So an EMDR feed running into my Redis server was the best solution for me, and I believe it to still be the best solution going forward. 

[Kenneth Jorgensen]

If it would require 2,500 req/s to cover everything at 5 minute intervals and the rate limitation will be in the thousands, it may be feasible for any one person to get full 5-minute granularity. If EMDR could offer blanket coverage with say, a <1 minute granularity by having multiple clients working together and sharing the load, I definitely see a value in it.

Just want to say because the data is cached at CCP having multiple account polling the same data does get you less than 1 min granularity, you just end up with dups and it would be a waste of time.In other words the first request CCP actually calculates the data and all other requests after that they just return the same data until the cache expires.

Very true, the granularity we get depends depends on the caching done at CCP.

[Ai]

I will be refactoring iveeCore to use the new CREST endpoint instead of EMDR.

However, I see the following reasons to keep EMDR up, at least for a while:
- Continuity: It is not trivial to switch order and history data from EMDR to CREST given one is push, the other pull
- Suitability: I can imagine some consuming apps benefiting from EMDRs push nature
- To register for authed CREST, users need to have paid CCP at least once with credit card. This can pose a serious hurdle for some people outside the US.

If EMDR is kept running with CREST as data source, be sure to include history data, as it would be ugly to require users to use both EMDR and CREST to get complete market data.

[Gregory Taylor]

So at this point, it looks like there's sufficient interest to justify experimenting with how to adapt EMDR to use CREST. I'd like to do this, but the prospect of paying $15/month (on top of what I pay to keep EMDR running) just to tinker isn't too attractive. Rather than mooch for PLEX just yet, I wanted to see if anyone had any CCP contacts that we could talk to about getting an EMDR-only CREST API key to play with. If you know of anyone from CCP who might be able to help, please point them at this post or give them my email address.

I emailed Thomas Meise, but haven't heard anything back from him. I'm thinking that's probably a dead end.

We'd be saving them some money and they'd be able to discontinue cache scraping without a bunch of developers getting pissed if we could get this sorted out in time.

[Michael Cummings]

I don't know a lot about them since I haven't looked into getting one but CCP does make free Fan Site accounts available and I'm sure I saw Dev comment that they will have CREST access so might want to do some checking into them. I'd look at contacting [CCP] FoxFour and he can either help you or direct you to the right people about them I'm sure.

[Gregory Taylor]

If you have his email address, can you send it my way?

--

[Jimi]

Regner Blok-Andersen is CCP FoxFour.

[Regner Blok-Andersen]

I am indeed [CCP]FoxFour. I am working on getting it setup that accounts which are currently inactive can still create third-party applications. The requirement of having paid us some money at some point will still be in place of course.

If you need access now on Sisi, or even when it comes to TQ, I would be happy to create an application tied to your inactive account manually. If thats something that interests you please feel free to contact me via my work email which is my first name at CCPGames.com :)

[Gregory Taylor]

Enter code here...

Enter code here...

In order to provide an update for everyone, I am set up with the CREST API. I'll begin experimenting shortly. Thanks for everyone's input, and to Regner for getting this set up.

[Marko Harjula]

I'm just trying to figure out some strange CCP anomaly here :) ... so actually suggesting to hitting HTTP API to generate EMDR data or did I miss understand?
I mean I would probably get spanked at work with about this kind of ideas when there is "stream" of market data changes going to SQL, which stream probably can easily tapped in & json:ed and connected directly EMDR as pub. (or provide even own zmq/rmq/websocket provider)
Of course if CCP have extra CPU cycles to spend on HTTP API no problem, but I just wondering as this concept sounds bit awkward :)

[Gregory Taylor]

On Fri, Dec 12, 2014 at 1:24 PM, Marko Harjula <marko....@gmail.com> wrote:

I'm just trying to figure out some strange CCP anomaly here :) ... so actually suggesting to hitting HTTP API to generate EMDR data or did I miss understand?

Yes, that's the currently suggested approach.

 

I mean I would probably get spanked at work with about this kind of ideas when there is "stream" of market data changes going to SQL, which stream probably can easily tapped in & json:ed and connected directly EMDR as pub. (or provide even own zmq/rmq/websocket provider)
Of course if CCP have extra CPU cycles to spend on HTTP API no problem, but I just wondering as this concept sounds bit awkward :)

I'm not sure I understand what you're trying to ask or suggest. Are you wondering why CCP isn't offering their own feed?

FWIW, I think CCP should be offering this sort of thing officially, as it would be a lot more efficient than HTTP API polling for sites that need the full range of data. Heck, pay or hire me to do it and I'm there. But that may just be me thinking that EMDR has been more valuable than it really is (I'm probably biased).

[Marko Harjula]

Yes, as hitting HTTP API just creates tons of useless CPU cycles if there is only small changes or no changes at all. EMDR data "collection" is (and messagebus generally) are very useful for client as I don't need to spam API to get data I need and building "real" time App is very easy, pull initial values from some HTTP API and just listen for incoming changes.
EMDR itself have been very useful for me, I only missed filtering like for example with item IDs and/or system  (i.e. like putting each systems/itemIDs as own pub/sub channels) as then I would have built mobile app to show needed items with live prices (now only in PC as bit too much data for phone/tablet to handle if not in wifi) :)
Of course direct market feed from server would be awesome, but in that case I think CCP might need add some minutes delay to data as else Eve market would be full of real time trading bots  ... though is difference to real world trading? :)



 

[Yann Ramin]

As much as I want real-time trading bots, it will sadly probably never happen :-)

A stream output is ideal, even one which batches changes and emits a new order summary every 5 minutes in reaction. Sadly, this requires polling with CREST, which is a *lot* of requests.

EVE-Central right now polls the entire market for everything. Running parallel at 2xregioncount still takes quite some time, eats a lot of bandwidth (about 1MB/s down 0.5MB/s up), and even causes CREST to fall over with NGinx proxy errors from time to time. I’m working on a reactive core, so queries will trigger CREST updates as opposed to scanning everything all the time as the only mechanism of updates. Having the entire market available for queries with low latency is totally essential for about half of EVE-Central’s functionality.

[Gregory Taylor]

If you are moving in a similar direction, maybe I could talk you into writing an EMDR backend that tosses the output out to an announcer via ZeroMQ (like our current gateways work) :)

[Yann Ramin]

Already have most of the infrastructure and code to do so, heck it even outputs EMDR Unified messages.

[Marko Harjula]

Yann: just as this is quite interesting stuff ... how long it actually takes to run one full scan for market data? :)

[Daniel Barnes]

 I did a similar thing tapping emdr and used redis pub/sub to listen for changes on certain items.

Daniel Barnes

[Daniel Barnes]

 I think someone did the math already in this thread, but your looking at ~11,000 market types across all the regions. I think that Polling everything would be quite excessive.

--

Daniel Barnes

[Regner Blok-Andersen]

"and even causes CREST to fall over with NGinx proxy errors from time to time."

I would be really interested in knowing more about this. If you can get me any details, such as when it happened, the user agent you were sending at the time (or IP of the server, but UA works just as well). This shouldn't be happening at all, none of our servers are even remotely under enough load to cause any kind of disruption to service.

Anyways, yea, any details you can offer would be greatly appreciated. :)

Regner Blok-Andersen
http://highguard.net/

You received this message because you are subscribed to a topic in the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eve-emdr/POrf0otoFZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eve-emdr+u...@googlegroups.com.

[Yann Ramin]

Once I ramp up the RPS and hit it I'll follow up with the details. 

[Ai]

I very nice characteristic of EMDR (or rather scraped data) which im going to miss with CREST is that item data frequency correlated strongly with actual market activity. Trit in Jita gets frequent updates, obscure items in Feythabolis do not, and the data typically came in when a user actually interacted with the market, i.e. caused changes. This is completely decoupled with CREST. There is no efficient way of getting the data that actually changed, or the hottest items. One has to scan the whole thing with 1.5 million requests at least (more if you go with the CCP suggested way of traversing the CREST nodes following hrefs).

[Otaci]

I'm late to the discussion, been away for a while, but just wanted to throw in my vote for keeping EMDR going. I'm willing to volunteer resources as needed.

Otaci

[Fuzzysteve]

Not many more though. 23 pages for items, and a few pages for regions. But those are cachable, for 1 hour (at least) each. Which helps cut it down.

[Regner Blok-Andersen]

Hopefully for the next EVE release, assuming the review doesn't get declined, I have added id, type[id], location[id], and volumeEntered to vnd.ccp.eve.MarketOrder-v1, which is used in the buy and sell CREST market orders. Hope that helps! :D

[Regner Blok-Andersen]

And just to spam you all some more: https://gist.github.com/Regner/74981cbbde05ce5faac1

An example of the market resource with the new field. :)

[Gregory Taylor]

That looks very helpful. Is it already on Sisi?

--

[Regner Blok-Andersen]

Not right now, it will go to Sisi tomorrow I hope. Will let you know when it does. :)

You received this message because you are subscribed to a topic in the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eve-emdr/POrf0otoFZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eve-emdr+u...@googlegroups.com.

[Regner Blok-Andersen]

The changes are live on Sisi: https://api-sisi.testeveonline.com/market/10000002/orders/buy/?type=https://api-sisi.testeveonline.com/types/587/

On Tuesday, January 6, 2015 4:13:11 PM UTC, Regner Blok-Andersen wrote:

Not right now, it will go to Sisi tomorrow I hope. Will let you know when it does. :)

On Tue Jan 06 2015 at 3:06:49 PM Gregory Taylor <gta...@gc-taylor.com> wrote:

That looks very helpful. Is it already on Sisi?

On Jan 6, 2015 9:30 AM, "Regner Blok-Andersen" <shad...@gmail.com> wrote:

And just to spam you all some more: https://gist.github.com/Regner/74981cbbde05ce5faac1

An example of the market resource with the new field. :)

On Tuesday, January 6, 2015 11:15:50 AM UTC, Regner Blok-Andersen wrote:

Hopefully for the next EVE release, assuming the review doesn't get declined, I have added id, type[id], location[id], and volumeEntered to vnd.ccp.eve.MarketOrder-v1, which is used in the buy and sell CREST market orders. Hope that helps! :D

On Sunday, December 21, 2014 7:38:21 PM UTC, Fuzzysteve wrote:

Not many more though. 23 pages for items, and a few pages for regions. But those are cachable, for 1 hour (at least) each. Which helps cut it down.

--

You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+unsubscribe@googlegroups.com.

To post to this group, send email to eve-...@googlegroups.com.
Visit this group at http://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eve-emdr/POrf0otoFZc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to eve-emdr+unsubscribe@googlegroups.com.