ANN: Deprecation of Market History messages

[Gregory Taylor]

With the CREST market history endpoint being superior to our market history message relaying in just about every imaginable way, I'd like to set a deprecation timeline for our end of this.

• Within the next week or two, I'll adjust the documentation to remove references to history messages. This goes for uploading or reading from the stream.
• On or around August 22, we'll stop relaying market history messages.

The end result is that developers that are currently reading market history messages from EMDR will need to adjust and use the official CREST endpoint by August 22 to avoid disruption.

Making this change will help us drastically reduce message volume, making EMDR less resource intensive and eliminating the need to rely on an imperfect data source. This is an upgrade for everyone in just about every imaginable way.

EMDR is still going strong and just as healthy as ever, so definitely don't read into this any other way. This just lets us reduce our data volume a bit, while also narrowing our scope. Win-win!

[zweiz...@element-43.com]

Nooooo - EMDR still has a couple of advantages:

1. Today's data is not available via CREST
2. Data older than 14 months can't be retrieved from the API, but is commonly found in EMDR history messages
3. No scraping necessary, no 10 connections/sec limitp

I'd propose a filtering mechanism for EMDR, which discards market data points that can be retrieved from CREST and only lets current (age < 24h) or datapoints older than 14 months pass the filter. This also would reduce traffic enormously.

I'm currently preparing version 2.0 of node43 - the NodeJS based consumer of Element43. As order/history messages hit the consumer, each region/type combination in that message will trigger a CREST history API call which only gets called once every 24h. If that same region/type pair hits the consumer 3 minutes later, another CREST update would not be scheduled. With the CREST history API comes another problem: you'd have to scrape the API if you'd like to get the entire market. This won't make neither CCP nor us 3rd party devs happy. So the idea is that node43 will submit the data collected from CREST directly to EMDR in UUDIF format. Since the API calls only occur for active items (region/type pairs that get sent to EMDR indicate activitiy most of the time, as the upload is triggered by people looking at items in the in-game market) and only once every 24h, redundancy of the EMDR history feed would be greatly reduced.
This would mean:

1. Reduced redundancy and traffic of the EMDR history feed
2. Current and very old records will still be available
3. No need to rewrite existing applications, as node43 uploads official data the same way players do
4. Official data available via EMDR -> reduced CREST server load, no scraping necessary
5. EMDR users could filter for node43's ip hash so they can decide to only consume official data
6. Reduces load and complexity for other EMDR consumers

[Gregory Taylor]

1. Today's data is not available via CREST

Seems like current order data (not history) would be better for this, though. You'll get all kinds of order data of varying ages on a given day.

 

2. Data older than 14 months can't be retrieved from the API, but is commonly found in EMDR history messages

This is true, but isn't a problem if you are good about storing data/history. EMDR by nature is pretty ephemeral, too.

 

3. No scraping necessary, no 10 connections/sec limitp

I can only speak to your particular setup (celery + Django), but you can easily scatter these updates over the course of a day and still see excellent performance and freshness of data. The CCP devs wouldn't expose these API calls as they are currently built unless they were OK with people using them frequently. There's really no other way to use them if you want the full market history.

I'm definitely not hard set against keeping history data, but I'm wanting to go that way unless there is a large outcry on here. You are currently the only one asking to keep it.

There's a definite balancing act. We can't match the accuracy/quality/safety of the CREST data. It's superior in just about every way. We use a lot of bandwidth on the EMDR side and the consumer side pushing out the history messages, which can get uncomfortable for our metered bandwidth relay volunteers and consumers.

I'm all ears, though. Anyone else feel strongly either way about this? I really wanted to get this kind of discussion over with weeks ago when I initially asked for opinions on this.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[zweiz...@element-43.com]

Seems like current order data (not history) would be better for this, though. You'll get all kinds of order data of varying ages on a given day.

Very good point. Have not thought about it that way yet. 

I'm okay with the deprecation of EMDR history data. In the end this would save us a lot of code, traffic and processing power. I just wanted to make sure that arguments of both sides were involved in the process.

[Gregory Taylor]

Thanks, Zwei. If anyone else has thoughts on this, please do speak up soon. We'll continue with the deprecation timeline outlined earlier unless something serious comes up soon.

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at http://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

[Gregory Taylor]

Consider this a reminder to all application developers to remove dependency on history messages by August 22. We are now a little under a month to go until we no longer relay market history.

[Gregory Taylor]

The deadline has came and passed, but I've been busy to the point where I haven't been able to get the history stuff removed. I'll do this as soon as I can spare a few hours to update the docs and the announcers.

For anyone still consuming EMDR history messages: Get switched over to CREST! I'll finish this history removal as soon as I have a spare moment, so there can't/won't be any extensions on this.