EMDR Dead?

[Suzee Sasen]

Did the CREST crawler that was being run for EMDR die? I'm getting very little bits of data now, and almost all data has stopped coming in. I need to know now so that I can either wait for repairs or look for a new solution.

Any and all help is greatly appreciated.

[James Muscat]

I think you might mean me...

Yeah, looks like my VPS was rebooted without anyone telling me (until I woke up to lots of alerts this morning). Up and running again now. (CREST was having some wobbles yesterday too.)

You know everyone, the point of a distributed system like this is to avoid single points of failure (and other CREST trawlers are available)... ;-)

James

On 11 July 2016 at 05:15, Suzee Sasen <hellha...@gmail.com> wrote:

Did the CREST crawler that was being run for EMDR die? I'm getting very little bits of data now, and almost all data has stopped coming in. I need to know now so that I can either wait for repairs or look for a new solution.

Any and all help is greatly appreciated.

--
You received this message because you are subscribed to the Google Groups "EVE Market Data Relay" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eve-emdr+u...@googlegroups.com.
To post to this group, send email to eve-...@googlegroups.com.
Visit this group at https://groups.google.com/group/eve-emdr.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Rauscher]

Hi James,

Nice to finally hear from you. I was getting worried you disappeared. I have a few questions for you:

1. You say this is a distributed system. Are you referring to the relays available here? Because I thought your trawler was the one that fed data to EMDR, which then pushes that data out to the relays.

2. Can you get me signed up to your alerts so that I know in the future if the trawler has gone offline? It'd greatly help me in troubleshooting.

3. You mention that there are other CREST trawlers. Could you point me to some? I haven't been able to find any.

4. Market hub data (Jita, Amarr, etc) is still not coming in. Do you know how long it will take before it does, or if it should be coming in already?

I really appreciate your efforts James. Hope to hear from you soon.

[Jonathan Rauscher]

Also, a-lot of the data we're getting in has a solarsystem_id of 0. Any idea on why this might be?

[James Muscat]

Hi Jonathan,

On 11 July 2016 at 14:33, Jonathan Rauscher <r.jon...@gmail.com> wrote:

Nice to finally hear from you. I was getting worried you disappeared. I have a few questions for you:

I didn't know I was in such high demand!

 

1. You say this is a distributed system. Are you referring to the relays available here? Because I thought your trawler was the one that fed data to EMDR, which then pushes that data out to the relays.

I'm not "part of" EMDR, just a keen user. I'm not the only one who's been using a CREST trawler - mine isn't (and should never be!) "the" one. Anyone can feed data in, it's just that not many are doing so right now.

 

2. Can you get me signed up to your alerts so that I know in the future if the trawler has gone offline? It'd greatly help me in troubleshooting.

No, but the alerts are triggered off of http://www.eve-markets.net/stats.json which might be of some use to you. (I should caveat that: it's an internal part of how my site functions, and could change in future; it's not public API.)

 

3. You mention that there are other CREST trawlers. Could you point me to some? I haven't been able to find any.

Digging not too far back in this group's history:

 - https://github.com/antihax/CrestEMDRBridge

 - https://github.com/theatrus/crestmarket

 - Element43 CREST EMDR Bridge (I don't think the source is public)

 

4. Market hub data (Jita, Amarr, etc) is still not coming in. Do you know how long it will take before it does, or if it should be coming in already?

It takes my trawler a couple of hours to cycle through all the regions (it does so in a random order).

Your follow-up email suggested some solarsystem IDs of 0 getting into the data, which I'll have to look into - it shouldn't be dropping anything that's in CREST's response, but I'll check.

James

[Jonathan Rauscher]

James,

I'd certainly appreciate the investigation for those 0 solar system ids. I look forward to your results about that.

Also, could I get an updated version of your trawler? Would like to look at it.

[James Muscat]

Derp, should have remembered this discussion but your mentioning zero put me off.

Short answer: solarSystemID is not available from CREST. My trawler specifies null, as per the specification [0].

Longer answer:

In most (but not all) cases you can reverse-engineer the solarSystemID from the stationID which is provided, using data from the static data dump (not available over CREST). Doing so in the trawler would introduce a lot of extra work, and the spec leaves the option open, so I took the path of least resistance and just fill in these values with null. If that data's important to you then you can do the lookup once you have the market data. (Or write a patch for the trawler, which is open source[1]!)

This won't work with citadel-based markets, which I believe are starting to appear in the feed.

That's probably not the answer you were hoping for...

[0] http://dev.eve-central.com/unifieduploader/start

[1] https://github.com/jamesremuscat/CRESTMarketTrawler

[Jonathan Rauscher]

Interesting. Before the data feeds went out about 6 days ago, we were gettting solarsystem_id s for just about everything for the past month or two prior. Thoughts?

[James Muscat]

As I said: I'm not the only person who's written a trawler. I suspect others have been less lazy than I have.

James

[Jonathan Rauscher]

Your trawler: is it only getting data from CREST, or is there another source it's pulling from as well?