Data access status

[Adam Sutton]

Hi All,

I've finally found some time to get back onto working on my grabber script. I'm almost at the point where I will personally be using it in anger. i.e. the sky box will be replaced by a home brew DVB-S2 solution:

Backend - tvheadend (with customisation) + pyepg (my EPG grabber script)

Frontend - XBMC PVR

It was already in a usable state when I left working on it several weeks ago, so I'm mainly working on tidying it up so it might actually be usable by others, packaging, configuration, etc...

The one thing that will be limiting at the moment though is the API key access, currently I have PA access which is great for me as it gives me a full(ish) freesat EPG. However for everyone else they're only likely to get what, BBC data?

I know there were discussions going on with PA (and the other data publishers) to try and open things up. I'm wondering how this is progressing?

Are you aiming at full data access? Would it be possible to aim at a "limited" access that basically provides data equivalent to that used in the XMLTV feed (i.e. no pictures etc.., not sure if this is what they're most keen to protect?).

Or can we at least get to a point where getting personal access keys for PA data is relatively straight forward and even better could be handled automatically by say my script? This would at least allow you/PA to track the usage while making the data generally available.

Regards

Adam

[Chris Jackson]

Hi Adam,

Discussions have gone well. We plan to provide exactly what you suggest - the current dataset from XMLTV, in the format of the Atlas API.

This has taken a while 'cos:

1. we want to ensure full compatibility between this new service and the forthcoming 4.0 API, to avoid migration after code is out in the wild 

2. we want to make sure we can do a later seamless migration to our new architecture (which will put us in a new place in the perpetual speed vs freshness vs flexibility tradeoff)

2. we've been very busy bringing new customers onboard

Hopefully we'll have something to report in the next few weeks. But meanwhile you can be confident that your work will be useful to a wider audience who don't have an API key.

Best,

Chris

--
ch...@metabroadcast.com -- +44 7967 756705

[Adam Sutton]

Chris,

That's great to here. I'm currently working with a group of devs who are looking to update the DVB backend software I use (tvheadend) and I'm hoping to help rewrite the EPG (and scraper) code using many of the concepts I've put into my script (stolen from you guys, since your the primary source of data :) ).

Didn't want to go down that line too far and find that most people simply wouldn't be able to use the script and it'd only be useful for the BBC data.

Is there any news on any of the other providers opening things up, or is this all still limited to PA?

Adam

[Chris Jackson]

On 15 May 2012 16:42, Adam Sutton <a...@adamsutton.co.uk> wrote:

Is there any news on any of the other providers opening things up, or is this all still limited to PA?

No news yet - but we're working on it! Actually, I anticipate that having your software there is going to be a great help in making progress. Hopefully together we can overcome the chicken-and-egg situation.

 

[Adam Sutton]

Any further updates?

My work on integrating my script and the general concepts gleaned from you guys (i.e. the hierarchical organisation of the data) into tvheadend (one of the preferred backend solutions for XBMC PVR) is going well.

And I do now have a few people interested in using my script, though I have to keep telling them that at the moment without requesting a key from you guys it won't be of much use. I'd really like to see it being used but short of my generating the data offline and providing that to end users (is that even allowed?) I think its going to be a bit of a pain.

Regards

Adam

[Chris Jackson]

On 28 May 2012 09:18, Adam Sutton <a...@adamsutton.me.uk> wrote:

Any further updates?

No - we are working in it, but these things take time. I have a meeting with one of the parties tomorrow. 

My work on integrating my script and the general concepts gleaned from you guys (i.e. the hierarchical organisation of the data) into tvheadend (one of the preferred backend solutions for XBMC PVR) is going well.

And I do now have a few people interested in using my script, though I have to keep telling them that at the moment without requesting a key from you guys it won't be of much use. I'd really like to see it being used but short of my generating the data offline and providing that to end users (is that even allowed?) I think its going to be a bit of a pain.

Great news that things are progressing well.

Bear with us - we will get there. I don't want to promise a date, because we want to do it properly and these things can take time. But it is  priority. People can always request keys for now. If you email us a list of name, email address, we could just generate a bunch of keys all at once.

Generating data offline would not be ideal.

[Adam Sutton]

Good to hear things are progressing, but some idea of a timescale would be nice :) or maybe some regular updates, maybe as part of the weekly API announcement? It would be nice to know, who you're talking to (is it just PA, or the rest of the big 5, BBC excluded of course), how each is progressing, what the likely openess is going to be (basic data, full metadata, image data, etc...).

To me, and probably others, its one of the most important topics. As previously mentioned, I'd be happy even if the default access was limited to basic scheduling info, i.e. not the pictures etc... (although they'd be great long term). There are a lot of projects going on these days around open PVR platforms, from the off the shelf Enigma stuff, XBMC PVR (my choice), MythTV, Windows Media something or other, etc.. And one thing where they all seem to struggle, to varying degrees, is handling EPG data and the related recording capabilities.

You database, and the basic structure, I believe can be a great eye opener to these communities which might in turn help generally with other such providers elsewhere in the world. And could ultimately lead to some great innovation and cool features beyond those currently available in the standard commercial (thinking Sky+, etc..) solutions.

Generating data off-line is indeed far from ideal, however if many people require the data in the same format (ala the XMLTV feed) then doing this a) reduces the burden on you guys (since the number of dynamic DB queries is reduced) and b) reduces the burden and time to process on the clients.

For example building a 14 day EPG using PA+BBC (the only feeds I have access to) takes between 5-10mins, depending a various factors. But ultimately serving up the resultant XML file (especially if compressed) would take considerably less time and require almost 0 cpu load at either end. So there is something to be said for it. Though it does have definite limitations where people want slightly different things (i.e. diff publisher settings, etc...).

So the question really is, is that allowed under the current licensing and my API key? As I definitely don't want to step on anyone's toes and cause problems with your efforts to generally open up the original source feeds etc... 

Regards

Adam

[Chris Jackson]

On 28 May 2012 21:38, Adam Sutton <a...@adamsutton.me.uk> wrote:

Generating data off-line is indeed far from ideal, however if many people require the data in the same format (ala the XMLTV feed) then doing this a) reduces the burden on you guys (since the number of dynamic DB queries is reduced) and b) reduces the burden and time to process on the clients.

For example building a 14 day EPG using PA+BBC (the only feeds I have access to) takes between 5-10mins, depending a various factors. But ultimately serving up the resultant XML file (especially if compressed) would take considerably less time and require almost 0 cpu load at either end. So there is something to be said for it. Though it does have definite limitations where people want slightly different things (i.e. diff publisher settings, etc...).

So the question really is, is that allowed under the current licensing and my API key? As I definitely don't want to step on anyone's toes and cause problems with your efforts to generally open up the original source feeds etc... 

Clearly we need to explain our Atlas 4.0 work in more detail. This will provide much faster schedule access. Watch this space.