Our intention is to retire the .dat files at some point...

[Ben]

"Our intention is to retire the .dat files at some point and move everyone across to the Atlas API. Once everyone (or at least the vast majority) has moved, we will stop publishing the .dat files." Jonathan Tweed

I currently have a few hundred users accessing the XMLTV .dat files via my Firefox add-on (UK TV Guide and Search). The add-on uses the csv data to populate an SQLite database on the user's own machine. Firefox is able to parse JSON strings so, on the face of it, I should be able to use a result set from the Atlas API.

I did look at the Atlas API but I have to confess that I find it very difficult to interpret - to my eyes the results look very... chaotic.

Is anyone able to provide examples (URLs using atlas.metabroadcast.com?) that 'approximately' include the same data as say: channels.dat, and 160.dat?

Is there any documentation for the many keywords used in the JSON datasets?

Would it not be possible to have an option for the API that returns something a little more akin to the XMLTV .dat files?

[{"title":"Olympics 2012","sub-title":"Day Three Mishal Husain", ...etc }]

I'm also not that clear about the 'API key'. When it's required, and when it's not. I assume that 'hard coding' a key into a Firefox add-on would not be permitted.

Apologies that this post is as scattered as my understanding of the Atlas API!

Ben.

[Adam Sutton]

I have a script that processes Atlas API into XMLTV (and other formats), that might be helpful? It's written in python.

github.com/adamsutton/PyEPG

I wrote this due to my frustrations with the performance of XMLTV script (which I think they have now fixed based on my feedback), but also the Atlas API does provide much greater flexibility. And I think represents a better solution going forward.

Regards

Adam

[steamydave]

Do we have any idea what the time-scale will be for the change over?

 

I would love to re-build using the current api, but am having a little difficulty getting my head around the new api, and getting it to work with php.

 

If the change-over is imminent, then I will not add any users to the system until I have re-written the code, but if we are talking months ahead, then I can build a second system to use the new api until I become competent with it and it's results.

 

Many thanks,

 

Dave

[Adam Sutton]

Dave,

I can't really answer for Metabroadcast, but I can hazard a guess. Basically it will depend a lot on XMLTV guys, if they implement Atlas API use (in favour of .dat files) today then the files will start to disappear more quickly. Since they represent the largest user base and once access levels start to drop I think that will be the time MB will pull the plug.

That being said, if XMLTV guys drag their heels and take too long to transition (not saying they will), then MB may just decide to pull the plug anyway. 

But that's not likely to happen any time soon (like in the next year I'd guess) since ultimately it'll take time even if the XMLTV guys do update for that to reach all users through distro update cycles etc...

However you will see that the .dat files are already being phased out, not all new stuff will be ported to the .dat generation from what I can see. E.g. temporary Olympic channels have are only available via Atlas API and not .dat files.

Adam

[Jonathan Tweed]

On Monday, 30 July 2012 at 13:15, steamydave wrote:

Do we have any idea what the time-scale will be for the change over?

Hi

Thanks everyone for your interest. I'm replying here as it's the most pertinent question, but this should hopefully answer all of the questions in this thread. It's good to see people talking about and helping each other with Atlas :)

Our plan for migrating users of the XMLTV feed is as follows:

• We will have a revised schedule API available in August. It will have more data that the current feeds, will be available without an API key and will be considered an alpha release. At this point we will add a warning to the .dat files that they will be discontinued in due course.
• Once the new API is stable, we will change the warning in the .dat files to state that they are deprecated.
• We will then allow a changeover period of several months.
• After most big and/or known users are transferred we will introduce occasional disruptions to the XMLTV feed, to attract the attention of users who have not heard that the .dat files are deprecated.
• We will then allow a further several-month period, before finally turning off the feed.

We will also not be making any further additions to the .dat files, with new functionality and channels only being available in the API. We may choose to make an exception for major channel launches, such as the recent launch of 4seven, but we do not expect any more of these to occur before we have the new schedule API available. At that point we would expect people to migrate to the API if they want to use the new channels.

For help getting started with the API, the reference documentation and the API explorer on the Atlas site are the best places to look:

http://atlas.metabroadcast.com/#reference

http://atlas.metabroadcast.com/#apiExplorer

The list is here also for people to ask questions. It is possible to use annotations to return related data as part of a request, so for example, the brand_summary and series_summary annotations can be used to return parent titles where available.

The most likely set of annotations for a basic schedule request are:

channel,description,broadcasts,brand_summary,series_summary

If anyone would like to get started with using Press Association data in the API for personal use, please send an email to at...@metabroadcast.com and we'll get you set up. We hope to make this easier alongside the new schedule API, but for now it still requires a key.

Hopefully that clears things up a bit.

Cheers

Jonathan

[Adam Sutton]

On 30 July 2012 15:59, Jonathan Tweed <jona...@metabroadcast.com> wrote:

On Monday, 30 July 2012 at 13:15, steamydave wrote:

Do we have any idea what the time-scale will be for the change over?

Hi

Thanks everyone for your interest. I'm replying here as it's the most pertinent question, but this should hopefully answer all of the questions in this thread. It's good to see people talking about and helping each other with Atlas :)

Our plan for migrating users of the XMLTV feed is as follows:

• We will have a revised schedule API available in August. It will have more data that the current feeds, will be available without an API key and will be considered an alpha release. At this point we will add a warning to the .dat files that they will be discontinued in due course.

I think you just made my day ;) Congratulations (assuming this isnt premature?) on finally getting PA to agree to open the data.

I look forward to trying to new API, the only potential barrier I'm aware of to widespread usage for home DVB now is timeliness of ingest (for last min re-schedules etc...) and async updates to handle this. But then maybe you've solved that in the new API, I've not looked.

Hopefully when I get a break from other work I will take another look at what you guys are up to.

Adam

[Ben]

Hi, Adam, I'm no python programmer but your code looks very interesting. Looking at atlas.py gives me the impression that there are several subtleties to extracting the data using the Atlas API.

Thus far I have been able to figure out how to get a list of channels using Javascript in a Firefox add-on, although, I'm not clear how to select the relevant channels from the 1,152 channels returned (ignoring things like iPlayer, and YouTube).

Looking at your code I think you are using the 'broadcaster' keyword to determine that the channel has a relevant EPG schedule...

if 'broadcaster' in data and 'key' in data['broadcaster']: c.publisher.append(data['broadcaster']['key'])

...but I'm not convinced about this - for example, none of the Sky Channels have a broadcaster, and iPlayer does. Some kind of broadcast_type list would be nice: Freeview, Satellite, Internet.

In case it's of help to others this is my 'log':

Database: C:\Data\Files\EPG\xmltv.sqlite
Executed in executeAsyncSql: CREATE TABLE IF NOT EXISTS Channels (Id TEXT PRIMARY KEY NOT NULL UNIQUE, Id_Description TEXT NOT NULL, Broadcaster TEXT, Media_Type TEXT NOT NULL) Parms()
Executed in executeAsyncSql: CREATE TABLE IF NOT EXISTS Log (Date TEXT NOT NULL, Time TEXT NOT NULL, Log_Text TEXT) Parms()
Executed in executeAsyncSql: DELETE FROM Log Parms()
Executed in executeAsyncSql: DELETE FROM Channels Parms()
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=0
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=100
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=200
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=300
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=400
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=500
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=600
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=700
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=800
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=900
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=1000
100 records returned for Channels.
Fetching: http://atlas.metabroadcast.com/3.0/channels.json?&limit=100&offset=1100
52 records returned for Channels.
Fetching Channels complete.

Yup, I did try limit=10000 but 100 is as high as is allowed.

Would it possible for you to publish a log file from the execution of your scripts? I'm assuming this would reveal the calls you are making to the API for schedule.json

Ben.

On Monday, July 30, 2012 12:53:24 PM UTC+1, Adam Sutton wrote:

I have a script that processes Atlas API into XMLTV (and other formats), that might be helpful? It's written in python.

github.com/adamsutton/PyEPG

I wrote this due to my frustrations with the performance of XMLTV script (which I think they have now fixed based on my feedback), but also the Atlas API does provide much greater flexibility. And I think represents a better solution going forward.

Regards

Adam