Proposal: XSI Content Negotiation (service filtering)

[Ben Poor]

Dear All,

There has been some discussion as to methods of providing smaller 'bite-size' XSI files - e.g. specific to a single service. The motivations for this are to allow for direct client/device ingestion of XSI (for situations where a directory service provider is not available), and also to cover more service provider situations beyond the traditional broadcaster model (which I have to admit I am naturally drawn to!).

To provide this functionality I propose following the path of 'Content Negotiation', i.e. The client will include parameters in the HTTP request headers, which the service provider may look at and use them to modify the response. The client should then inspect the response and process it. The advantage of using this approach is that it allows for a simple monolithic XSI, but providing for advanced behaviour for clients and service providers that choose to implement it - allowing both big and small service providers and complex or simple devices to participate, with the same set of rules.

I propose we add functionality to the XSI section which will allow for Clients to add the optional HTTP request header when asking for the XSI document: X-Bearer

This would indicate the 'bearer' that the client wishes to know the service information for. Typically this would be for the use case of a client, having tuned to a service, wishing to know the service details. They would take the broadcast parameters and form it into the bearer URN format detailed in the RadioEPG specification. So for example, a service tuning into Capital London on FM:

X-Bearer: fm:ce1.c479.09580

This information may be used by the Service Provider to return a more specific match for this service, but it is important to note that whatever the response it is the client's responsibility to examine the response and perform any matching against bearer details. The client must not assume that the response perfectly matches only the requested bearer details.

This could also potentially be used in different markets to fulfil different requirements for returning data to the client. For instance, in markets where service providers wish to collaborate regionally, the bearer information could be returned to show a 'regional' view of services (i.e. the requested service and others in the locality).

A better user experience could potentially be created if the device provides this information and properly interprets the results, and the service providers reacts to this information and properly sends back the most appropriate response. Again, we are not proscribing behaviour but setting up the conditions such that there are clear benefits in both client and service provider implementing this functionality.

I'd be interested to hear views on this one. Again, like elements in other RadioDNS specifications it has no impact on the core specification but allows us to look beyond a small set of use cases and to future uses and users.

Ben

ben....@radiodns.org

[Nick Piggott (RadioDNS)]

Hello,

I'd just like to emphasise that the thinking of this EPG group is very important as we head towards a ratification of the RadioEPG specification. The proposal that Ben has presented to you is in response to some real-world prototypes, and understanding use cases better.

To give you an example of why your input is important; in the RadioVIS specification, and as a result of people starting to build implementations, some changes have been made to make it easier to show RadioVIS content as part of a HTML web page. That use case only became obvious once people had started to play about with VIS.

If you can find 30 minutes to work through how you might use content negotiation for XSI files, and report back here, it would be very valuable.

Nick

Chairperson

RadioDNS Project

--
Nick Piggott
Chairperson
RadioDNS Project

http://radiodns.org // @RadioDNS

[Andy Buckingham (RadioDNS)]

Hi Ben, Group,

One thing worthy of at least some partial consideration is the impact
of using HTTP headers for differing the body of the response.
Unfortunately I do not myself hold enough knowledge of my own to
follow this enquiry up, so I'm hoping others can assist.

My understanding is that if a transparent cache between client and
server does not have specific knowledge of the custom headers being
used, it may be unaware that these have any impact on the response
returned and therefore return an incorrect cached response?

For examples Client A and Client B are both communicating with Service
Provider via the same transparent cache. Service Provider will return
a narrower subset of stations (for example, 3 services) in the XSI if
you specify params via the X-Bearer header, instead of a standard
request where all known services (for example, 15 services) are
listed.

Client A makes a request for /XSI.xml with an X-Bearer header and gets
3 services returned in the response. The transparent cache holds the
response keyed against the resource requested /XSI.xml. Client B makes
a request without an X-Bearer header. The transparent cache spots the
same resource request and immediately returns the previously cached
document with only 3 services.

Realistically if this is a genuine problem, I would have thought this
would break many other things beyond our own focus, for example OAuth
prefers header-based auth negotiation. I would assume, therefore, that
most caches take headers in to consideration - but I have little
experience of how production systems deal with this. Do they have a
whitelist of headers to be considered of which we'd need to be
included? Is it of little concern?

Andy.

[Paolo Casagranda]

Dear Ben and colleagues,

some thougths about the proposal, with the intent of better understanding it and trying to secure it against some special use cases.

1. The functionality seems to be very useful to us, specially having a high number of available services. The possibility to address regional content is also very important.

2. Perhaps the technical solution should also be usable by not already tuned in services. I can figure out use cases where only the EPG data are requested (e.g. building a simple programme guide or recommendation list based on user preferences). Is it already supporting this case?

3. Here we have several FM bearers for the same service in the same area. So I can figure that in some cases the X-Bearer value could be not obvious. Maybe taking one bearer in a list of equivalent bearers is enough. In any case, it should be taken into account.

4. Last thing, did you already compare HTTP Headers modification with GET/POST parameters? GET/POST parameters can be safely ignored, and they are well supported by existing sw components and infrastructure. The xml files could be dynamically built using the Bearer value if the service provider supports it; the high level functionality is the same. Is there some technical disadvantage or advantage in using a simple GET/POST?

I hope it helps the discussion

Kind regards

Paolo

[Andy Buckingham (RadioDNS)]

On 26 March 2012 11:07, Paolo Casagranda <crit.r...@gmail.com> wrote:
> 4. Last thing, did you already compare HTTP Headers modification with
> GET/POST parameters? GET/POST parameters can be safely ignored, and they are
> well supported by existing sw components and infrastructure. The xml files
> could be dynamically built using the Bearer value if the service provider
> supports it; the high level functionality is the same. Is there some
> technical disadvantage or advantage in using a simple GET/POST?

HTTP caching is not available for POST requests or GET requests with
query strings, due to their typical implementation for entirely
dynamic responses based on their input. As our intention lies
somewhere between dynamic and static it is preferred to find a method
that works with traditional HTTP caching, but allows us to be somewhat
dynamic in our responses (which then leads back to my other post of
the suitability of custom HTTP headers).

[Paolo Casagranda]

Hi Andy,

of course POST cannot be cached.  Just to better understand, do you think GET has any drawbacks? If the querystring is the same (i.e. same bearer), it can be cached. Of course if the bearer changes, the resource cannot be cached, but shouln't it be the intended behavior?

Kind regards

Paolo

[Andy Buckingham (RadioDNS)]

There are a number of older clients, proxies and caches which refuse
to cache content served with a query string as the contents of the
response is likely to be considered dynamic.

I am unsure if this is still as big a concern today as it was - could
anyone clarify this situation?

[Andy Buckingham (RadioDNS)]

I found this specific reference from Google advising against the use
of query strings:

"Don't include a query string in the URL for static resources. Most
proxies, most notably Squid up through version 3.0, do not cache
resources with a "?" in their URL even if a Cache-control: public
header is present in the response. To enable proxy caching for these
resources, remove query strings from references to static resources,
and instead encode the parameters into the file names themselves."

http://code.google.com/speed/page-speed/docs/caching.html#LeverageProxyCaching

Whilst our content isn't static, we are in a "psuedo-static" state -
we want it to be have like a static resource wherever possible. My
understanding of Google's statement is that you cannot reliably
guarantee content with a query string will be cached between client
and server or, worse still, incorrectly cached. The way to remove
doubt is to avoid the use of query strings.

On 27 March 2012 10:18, Andy Buckingham (RadioDNS)

[Andy Buckingham (RadioDNS)]

Digging a bit further, I think I've found a solution to the custom
header question.

HTTP/1.1 has a 'Vary' header which allows you to define any custom
headers that affected the returned response. This tells caches
downstream that the key used to store the content in the cache must
include the value for the defined headers.

Therefore, if we used X-Bearer, you would send a Vary header in the
response such as...

Vary: X-Bearer

It would be important to look at the HTTP version of the incoming
request too and if it's only HTTP/1.0 never vary the content upon the
header (return all known services). This way old caching layers
mid-stream wouldn't trip up by not understanding the vary header.

Useful discussion here for reference:
http://www.subbu.org/blog/2007/12/vary-header-for-restful-applications

On 27 March 2012 10:30, Andy Buckingham (RadioDNS)

[Ben Poor]

Hello,

Theres some really interesting stuff here - and thanks for digging around the finer points of the HTTP specification Andy.

I'm not as attached to using HTTP headers as I may have seemed, and really a discussion about whether we are returning static content or dynamically selected content is mostly a semantic one. However, probably from an HTTP point of view, we should be using a query string to return the content differently.

So, whilst we *can* form headers in a way that will support our functionality I'm concerned that this goes beyond the complexity level (low) for the lowest common denominator across HTTP clients and devices.

Lets explore querystring a little more as that could be a simpler way to achieve what we're doing.

I'm presuming with that, you can still use If-Modified-Since and Last-Modified to cut down on client-server bandwidth?

Ben

[Paolo Casagranda]

Dear Ben and colleagues,

a few points about the two technical solutions.

New Header ("X-Bearer") solution:

1. with the "Vary" header, it completely solves the problem of caching

2. a web client can easily be implemented using AJAX (however, no simple URL, and attention to cross-domains); server side easily implemented in most cases

Downside: an access to the HTTP Header is necessary :implications on CE devices?; are we sure that old proxies don't mess things up? 

By the way, I've found a (trivial) thing about the name for the first solution. The "X-"  of "X-Bearer" will probably be deprecated, so it may be better "Bearer".

http://tools.ietf.org/html/draft-ietf-appsawg-xdash-04

GET + query strings solution:

1. Client side implementation is trivial, it can be implemented simply modifying the URL

2. Server side easily implemented

Downside: Caching problem with e.g. old squid; "Expiry" and "Cache-control: public" headers could help with caching, however as Andy pointed out, some older caches wouldn't work in any case. 

On the other hand, encoding query parameters in the url, as Google suggests, can be tricky and risky, I think (http://www.mikebrittain.com/blog/2009/06/11/caching-urls-with-query-strings/)

Kind regards

Paolo

[Ben Poor]

Hi Paolo and all,

Based on the feedback and discussions I've had outside the group, I'm minded to suggest we move from using HTTP Headers to using the Querystring. This comes with the following advantages:

* Easier to implement

* Ability to include multiple bearers

My thinking before was to try and allow caching, but we should probably think of a tailored response in this way as being uncacheable as it is represents dynamic information. Clients should not try and cache the document (although obviously they can cache the parsed information from it). 

A Service Provider may choose to respond differently to different clients using the same URL, and may alter that response over time.

I propose we set this implementation to something like the following:

A client can additionally include desired bearer to filter the response on the querystring. One or more querystring parameters bearer can be used, for example:

http://epg.musicradio.com/radiodns/epg/XSI.xml?bearer=fm:ce1.c479.09580&bearer=fm:ce1.c181.10100

The service provider may choose to filter its response based on these parameters, but the client should still parse this and locate the required service(s) by their bearers.

So the service provider can return what it chooses, meaning that in this case, and the case where no querystring parameters are specified the client must still properly parse and examine the response - not assume it fulfils what they require.

That behaviour would have applied however the filtering process would have been implemented - I'm just posting it here for clarity.

Any thoughts?