Request For Comments: Web Service API

[Chris Davenport]

I have pleasure in announcing the publication of the first draft of a web services API specification for the Joomla CMS.  This is an RFC document (a "request for comments") so your feedback is encouraged especially if it includes specific suggestions for improvement.

You can read it here: https://docs.google.com/document/pub?id=1PfJg8ec9PWa8oumIPXbVlO0GY8_8KQoL1Tx47K5blGA

This is the first tangible product of the Web Services Working Group and I'd like to thank all those who have participated in moving this project forward and all those who commented on earlier versions of this document.  You can find more information on the WSWG here: http://docs.joomla.org/Web_Services_Working_Group

Please bear the following in mind:

• This is still a draft.  Please don't think of it as set in stone.  If there's something in it you don't like then please let us know as soon as possible.  If you can suggest improvements that's even better!
• It may still contain errors or inconsistencies.  Please point them out so we can eliminate them.
  
• There are still areas where it is incomplete.  For example, the section describing a possible implementation of the API for Joomla 3.x only covers com_content and com_weblinks.  If you'd like to help out by increasing the coverage then please raise your hand.
• This API specification is independent of any specific Joomla version (except that some sections describe a possible implementation in Joomla 3.x).  I'd certainly like to see this implemented in Joomla 3.x, perhaps only partially in 3.1, but becoming more complete and mature in 3.2 and 3.5.  It might also be implemented as an extension for Joomla 2.5.  Joomla 4.0 is of particular interest and feedback from those who are thinking about "Joomla Next" is especially welcome.
• There is presently no code to back this up.
  

Some areas in need of improvement and questions that need to be answered:-

• Conditional requests should mention strong and weak ETags and the differences in behaviour expected.
• I think most people learn best by looking at examples, so in particular we need more examples of JSON responses, "pretty-printed" for ease of reading.
• Multi-language support needs more thought.
• Bulk update and bulk deletion.  Is it safe?  Can it be made safer?
• Caching.
• Is there a need for a separate "hit" request to increment a hit counter?
• In attempting to follow the HATEOAS principle there are no id numbers in the Joomla 3.x suggested implementation.  Is everybody okay with that?
  

If something isn't clear, please ask.  We want this document to be as readable and understandable as possible, so if there's something you don't understand it's likely my fault for not explaining it properly.  Your questions will help improve future versions.

Thank you.

Chris.

--
Chris Davenport
Joomla Production Leadership Team

[Chris Davenport]

This might be a better link: https://docs.google.com/document/d/1PfJg8ec9PWa8oumIPXbVlO0GY8_8KQoL1Tx47K5blGA/edit

If anyone wants to add comments directly in the document, drop me an email and I'll share it with you.

Chris.

[Andreas Tasch]

Great work guys and gals! Thank you for this additional X-Mas present.

From a first reading everything looks consistend and well thought. The only thing I find a little bit confusing is the PARTIALS part but I will have to read/understand it in more detail to make any comments.

Following the suggestions of apigee/Mulloy is imho a really good idea as they seem to have great experience in RESTful API design and projects.

[Marco Dings]

Thanks very much apreciated and anticipated..

[Joe Palmer]

This is great stuff and exactly what I am interested in!

I'm not sure if people are aware of this project but we may be able to make use of it of get ideas for some of these requirements:

https://github.com/rcorral/com_api

I really like how the API services are implemented through plugins.

[Chris Davenport]

Hi Joe,

Thanks.  I wasn't aware of that project.  I'll add a link to it on the WSWG wiki page.

I think that if someone wants to implement the spec for Joomla 2.5 then it will need to be done as an extension, like the one you pointed to or the one from the TechJoomla guys.  Maybe there are others out there too.  It would be great to see that happen.

But for future Joomla versions people should be thinking about different architectures.  I love Andrew's proposal for making services the core of what the CMS does and while there may be limited scope for architectural change in 3.x, the sky's the limit beyond that.

Joe, if you'd like to help in any way with this project, you are very welcome.

Chris.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/DjX_BoHtCnYJ.

To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

[Gary Mort]

I'm having a bit of difficulty going from the high level overview to a specific implementation.

First off there is the question of "how do you recognize an api request and route it"

So you have a top level URL of:

GET /api

Possibilties to route this include:

A component, com_api, which using JRoute and Menu entrees. maps itself to /api

A plugin which detects the keyword and hijacks the connection.

The CMS itself recognizes API and routes it.

OR, and here is my preference, scrap all the above as hacks and use the the model for JSite, JAdmin, JInstaller and create a new application, JSiteAPI

JSiteAPI can sit in it's own directory, just like JAdmin - appropriately called api so calls to /api automatically load api/index.php

Just like a component can have a frontent or a backend or both, a component can have an api or not.  A module be designated as an API module or a Site module or an Admin module.

Once you do this, you can then leverage every existing Joomla framework - templates, plugins, session, authentication, and menu's.

Menu's solve the entire question about how to configure an api for external and internal use.  You publish your api menu links and use Joomla built in security.

Logon: alias logon, public

List Users: alias users, public

Edit User: alias users/edit, security assigned for only registered users

Save User: alias users/edit/save, security assigned for only registered users

Anonymous API calls will only see logon and users.  

After Logon, the same API link can show logon, users. edit

Using the various menu generation tricks we use today to hide/show menu entrees depending on where in the site someone is, the users/edit/save uri will only be displayed as part of the reply from Edit User...  note: while traditionally one doesn't think of invoking edit ans then save via an API - in terms of the CMS edit provides the function server side to lock the record while it is being edited - thus it is required before a Save can be performed.

I know some people may think the idea of using a template for an API response is silly, however templates have an important role to play with an api - they allow you to include boilerplate legalese - such as:

GET /api/v1/users

pseodo JSON response:

{ 'source':'http://mydomain.com',

'apiuri':'/api/v1/',

'copyright':'copyright 2012 My Domain Company',

'notice':'This information may only be used for programatic interaction with the source website.  This information may not be re-distributed in any manner.',

'commands':['logon':'/logon', 'showUsers':'/users', 'editUser':'/users/edit', 'saveUser':'/users/edit/save'],

'users':['johnsmith':{'username':'johnsmith', 'name':'The Doctor', 'age':587, 'type':'Time Lord'}]

}

All in all, this solves a number of questions such as:

Q:As a site owner, how do I control which of these API actions are accessible? 

A:Edit the ApiMenu item and disable/restrict options as you want

Q:As a site owner, how do I add API actions which are missing? 

A:Edit the ApiMenu item and add an item through the normal means

Q: How do I provide legal protection for my content from being spidered and republished

A: Use a template which provides boilerplate legal notices

Looking over the docs, how version is handled is inconsistent.  In some places version is specified as required, in others it is specified as required.   Is there a difference between version, v, and apiVersion?  I would assume not. Also I really don't like requiring a version number, while it is "good practice" most successful api's do not require a version number and will default to the latest version.  As such, I'd move it from the path to the query parameters by default:

URI = scheme "://" authority "/" path [ "?" query ] [ "#" fragment ]

I'd revise that to

URI = scheme "://" authority "/" path ["?" ["apiVersion="version"&"]  [ query ]][ "#" fragment ]

And document aliases as:

Name: apiVersion

Type: int

Aliases: version, v

Description: Optional in requests; required in responses.  The desired version of the service API in a request or the actual version of the service API in a response.  A service SHOULD switch to using the requested API version if it is able to do so, otherwise it MUST respond with a 400 Bad request status.

Lastly, if it is optional, then placing it before the component entree point is problematic:

GET /api/version1/somestring

Using JRoute with /api as the base uri, the above could be referring to:

API version 1, component com_somestring

component com_version1, task somestring

GET /api/version/sometask

Using JRoute with /api as the base uri, the above should refer to

API application, component com_version, task sometask

But it will require a nasty bit of code to parse the second parameter into 2 parts[version and everything after version] - then test everything after version to ensure that there is something after version and it is an integer with no other charectors..[ie casting test4now as an integer will result in the integer 4, so it must be checked to ensure that no data was lost]

So for version 1 of the API, don't allow embedding the version in the path - suggest it to be in the query - then let the 'market' decide if moving it to the path is preferred and if so, where in the path to place it and how to jiggle it.

IE /v1/api would be easier to handle in JRoute but not as elegant.  /api/v1 would be more elegant and could be handled with either little routing logic or with a little mod_rewrite magic so that if there is no /vX in the uri a default /v0 is added.

/api/v1/users ==> users component

/api/users ~rewite~ /api/v0/users ==> users component

In short, punt on version in the URI until version 1.1 and let the active usage of it determine how it shall be handled.

BTW if it is optional, MY personal preference would be to not use the URI to specify the version at all and instead handle it via HTTP media negotiation. IE

Accept: application/json, application/vnd.joomla+json.v1;q=.1, application/vnd.joomla.v3+json;q=.5, application/vnd.joomla.v2+json;q=1 

Or in english:

Joomla API version 2 encoded as json is preferred[q=1]

Joomla API version 3 encoded as json is acceptable[q=.5]

Joomla API from any version encoded as json if the specific versions are not available[q=.1]

application/json is the fallback if none of the above exist

See http://developer.github.com/v3/media/ for a full fledged example.

It moves all the versioning crud outside the URI and allows for lots of flexibility.  

Convenience fallbacks can be designated to specify versioning information in the Query, the Path, and/or the domain as developers decide what works best for them.

[Chris Davenport]

Hi Gary,

Great feedback.  Thanks.

See inline...

On 28 December 2012 20:13, Gary Mort <jooml...@gary.mort.net> wrote:

I'm having a bit of difficulty going from the high level overview to a specific implementation.

That's probably because the specification deliberately aims to be agnostic about how it is implemented.  Plus, I don't have a clear picture of how it would be implemented myself yet.

 

First off there is the question of "how do you recognize an api request and route it"

So you have a top level URL of:

GET /api

Possibilties to route this include:

A component, com_api, which using JRoute and Menu entrees. maps itself to /api

A plugin which detects the keyword and hijacks the connection.

The CMS itself recognizes API and routes it.

OR, and here is my preference, scrap all the above as hacks and use the the model for JSite, JAdmin, JInstaller and create a new application, JSiteAPI

JSiteAPI can sit in it's own directory, just like JAdmin - appropriately called api so calls to /api automatically load api/index.php

I assume you're talking about an implementation in Joomla 3.x, right?  I tend to agree that adding a separate application would be the preferred approach, although I would tend call it JApi.

One reason for doing that would be to get maximum flexibility in where the API code it sits in the directory structure.  For example, some people might prefer to have the API accessed from a sub-domain (eg. http://api.domain.com), so in that case you'd probably just have the JApi classes in the root directory (plus the libraries and any other dependencies).

I think that a default Joomla installation should have the API code installed in /api/v1, but I also think that the code should be written to work regardless of where it is located.

 

Just like a component can have a frontent or a backend or both, a component can have an api or not.  A module be designated as an API module or a Site module or an Admin module.

Once you do this, you can then leverage every existing Joomla framework - templates, plugins, session, authentication, and menu's.

Agree that it needs to be modular so that access to particular content types can be enabled/disabled as required.  For example, a site that publishes data on, say, city air pollution levels, might never want to provide access via the API to the regular Joomla CMS content types.  How best to actually achieve that, I'm not really sure.

Minor point, but I don't see any reason for the API to have access to or use of session data.  The API should be stateless, so sessions are not needed.

 

Menu's solve the entire question about how to configure an api for external and internal use.  You publish your api menu links and use Joomla built in security.

Logon: alias logon, public

List Users: alias users, public

Edit User: alias users/edit, security assigned for only registered users

Save User: alias users/edit/save, security assigned for only registered users

Anonymous API calls will only see logon and users.  

After Logon, the same API link can show logon, users. edit

Using the various menu generation tricks we use today to hide/show menu entrees depending on where in the site someone is, the users/edit/save uri will only be displayed as part of the reply from Edit User...  note: while traditionally one doesn't think of invoking edit ans then save via an API - in terms of the CMS edit provides the function server side to lock the record while it is being edited - thus it is required before a Save can be performed.

I'm not sure I would want to couple the API to the menu system, but I'm open to argument. ;-)

 

I know some people may think the idea of using a template for an API response is silly, however templates have an important role to play with an api - they allow you to include boilerplate legalese - such as:

GET /api/v1/users

pseodo JSON response:

{ 'source':'http://mydomain.com',

'apiuri':'/api/v1/',

'copyright':'copyright 2012 My Domain Company',

'notice':'This information may only be used for programatic interaction with the source website.  This information may not be re-distributed in any manner.',

'commands':['logon':'/logon', 'showUsers':'/users', 'editUser':'/users/edit', 'saveUser':'/users/edit/save'],

'users':['johnsmith':{'username':'johnsmith', 'name':'The Doctor', 'age':587, 'type':'Time Lord'}]

}

I wouldn't say that using templates is silly, but I think that it may be unnecessarily complex for what is required.

However, I do think that having a separate view that is responsible for transforming the internal data representation into JSON is a good idea.  By allowing that to be overridden it becomes trivially easy to support JSONP and considerably easier to support other output formats, such as XML or CSV, if someone wants to do that.  Adding boilerplate to the JSON would then be done by simply adding a custom override of the view.

 

All in all, this solves a number of questions such as:

Q:As a site owner, how do I control which of these API actions are accessible? 

A:Edit the ApiMenu item and disable/restrict options as you want

Q:As a site owner, how do I add API actions which are missing? 

A:Edit the ApiMenu item and add an item through the normal means

Q: How do I provide legal protection for my content from being spidered and republished

A: Use a template which provides boilerplate legal notices

Looking over the docs, how version is handled is inconsistent.  In some places version is specified as required, in others it is specified as required.   Is there a difference between version, v, and apiVersion?  I would assume not. Also I really don't like requiring a version number, while it is "good practice" most successful api's do not require a version number and will default to the latest version.  As such, I'd move it from the path to the query parameters by default:

You're right, there is an inconsistency there.  Well spotted.

API versioning is one of those topics where everyone has an opinion, opinions are divided and there is no right answer, but let's see if we can clarify things a bit.

There are three places where the API version could be included in requests and two where it might be included in responses.  In the request, it could be in the URL path, in the HTTP headers or in the query part of the URL.  (For POST and PUT it could also be in the request body (as a JSON field), but let's ignore that possibility for now).  In the response, it could be in the HTTP headers or the response body.  In choosing which of these options to support (or indeed none of them!), there is one guiding principle that I think is important: the service should be liberal in what it accepts, but conservative in what it provides.

So, here's my thinking on the subject...

Firstly, I think there is a need for clients to be able to communicate which versions they are expecting or can support to servers.  At some point we will want to release a v2 API which will undoubtedly break something somewhere and we know from experience that clients and servers will be at different points in their upgrade cycles.  We need to be able to handle that situation gracefully.

The server should not have to parse the URL base path in order to determine which version the client is asking for, so I think having the version number in the URL path is purely for human consumption.  But I happen to think it is desirable (and the Apigee folks recommend it too) for a couple of reasons.  Firstly, a change big enough to warrant a v2 release is likely going to need new code and being able to drop that new code into a separate sub-directory while continuing to support the older API is a clean and simple way to do that.  Secondly, being able to paste a URL into a regular web browser to check the response is something developers do all the time, but it's harder if the API needs specially-crafted headers or a change in the URL in order to produce the same behaviour.  And thirdly, it increases the visibility of the version to developers, making it just a little more likely that stuff will be upgraded over time.

Using HTTP headers is the "correct" way according to the RESTafarians, but I believe some clients can have difficulty adding HTTP headers to their requests so I think there needs to be a fallback mechanism.  I would suggest that adding a v=1, or version=1 or apiVersion=1 (take your pick) into the query part of a request would allow the Accept header version to be overridden.

In cases where the version in the URL path conflicts with the header/query-part version then if the API is capable of supporting the version requested in the header/query-part it should do a 301 redirect to the URL with matching path version; otherwise give a 404 Not found.  There would have to be some configuration variable somewhere that would specify what to do if the versions don't match.

I think that scheme would give enough flexibility to allow clients and servers to be upgraded independently while continuing to function as expected during the transition.

 

URI = scheme "://" authority "/" path [ "?" query ] [ "#" fragment ]

I'd revise that to

URI = scheme "://" authority "/" path ["?" ["apiVersion="version"&"]  [ query ]][ "#" fragment ]

And document aliases as:

Name: apiVersion

Type: int

Aliases: version, v

Description: Optional in requests; required in responses.  The desired version of the service API in a request or the actual version of the service API in a response.  A service SHOULD switch to using the requested API version if it is able to do so, otherwise it MUST respond with a 400 Bad request status.

Lastly, if it is optional, then placing it before the component entree point is problematic:

GET /api/version1/somestring

Using JRoute with /api as the base uri, the above could be referring to:

API version 1, component com_somestring

component com_version1, task somestring

GET /api/version/sometask

Using JRoute with /api as the base uri, the above should refer to

API application, component com_version, task sometask

But it will require a nasty bit of code to parse the second parameter into 2 parts[version and everything after version] - then test everything after version to ensure that there is something after version and it is an integer with no other charectors..[ie casting test4now as an integer will result in the integer 4, so it must be checked to ensure that no data was lost]

If JRoute can't handle it then I think I'd probably just not use it and write a custom router for the API instead.  Remember though that I'm suggesting that the default base path for the API would be /api/v1 so the router wouldn't need to be concerned with parsing the version number.  ie. the entry point for API code execution would be /api/v1/index.php.

 

So for version 1 of the API, don't allow embedding the version in the path - suggest it to be in the query - then let the 'market' decide if moving it to the path is preferred and if so, where in the path to place it and how to jiggle it.

IE /v1/api would be easier to handle in JRoute but not as elegant.  /api/v1 would be more elegant and could be handled with either little routing logic or with a little mod_rewrite magic so that if there is no /vX in the uri a default /v0 is added.

/api/v1/users ==> users component

/api/users ~rewite~ /api/v0/users ==> users component

In short, punt on version in the URI until version 1.1 and let the active usage of it determine how it shall be handled.

BTW if it is optional, MY personal preference would be to not use the URI to specify the version at all and instead handle it via HTTP media negotiation. IE

Accept: application/json, application/vnd.joomla+json.v1;q=.1, application/vnd.joomla.v3+json;q=.5, application/vnd.joomla.v2+json;q=1 

Or in english:

Joomla API version 2 encoded as json is preferred[q=1]

Joomla API version 3 encoded as json is acceptable[q=.5]

Joomla API from any version encoded as json if the specific versions are not available[q=.1]

application/json is the fallback if none of the above exist

See http://developer.github.com/v3/media/ for a full fledged example.

It moves all the versioning crud outside the URI and allows for lots of flexibility.  

Convenience fallbacks can be designated to specify versioning information in the Query, the Path, and/or the domain as developers decide what works best for them.

Broadly speaking I think you're right, although I still think that including the major version number of the API in the base path is a good idea.

Thanks again for the excellent feedback.  I'll try to update the document with your suggestions soon.

Chris.

 

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/UGws5mfnIDcJ.

To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

[Gary Mort]

On Saturday, December 29, 2012 3:04:59 PM UTC-5, Chris Davenport wrote:

First off there is the question of "how do you recognize an api request and route it"

So you have a top level URL of:

GET /api

I assume you're talking about an implementation in Joomla 3.x, right?  I tend to agree that adding a separate application would be the preferred approach, although I would tend call it JApi.

Or 2.5 for that matter.  3rd party developers are free to create whatever extensions they want to add this functionality for early 2013.   By having the overall spec published it makes it really convenient for 4th party developers to write apps without having to struggle nearly as much.

Ie if 3rd parties take this spec and run with it, then others could write desktop application to edit content with a  degree of confidence that when there is an "official" API access in Joomla the tools written will mostly work with that version as well.

 

 

Just like a component can have a frontent or a backend or both, a component can have an api or not.  A module be designated as an API module or a Site module or an Admin module.

Once you do this, you can then leverage every existing Joomla framework - templates, plugins, session, authentication, and menu's.

Agree that it needs to be modular so that access to particular content types can be enabled/disabled as required.  For example, a site that publishes data on, say, city air pollution levels, might never want to provide access via the API to the regular Joomla CMS content types.  How best to actually achieve that, I'm not really sure.

There are 2 ways of soliciting input for storage in Joomla![really large brush strokes].

A component can have it's own custom interface which is displayed within the Joomla framework.  If they want their application to be API accessible, it's up to them to provide the additional hooks for it.

A plugin can at various points insert extra fields into the form the component is going to use.  The User Profile plugin does this - adding new fields dynamically.  To support that would require creating something like JForm for api usage - and does not have to be in the initial implementation.

 

Minor point, but I don't see any reason for the API to have access to or use of session data.  The API should be stateless, so sessions are not needed.

JSession is used much more extensively than for tracking user data across sessions.  There are many places which call use it as a sort of short term cache - for example a plugin that runs at afterRoute and at beforeRender may need to pass some data to itself later on.  I've seen it shoved into the session and deleted later.  One can default to an in memory only session and kill the session cookie.

While rest is "stateless" - that does not mean it is "anonymous" - which means the website has to provide some means of authentication to API clients.  This could be SSL certificates, an API key, or having a short term API key generated automatically via some logon process.

The key here is, an API key of any sort IS a session - it may be a permanent session or a short term session, but it is a proof of authorization id which is all a session really is.  So rather than create all new security mechanisms and then having to secure them - a 3rd party can use the services built in to Joomla if they wish.

 

 

Menu's solve the entire question about how to configure an api for external and internal use.  You publish your api menu links and use Joomla built in security.

I'm not sure I would want to couple the API to the menu system, but I'm open to argument. ;-)

I hate the Menu system for a lot of reasons - however creating a completely new system clouds the issue for end users IE for your website owner who wants to allow API access to some sub-component.  Right now they know that they can add a menu item for a component, assign it some permissions, and everything works.  Why should they waste the brain cells in learning 2 different ways of doing things?  [Note, I'm ALL for improving the menu system at the same time. I'm just for limiting complexity.  

  

However, I do think that having a separate view that is responsible for transforming the internal data representation into JSON is a good idea.  By allowing that to be overridden it becomes trivially easy to support JSONP and considerably easier to support other output formats, such as XML or CSV, if someone wants to do that.  Adding boilerplate to the JSON would then be done by simply adding a custom override of the view.

View, template, whatever.  I view that as all semantics.  "A system that provides site owners the ability to override the output of a component without having to edit core code".

 

The server should not have to parse the URL base path in order to determine which version the client is asking for, so I think having the version number in the URL path is purely for human consumption.  But I happen to think it is desirable (and the Apigee folks recommend it too) for a couple of reasons.  Firstly, a change big enough to warrant a v2 release is likely going to need new code and being able to drop that new code into a separate sub-directory while continuing to support the older API is a clean and simple way to do that.  Secondly, being able to paste a URL into a regular web browser to check the response is something developers do all the time, but it's harder if the API needs specially-crafted headers or a change in the URL in order to produce the same behaviour.  And thirdly, it increases the visibility of the version to developers, making it just a little more likely that stuff will be upgraded over time.

Using HTTP headers is the "correct" way according to the RESTafarians, but I believe some clients can have difficulty adding HTTP headers to their requests so I think there needs to be a fallback mechanism.  I would suggest that adding a v=1, or version=1 or apiVersion=1 (take your pick) into the query part of a request would allow the Accept header version to be overridden.

 

My only objection was to specify it as MUST be part of the path.  Moving the requirement to the query makes me happy - and making it optional there.  Also allowing the server to accept a version in any one of a number of other formats as the implementer wishes is good - without getting bogged down in too many details[see following...and I think I'm including too many details already so shortening it would be good]

=========================

Clients MAY specify a specific API version they can accept by specifying it as a query parameter.  

Servers MUST check for the presence of the version tag in the query parameters.  

If the client species a version number, Servers MUST compare that version to their own using any comparison utility which complies which PHP Standardized version comparison.

http://us1.php.net/manual/en/function.version-compare.php

If the api versions the server supports are all less then the version specified by the client the server MUST [reject request verbage]

If the versions the server supports include the version specified by the client the server MUST reply using that version.

If the versions the server supports are greater than the version specified by the client the server SHOULD redirect the client with a 303 redirect to a version specific uri for the response.  

Servers MUST accept both the version and v query tags for specifying version.  If both are given, version takes precedence over v.

Servers MAY embed the version information in the path.  If it is embedded in the path it MUST be embedded as vNN, where NN is the version number.

Servers MAY provide any other mechanisms they wish for specifying version[domain name, HTTP header, etc].  

If a Server accepts multiple version specification formats, if multiple client specifications are given in different formats, the query parameter version MUST be honored if set.   If not set, then the query parameter v MUST be honored if set.  

If neither official query parameter is used, then order of precedence shall always be a Query Parameter has precedence over all others.  A path parameter, if supported, has precedence over any non-Query Parameter.

To me, that is sufficiently exacting while allowing for creativity and invention.  Supporting both version and v because coders are lazy and will object to having to type 6 extra characters while end users dislike obtuse labels like "v".

It deals with the server MUST always honor the Client's wishes - but is allowed to substitute uplevel versions.  It insists that if the Server fails to honor the clients wishes it will at least tell the client.  It suggests proper procedure, without getting too bogged down in enforcing it.

It also defines version in terms of a readily available PHP function...not the best version system but guaranteed to be available.

[Chris Davenport]

All good points.

The only thing I would add is that when targetting 2.5 modifying core code is not really an option, so I think it would need to be done as a separate application and/or some package of downloadable extensions.  With 3.x there is more flexibility because core files can be changed as long as backwards compatibility is maintained.  So you might end up with different approaches for the different versions.  CMS 4.0 can and should, I believe, have a service-centric architecture, which is a radical change, but a well-thought out and well-specified web API should make migration really easy (famous last words?).

I just want to add that it is not my intention to get involved in coding this stuff, so I remain pretty neutral as to how people want to actually make it work.  If I do any work on this at all, beyond editing the specification, it will most likely be on testing.  I want to put most of my effort into improving various aspects of smart search from now on.  Not that I'm not interested, but I simply don't have the time and I need to prioritise.

So please, coders out there, don't hold back.  Let's see more ideas about how to make this work.

Chris.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/Rf0vupkZtsgJ.

To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

[Alonzo Turner]

Thanks for all your hard work on this very well thought out document. I have only one concern that jumped out at me. The requirement for ETags may run up against some known PCI compliance issues. I think that the web services architecture could be very useful to a number of e-commerce solutions, but not at the expense of PCI compliance which is mandatory for credit card transactions in the United States.

I see that you've already pointed out that there is some need for greater clarification regarding the use of ETags. I humbly ask that you consider PCI compliance when sorting out the details. Thanks again so much, this looks really great.

[Chris Davenport]

As far as I can tell there is no reference to ETags in the PCI DSS, nor would I expect there to be as I can't really see how they would have any impact on security if used properly.

However, it appears that there was a bug in Apache, long since fixed, which allowed attackers to infer something about the server from the ETags it was emitting and some PCI compliance companies appear to have decided that ETags are therefore a "bad thing".  In fact they are an important part of web caching and concurrency control so they are really a "good thing" and should not be discarded lightly.

But in general, whilst using HTTP headers to convey information in both requests and responses is the way it is *supposed* to be done, there appears to be a relatively small number of cases where either clients or servers are unable (or unwilling) to handle headers properly.  I think this is one such instance.

So, as regards the web services API specification, I think we should encourage use of HTTP headers in the correct way as per the relevant RFCs, but provide an override mechanism so that the service will still function correctly even if the client or server is unable to amend its headers.  This can be done by specifying optional additional arguments in the query part of URIs and/or as top-level metadata properties in the message body, that will override the equivalent headers.  It's then up to implementations as to how they want to allow the user/developer to enable/disable these overrides.

I'll run through the specification and make sure that we have that covered.

Thanks for bringing this issue to my attention.  I would not have been aware of it otherwise.

Chris.

On 3 January 2013 01:57, Alonzo Turner <alonzo...@subtextproductions.com> wrote:

Thanks for all your hard work on this very well thought out document. I have only one concern that jumped out at me. The requirement for ETags may run up against some known PCI compliance issues. I think that the web services architecture could be very useful to a number of e-commerce solutions, but not at the expense of PCI compliance which is mandatory for credit card transactions in the United States.

I see that you've already pointed out that there is some need for greater clarification regarding the use of ETags. I humbly ask that you consider PCI compliance when sorting out the details. Thanks again so much, this looks really great.

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/vnpuAmz3nKUJ.

To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

[Alonzo Turner]

Thanks for taking a look at this issue Chris. I agree that this is somewhat of a bogus security issue. The objection raised by PCI security scanning services is that the default Apache ETag header exposed the inode numbers for files on the server. In theory this could be remedied by altering the default ETag behavior. However in practice, I have found that many admins simply disable ETags altogether because that gets you the green approved stamp from the PCI scan service. Sadly many corporate IT managers aren't interested in the details regarding why this isn't really a security problem, and that ETags are a proper and helpful part of HTTP headers. They pay a lot of money to the scan service, and if it shows up on a report, it must be a problem.

[Andreas Tasch]

I just stumbled over an interesting talk about RESTful APIs by David Zuelke (@dzuelke) on ApiDays. Very interesting insights into api design and best practices. Especially on the part of API VERSIONING: he suggests to NOT use versions in the path (e.g. v1, v2, ...) as "different URLs mean different ressources" and because of bookmarks to ressources, proxies. See the slides here:

https://speakerdeck.com/dzuelke/designing-restful-web-services-apidays2012-2012-12-04?slide=79

and talk here:

http://youtu.be/BNrGkFq6M9U?t=26m8s

For me, very plausible points made, what do you think?

He also has some interesting other points (forums, cms apis!), so the whole talk is very informative and insightful, e.g. XML vs. JSON

PS:

apidays on youtube has a lot of other talks:

https://www.youtube.com/user/apidays?feature=watch

[Olivier Nolbert]

There's a webcast session today about API design :

http://forms.apigee.com/acton/form/549/0064:d-0005/0/index.htm

[Andreas Tasch]

Digging deeper into this I really think we should dicuss the MESSAGE FORMAT again, maybe JSON is not the way to go? (I really thought JSON is XML but better and easier, but it seems that it is not):

As we have linked ressources we already abstracted the workflow and a client only needs to know the entry point. We can link them to a custom joomla media type (e.g. vnd.joomla.article+xml):

https://www.youtube.com/watch?v=XzgCzjMdvRE

Further explanation difference XML/JSON, please listen especially around 0:40:00

http://youtu.be/XzgCzjMdvRE?t=39m3s

What I'm concerned about is if we stick with JSON and want to add an attribute to a resource the whole JSON structure is changed an will break all consuming clients? So for the long run XML may be much easier to maintain and extend?

see:

https://speakerdeck.com/dzuelke/designing-restful-web-services-apidays2012-2012-12-04?slide=63

https://speakerdeck.com/dzuelke/designing-restful-web-services-apidays2012-2012-12-04?slide=64

[Andreas Tasch]

One strong point for JSON tough is its easier parsing by javascript and this is probably the no. 1 use case for the api right now. It seems that XML -> JSON conversion is easier as the other way around - so an option may be a XML core with JSON parser to support both types. 

The apigee guys on the webcast mentioned above think that JSON should be first choice and XML only if needed - but did not further explain when it actually makes sense. Dunno, you?

[Gary Mort]

On Thursday, January 10, 2013 1:42:56 PM UTC-5, Andreas Tasch wrote:

What I'm concerned about is if we stick with JSON and want to add an attribute to a resource the whole JSON structure is changed an will break all consuming clients? So for the long run XML may be much easier to maintain and extend?

Adding an attribute in JSON will no more or less "break" consumers then changing XML.

If a consumer is setup to only accept a certain set of attributes, it breaks.  If not, it doesn't.  This is true of xml and json.

XML is only easier when/if you include maintaining the DTD files associated with the XML so that xml data can be verified.  As this is not done today with most of the XML in Joomla! [Installation manifest files, Template files, and JForm fields] there is no reason to expect volunteers to create and maintain DTD's for the API.   

The primary breakage is actually for the reverse, consumers to the server.  If the server changes to require a specific field, then the consumers "break"...but again this is true for JSON and XML.  The primary difference is that if your maintaining the DTD files, when you change the requirements it's all documented in your DTD so consumers can report why they can't function - while with JSON it's all guesswork based on what the server sends back for a reply.

[Ian]

There's nothing that says you can't define your own json specification language to do the same thing with json.  I've built a services testing framework that can be used to verify the format of service requests, and this includes a way to define what the output of a service should look like.  I've also seen discussions of other similar efforts (like http://nico.vahlas.eu/2010/04/23/json-schema-specifying-and-validating-json-data-structure and others I unfortunately didn't bookmark).  DTDs don't write themselves and the effort put into writing XML DTDs could just as easily be put into writing json dtds.

Ian

[Tiago Espinha]

On Monday, December 24, 2012 1:57:33 AM UTC+1, Chris Davenport wrote:
> I have pleasure in announcing the publication of the first draft of a web services API specification for the Joomla CMS.  This is an RFC document (a "request for comments") so your feedback is encouraged especially if it includes specific suggestions for improvement.
>
>
> You can read it here: https://docs.google.com/document/pub?id=1PfJg8ec9PWa8oumIPXbVlO0GY8_8KQoL1Tx47K5blGA
>
> This is the first tangible product of the Web Services Working Group and I'd like to thank all those who have participated in moving this project forward and all those who commented on earlier versions of this document.  You can find more information on the WSWG here: http://docs.joomla.org/Web_Services_Working_Group
>
>
> Please bear the following in mind:
> This is still a draft.  Please don't think of it as set in stone.  If there's something in it you don't like then please let us know as soon as possible.  If you can suggest improvements that's even better!
> It may still contain errors or inconsistencies.  Please point them out so we can eliminate them.
> There are still areas where it is incomplete.  For example, the section describing a possible implementation of the API for Joomla 3.x only covers com_content and com_weblinks.  If you'd like to help out by increasing the coverage then please raise your hand.
> This API specification is independent of any specific Joomla version (except that some sections describe a possible implementation in Joomla 3.x).  I'd certainly like to see this implemented in Joomla 3.x, perhaps only partially in 3.1, but becoming more complete and mature in 3.2 and 3.5.  It might also be implemented as an extension for Joomla 2.5.  Joomla 4.0 is of particular interest and feedback from those who are thinking about "Joomla Next" is especially welcome.
> There is presently no code to back this up.
> Some areas in need of improvement and questions that need to be answered:-
> Conditional requests should mention strong and weak ETags and the differences in behaviour expected.

> I think most people learn best by looking at examples, so in particular we need more examples of JSON responses, "pretty-printed" for ease of reading.Multi-language support needs more thought.
> Bulk update and bulk deletion.  Is it safe?  Can it be made safer?Caching.Is there a need for a separate "hit" request to increment a hit counter?In attempting to follow the HATEOAS principle there are no id numbers in the Joomla 3.x suggested implementation.  Is everybody okay with that?

>
> If something isn't clear, please ask.  We want this document to be as readable and understandable as possible, so if there's something you don't understand it's likely my fault for not explaining it properly.  Your questions will help improve future versions.
>
>
>
> Thank you.
>
> Chris.
>
>
> --
> Chris Davenport
> Joomla Production Leadership Team

Hi everyone,

I'm a Ph.D. student in the Netherlands currently doing research in web API maintenance and evolution and I was wondering whether this would be the right group to ask the web API developers of Joomla some questions about the Joomla web API.

Also as far as I understand, the Joomla web API is still being planned right? Is there already a feature-complete version out there?

Best,
Tiago

[Chris Davenport]

You can see the current status here: http://docs.joomla.org/Web_Services_Working_Group

I've just started working on a 3rd draft of the specification.  Still a long way off feature-complete code yet.

Chris.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

To post to this group, send an email to joomla-...@googlegroups.com.

Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Davenport]

Hi,

On 3 November 2013 20:52, mc007 <baumgart...@gmail.com> wrote:

Hi,

I tried your 'prove' of concept code. 

- Its nice but quite slow as it loads too many dependencies which are not needed for a RPC call. In a mobile application and ultra good server it still caused over 1800ms to receive the result. It mus be done under 300ms otherwise its useless or better said disappointing for users.

It's a proof-of-concept and is a long way from being production quality code.  It should definitely be built on the Framework, which wasn't available when I first sat down to write it, which should give it a speed boost.  I'm sure there are plenty of other performance optimisations that could be done too.  It wasn't built to be fast; it was built to show what could be done.

 

- "prevent caching' is missing 

So is POST/PUT/DELETE support, authentication, filtering, search, partial requests and a dozen other things that would be needed for a production web services implementation.  I never claimed it was complete.  It's just a proof-of-concept, meaning it's something to look at and consider if it might be done that way.

 

- I also miss some 'service' introspection. So its easier to adopt JS clients to the services without reading all the docs. This is how exposed PHP classes will enumerate their methods  :  www.pearls-media.com/joomla25/components/com_xas/xapp/index.php. This implementation is based on Dojo's SMD proposal, widely used in high-tec apps from IBM. You can also switch transport to JSONP by appending &callback=bla.

Enumeration of methods should not be necessary as that would imply tight coupling between client and server.  The idea is to use the HATEOAS constraint and have clients navigate through the application state space by following links in hypermedia responses.  This encourages loose coupling between clients and servers so that servers (in particular) can be updated with much less risk of breaking clients.  This is all part of the REST architectural style.

 

- More transport layers  should be in the boat  : JSON-RPC via POST for big calls, JSONP for small calls.

We're not trying to do RPC; we're trying to do REST.  That's a very different style of interaction and it fits with the web's own architectural style, making use of HTTP as it was intended rather than just regarding it as something to tunnel through.

JSONP should be supported and is trivial to implement on top of the existing JSON support.  That is noted in the WSWG specification.

 

You can download our RPC server implementation here : http://www.pearls-media.com/com_xas_1.1.zip (site/xapp/lib/rpc). Its the best JSON-RPC implementation for PHP around :-) the service entry point is setup in site/xapp/index.php.

If you consider migrating our RPC code, let us know. We're very glad to help as we're waiting for a proper RPC implementation for Joomla since 2005.

Thanks, don't hesitate to contact me. I wanna see it rocking, not crouching :-)

Thank you, but I'm personally just not interested in an RPC implementation.  There may be situations where RPC might make sense, but by and large the world has moved on and RPC is no longer seen as the way forward for web applications like Joomla.  Read Roy Fielding's dissertation and you will understand why I say that.

Chris.

[mc007]

Thats really great, just today our component got unpublished from the JED after posting last weekend here the link, with no real indication for the why.  If that is the response of Joomla/You to kickout willing and REAL and unbiased opensource minded projects and developers, I really doubt that we ever go on with Joomla at all. 

I take this personal, yes, because I worked for many years (8!) my ass off, not earning any cent of Joomla customers except custom development (equals still almost zero). My goal was a freemium and hidden strings free mobile solution for their Joomla and other systems. Also integrating now-days and modern back-end apps to solve complex situations with ease, free and without commercial interests

Thanks you all, this time you made it all clear. Greets from a Mambo-Fan/Dev

being pissed,

Guenter Baumgart

[Bakual]

I don't think it's related to this thread. It wouldn't make sense to me. Did you contact JED and asked them for the reason? Usually you should get an email explaining why it was unpublished.

[Nick Savov]

Yes, it's not related to this topic. Please double check your JED
submission for a detailed reason. Once the issue is corrected, you'll be
able to resubmit your extension.

Kind regards,
Nick

>
> I don't think it's related to this thread. It wouldn't make sense to me.
> Did you contact JED and asked them for the reason? Usually you should get
> an email explaining why it was unpublished.
>

> --

[Tiago Espinha]

Hi Chris,

I've had a look at the link you posted and there's a few other questions that perhaps you can help me with regarding the development of the Joomla web API.

Since Joomla is at a fairly early stage in the development of its web API, it is a very interesting case study for academic purposes. Because of this, there's a few questions that popped up, for instance:

• Has it ever been a concern to develop the Joomla web API using a "design for change" approach? By this I mean, being conscious that your web API will have to change in the future and therefore you took measures to minimize the impact it will have on clients.
• Are you are of the "semantic versioning" way of versioning software? Is this something you are using/will use in the Joomla web API? This semantic versioning also means that breaking changes (i.e. backwards incompatible) should be left for major releases (X.0.0) and minor and patch (0.X.0 and 0.0.X) should be backwards compatible. Is this a concern for the Joomla web API developers?

If you could have a go at answering these for me, I will most definitely be using your insights for my academic research!

Best,

Tiago

[Chris Davenport]

Hi Tiago,

On 8 January 2014 11:00, Tiago Espinha <ti...@espinhas.net> wrote:

Hi Chris,

I've had a look at the link you posted and there's a few other questions that perhaps you can help me with regarding the development of the Joomla web API.

Since Joomla is at a fairly early stage in the development of its web API, it is a very interesting case study for academic purposes. Because of this, there's a few questions that popped up, for instance:

• Has it ever been a concern to develop the Joomla web API using a "design for change" approach? By this I mean, being conscious that your web API will have to change in the future and therefore you took measures to minimize the impact it will have on clients.

Very definitely.  We know from our experience with developing the Joomla CMS over the past years that managing change is one of the biggest challenges that we face as a project and this has been very much in our minds during the design process.  Indeed one of the motivations for developing a "web" API in the first place was to allow much better, cleaner decoupling of producers and consumers that would allow greater freedom for both to evolve independently over time. That's why I see the HATEOAS principle as being hugely important, for example.

 

• Are you are of the "semantic versioning" way of versioning software? Is this something you are using/will use in the Joomla web API? This semantic versioning also means that breaking changes (i.e. backwards incompatible) should be left for major releases (X.0.0) and minor and patch (0.X.0 and 0.0.X) should be backwards compatible. Is this a concern for the Joomla web API developers?

I think that Joomla in general will gradually move closer to SemVer over time even though for the web API specifically it is probably more fine-grained than we would need.  The producers and consumers of an API are expected to be automated agents rather than humans and the only kind of change that is important to a bot is a breaking change.  So I suspect that the only version number that the API needs to report will be the major number.  I would expect the minor and patch numbers, if they are exposed at all, to be ignored.

I think it's important to distinguish the specification from the software that implements the specification.  SemVer can only work if there is a clear "public contract" that defines what each party can expect to remain unchanged when new versions are released.  The specification can be thought of as that public contract.  Provided that API producers and consumers comply with that contract then each should be able to evolve independently without breaking the other party (up to a major release anyway).  The specification itself should change only very slowly and I would hope that each version would survive across several major releases of the CMS.

I like to keep in mind the different frequencies of change within the system and use the "encapsulate what changes" principle so that elements that are likely to change rapidly are isolated from those that change only slowly.  At the level of the generic web API, which is the primary subject of the specification, change is expected to be relatively slow. However, at the level of individual Joomla components, change will be more rapid as, for example, new fields are added or removed from resources, or entirely new data structures and state transitions are implemented.  My current thinking is that versioning at that level should be the subject of "profiles" that will use the full SemVer triad (although bots will probably still just ignore anything other than the major number).

You might reasonably ask why we are writing a detailed specification.  For most projects the API producer *is* the specification and merely needs to be documented rather than specified.  However, Joomla is a highly modular CMS with a thriving community of independent third-party extension developers.  A typical Joomla installation will consist of a mix of core and third-party components.  In order to expose a consistent API with such an installation it is important that all the components "play by the same rules".  It would be difficult to write an API client if it needed to take into consideration all the different ways in which each developer might decide to implement their part of the overall API.  The specification is as much a contract for core and extension developers to adhere to as it for client developers to use.

 

If you could have a go at answering these for me, I will most definitely be using your insights for my academic research!

I hope that helps.  Please feel free to ask any further questions you may have.

Best of luck with your research.

Chris.