Web Services Working Group update

[Chris Davenport]

This is an update following the inaugural meeting of the Web Services Working Group which took place at J and Beyond 2012 recently and which was attended by a group of people with a common interest in implementing web services in Joomla.  Of course, not everyone with an interest in the subject will have been able to make it to Germany, so please read this summary and let us know if you would like to help out in any way.

The objective is to enable pretty much any action that is possible via the user interface to be carried out via a web API, subject to appropriate authorisation.  So, to be clear, the aim of this working group is to allow Joomla to be a provider of web services, rather than a consumer, with a typical use-case being to allow mobile devices to access or even manage resources on a Joomla site.

It was generally agreed that we need to implement a REST interface in Joomla although it would be nice if it could be coded to be sufficiently generic so as to allow other API styles to be implemented too, either now or in the future.

One idea that was discussed was to define a standardised way of getting data from the model so that JController could be extended to provide a REST interface without having to write any additional code in each of the components.  This is something that might be possible for the core components but it would not be practical to enforce it.  So it was suggested that we might have a standardised way of getting data from the model for core components which can be overridden by extensions if they want to do it differently.  The feasibility of this approach needs further consideration.

So, the first task of this working group will be to define an architecture that can be used as the basis for building the API.  This will go hand-in-hand with defining a standardised API syntax that can be applied to core components as well as being made available to third-party extensions.

This is expected to be a medium to long-term project and we will most likely not have any code ready to go in to Joomla 3.0.  However, development can easily proceed in parallel during the 3.x cycle.

Right now we need someone with time on their hands to step forward and take on the role of coordinator for this working group.  As I said at the meeting, I am happy to continue being the PLT contact, but I am spread far too thin to take on the coordinator role.

For the time being this dev cms mailing list will be the primary channel of communication.  If the volume of traffic becomes such that it might be deemed to be a nuisance by those not interested in web services, then we will most likely start a separate mailing list.

Some code and other material to look at and consider:

com_api on Github (https://github.com/techjoomla/com_api)
Aaron Schmitz has done some work on an OAuth 2.0 library here: https://github.com/aaronschmitz/joomla-platform/blob/b2b68507006e4e575858e1d103a75f2a56907215/libraries/joomla/oauth/oauth.php
Recommended reading: Web API Design: http://offers.apigee.com/api-design-ebook-rr/ (WARNING: Requires registration to download).
http://docs.joomla.org/Xml-rpc

Since the meeting it has come to my attention that there is a GSoC student project looking at web services.  Although it appears to have a much more limited scope, there may be opportunities in the overlap.  See http://magazine.joomla.org/issues/Issue-June-2012/item/772-RESTful-Web-Service-API

Finally, a big thank you to everyone who came along to our meeting in Germany and who provided many insights into providing a web services API in Joomla.

Chris.

--
Chris Davenport
Joomla Leadership Team - Production Working Group
Joomla Documentation Coordinator

[Herman Peeren]

Okay, I'll do some coordination work for the Web Services Working Group. I didn't step forward earlier because I haven't got much time on my hands and have too many unfinished projects. But this is something that has to be done, I need it myself, I had allready promised to participate in this working group, I have invested some time in the subject before and it should not wait for another 2 years... See my presentation at jab10: http://www.slideshare.net/HermanPeeren/webservices-connecting-joomla-with-other-programs-4399412

Unfortunately I was not at jab12. Thanks for the update.The GSoC-REST-API-project is indeed related and worth watching. Also a lot can be learned from the way Nooku implemented a RESTful interface.

Who are in this working group now? Besides myself I know of Parth and Ashwin from Techjoomla. Others?
Let's see if we can make a plan to get this done.

Ciao,
Herman Peeren,
Yepr, Rotterdam

[Ashwin]

Thanks a lot for the summary Chris! And I'm ok with co-ordinating the group as well!

Herman, we had a decent turnout at the working group meeting, there were about 5-6 heads and everyone of us looked very interested. Off my head I can remember Achim Fischer and Jisse from Yireo who are interested. 

The most important thing right now is definitely drawing up some architectural plans for the API. I've written to Andrew Eddie who is mentoring the GSoC project on some ideas that he has on the architecture. I think once we hear back from him, we could have a Skype meeting sometime next week to identify the possible architectures.

Meanwhile I am going to contact and contact the other people who attended the working group meet and point them to this thread, so that they can start contributing.
@Chris - Do you have the names/emails of the guys who were present for the meeting ?


Thanks
Ashwin

[Herman Peeren]

Hey Ashwin,

That's great if you want to do the co-ordinating of this workgroup! As said, I have not enough time and was only volunteering in case no one else would step forward. Now I will concentrate on architecture and implementation. I'll list my own ideas on the subject soon!

I had contact via e-mail with Jisse this morning: tomorrow he is leaving for a 7 weeks trip by bike to Spain.

I'm very glad this workgroup really starts now. Let's connect Joomla!.

Ciao,
Herman

[Herman Peeren]

As an appetizer, here is some exercise-code for tomorrows workshop on webservices for the Dutch PHP Conference 2012:
https://bitbucket.org/lornajane/web-services-tutorial-code

And this is the GitHub for the GSoC-REST-project: https://github.com/stefanneculai/Web-service-API

[Ashwin]

Hi Herman,

Also, not sure if you have seen this - https://github.com/techjoomla/com_api

This is a much better implementation that the one I spoke about at last year's JAB, do give it a shot.

- Ashwin

[Ashwin]

And, we missed your giant boots this JAB!

[Herman Peeren]

On 6-6-2012 16:46, Ashwin wrote:

Also, not sure if you have seen this - https://github.com/techjoomla/com_api

This is a much better implementation that the one I spoke about at last year's JAB, do give it a shot.

I'll for sure look at it! Had indeed not looked again after last year. Thanks! Let you know what I think.
Herman

--

[Andrew Eddie]

Awesome stuff.  Regarding Stefan's GSOC project, the work can be found here:

https://github.com/stefanneculai/Web-service-API/

The premise of this is to create a standalone platform application on which to design the web services.  It's based on what we are working on internally at eBay to service our social data sharing CMS.  Stefan can tell you more about it himself :)

Louis has also put a lot of working into the oAuth connector.  See here:

https://github.com/LouisLandry/joomla-platform/tree/oauth

In terms of web services there are two parts that I see for the project.  The first part is establishing the base architecture for anyone to design a web services platform for any project.  This is basically what Stefan's project and this is also the part that I'd be very interested in being involved with.

The second part, or possibly it's a second layer, is converting Joomla the CMS to a web services architecture.  Now, I would advise that you don't just think of duplicating CMS functionality through web services.  You need to think of the web services as the very core of the data Joomla itself, and that the CMS is just another client for that data.  The CMS should be designed so that you are at least using the same models if the CMS is building the page on the server, or the CMS is fetching some JSON data from the web service via JavaScript.  That's a profound but necessary shift in mindset and architecture.  Of course, you'd take baby steps but this is the way I would encourage people to think.

I don't think I would go down the path of creating com_api.  Instead, I would look at creating a new application in the CMS tree to provide the web services where both the web services client and the CMS client work off the same base code.  For example, the CMS tree could be reconfigured like this:

/joomla-cms

../code

../www (the CMS entry point)

..../media (all the web media)

../api (the web services API entry point)

../code (?? the common core code)

../extensions (?? where extension PHP is installed)

That's just thinking out loud but you can quickly see that multisite starts to be possible with this refactor as well.  That kind of thing is where I would be planning to have the end goal, knowing full well if could take many steps to get there.  But the point is, stop thinking about data in terms of the CMS.  Think of the CMS as just one way of accessing the data.

I'll give this some more thought while I'm flying back to Australia.

Regards,

Andrew Eddie

[Herman Peeren]

@Ashwin: aha... this com_api in https://github.com/techjoomla/com_api is totally different from what you presented last year and is at https://github.com/techjoomla/Joomla-REST-API
Thanks for pointing me at it.Still reading it and playing with it.

In big lines my thoughts go in the same direction as Andrew's. That is also why I like Nooku's REST-interface: once you have written an extension you don't have to write a webservice for it too: that is just automatigically done. For when an extension is written, you allready have defined the data you want to display; with a webservice, you just output them in another way (and that way is provided by the core classes). A difference with Joomla! is that the controllers are task-oriented (the way of doing in Joomla! nowadays is: one task per controller). I call that "verb-oriented", in contrast to "resource-oriented" (or: "noun-oriented") with a uniform interface. A uniform interface is a limited set of methods ("tasks") that is implemented in resources. In that way Joomla!'s core architecture fits more to an rpc-style webservice (like XMLRPC and SOAP) than REST. If however we want to implement a REST-style architecture in Joomla! (as that is more popular now and has some advantages), we'll have to start thinking in resources with a uniform interface instead of tasks. Another direction would be: to make some hybrid webservice, as BTW is done by many others too, like Facebook, Flickr, Twitter etc. So: using ideas from REST (resources, nouns), but combining it with some "good old" reminiscants of procedural thinking (tasks, verbs); so you get resources with a not so uniform interface. Taking freedom to do that, not being puristic, is also what I read in that Web API book of Brian Mulloy.

[Ashwin]

Absolutely agree with the approach. I have seen the Nooku pattern and that is close to where we want to go. One thought that I had in mind is that Joomla already supports output formats format=raw, format=json  Although these are not really REST friendly and definitely not intended as webservice use, could act as a starting point. That way you can already re-use the existing models.

Another area that would need thought about is the authentication. We should be definitely looking at supporting OAuth, digest, key based login and also plain username/password based login.

Lastly, the com_api has simply evolved from Joomla 1.5, and since this was the faster & easier approach, was implemented. We definitely do not want a com_api in the core.

Looking forward to more inputs from Andrew and the GSoC project! 

- Ashwin

[Herman Peeren]

On Thursday, 7 June 2012 09:42:35 UTC+2, Ashwin wrote:

One thought that I had in mind is that Joomla already supports output formats format=raw, format=json  Although these are not really REST friendly and definitely not intended as webservice use, could act as a starting point. That way you can already re-use the existing models.

Yes, that is a good starting point. I think it was originally intended for webservice use; read how the plans were described in http://docs.joomla.org/Xml-rpc
It is often used in an Ajax-call, which is in fact also a webservice call ("An Ajax application is a web service client that runs inside a web browser", Chapter 11 in "RESTful Web Services" by Leonard Richardson & Sam Ruby; my little "bible" of REST).

On Thursday, 7 June 2012 09:42:35 UTC+2, Ashwin wrote:

Another area that would need thought about is the authentication. We should be definitely looking at supporting OAuth, digest, key based login and also plain username/password based login.

Yes, my ideal would be to make several authentication methods possible (strategy pattern).

[Andrew Eddie]

Louis has new router proposals (at last I hear the crowd roar):

https://github.com/joomla/joomla-platform/pull/1259

The key to any web service is the router class that selects the
controller, so would be worth looking over.

Ok, in terms of moving forward which mailing list are we going to use?
I'd suggest anything including changes to the platform and getting a
new non-CMS web service application written (that anyone can use as a
base for creating web services, whether for the Joomla CMS or a
bespoke project) we do on the platform list. Anything to do with the
Joomla Platform consuming other web services (like github, Facebook,
etc) should also, I think, be on the platform list. Then anything to
do with building web services for the CMS itself can be done on the
CMS list, leveraging whatever we are doing in the platform for generic
solutions. How does that sound?

Regards,
Andrew Eddie

[Herman Peeren]

Thank you; interesting.

I thought the purpose of these working groups is explicitly to get things done "downstream" in the CMS. If so, then the emphasis should be on the practical side of that, I think. If we produce something that could also be used "upstream" we should certainly make a pull request on that.

Or am I seeing it wrong? Would be handy to get this clear. So: what is the purpose of this Working Group (in the light of CMS - Platform, downstream - upstream)? Also: for what version of the CMS do we aim to work?

Ciao,
Herman

[Andrew Eddie]

Just from my personal preference, I'm more intested in talking about architectures for web service applications whether that be for the cms or not. If the web services group is only downstream of the cms then I'll leave you gentleman to it :)

Regards

Andrew Eddie 

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/HDIcMN09W84J.
To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

--
Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla developers

[elin]

I think for the  future of the CMS it's pretty much necessary to start to think of it as a set of coupled applications somewhat along the lines of what Andrew is describing.  Both providing and consuming web services are central to what CMS 4.0 needs to be if it's going to be relevant as along as the current architecture has been. So I'd encourage people to take Andrew up on his approach.

 

Elin

Andrew Eddie 

To post to this group, send an email to joomla-dev-cms@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cms+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

[Herman Peeren]

My personal motivation to put effort in this working group is, that I just need webservices now in Joomla CMS 2.5. It would be good if we can work that out to something more general in 3.5. We would of course constantly check with future plans to get something great in 4.5. But some people just need webservices in a production environment for the Joomla! CMS in 2012 and will not wait with that until 2015.

It's only a different focus, coming from a practical need to "scratch our own itch". So, although I personally am really more interested in the architectural basis and the big picture, and will surely put effort in and communicate about that too, my proposal for this workgroup is to take a practical road first to produce something usefull this year. At least, that is what I'm gonna do (as said: because we just need those webservices now for CMS 2.5).

I'd like to know what the needs of other participants in this working group are. If we have more focus, we can better plan who is doing what, where, when and why.

Herman

[elin]

For sure we can think of it as two separate projects and push what Andrew is discussing into the to the UCM/4.0 working group. 

Elin

[Ashwin]

I'll go with Herman on this one. We definitely need webservices today, and any changes or code we produce can be pushed upstream for inclusion in the platform.

This is something the CMS is missing for years, and it needs to be added sooner than later. 

I would say it's simply about the numbers - there are far more CMS users that we have platform users, so the CMS is where it needs to start and then bits and pieces can later be added to the platform.

Consumption of webservices is something that will also come into picture, I had frankly not even thought about that. On that front too, we have a GSoC project, and  I suggest that we wait until that project is complete to think about how the consumption aspect can be handled, unless of course we have more helping hands on the webservice group!

- Ashwin

[Anaekwe Kaosiso]

Hello,
I am new to this group. My primary purpose here it to get help with my
project which is an external application, I intended to integrate with
a Joomla CMS 2.5 using web services specially to register users, read
and create articles, etc

Please is anyone here, who has idea on where I can go to find resource
on Joomla web services for 2.5 version.

On the other hand, since I have full access to joomla database and
server files, I am thinking of creating independent service to
abstract data from joomla database, this is because I have little
knowledge of joomla framework, and It appears it will pose a huge
learning curve trying to learn everything.

But, if I can see a web service guide for 2.5, I will be more than
happy.

Kaosiso

On Jun 9, 9:38 pm, Herman Peeren <herman.pee...@gmail.com> wrote:
> My personal motivation to put effort in this working group is, that I just

> need webservices *now *in Joomla CMS 2.5. It would be good if we can work

> that out to something more general in 3.5. We would of course constantly
> check with future plans to get something great in 4.5. But some people just

> need webservices in a production environment for the Joomla! CMS in *2012 *and

> will not wait with that until 2015.
>

> It's only a different *focus*, coming from a *practical need* to "scratch

> our own itch". So, although I personally am really more interested in the
> architectural basis and the big picture, and will surely put effort in and
> communicate about that too, my proposal for this workgroup is to take a
> practical road first to produce something usefull this year. At least, that
> is what I'm gonna do (as said: because we just need those webservices now
> for CMS 2.5).
>
> I'd like to know what the needs of other participants in this working group
> are. If we have more focus, we can better plan who is doing what, where,
> when and why.
>
> Herman
>
>
>
>
>
>
>
> On Saturday, 9 June 2012 21:35:18 UTC+2, elin wrote:
>
> > I think for the  future of the CMS it's pretty much necessary to start to
> > think of it as a set of coupled applications somewhat along the lines of
> > what Andrew is describing.  Both providing and consuming web services are
> > central to what CMS 4.0 needs to be if it's going to be relevant as along

> > as the current architecture has been. So I'd encourage *people* to take

[Herman Peeren]

When I look at Stefan's code until now, the GSoC-project is also mainly server-side, not the consumption-side of webservices. That's on the same track as we are mainly on and it will be worth checking what is happening there this summer. The future plans, like Louis router (including his JApplicationWebRouterRest) are also in the same direction. So, it's all related and we can certainly all inspire each other!

My main idea about webservices, allready for some years, is: they are just another representation. You essentially do exactly the same things, no matter if you approach the data via HTML, or via REST, SOAP, JSONRPC or whatever. To keep things DRY, a minimum of extension-specific code - if any - should be necessary to CRUD the data that is already been CRUD by existing components (talking in CMS-idiom; but the same holds for more general applications). My ideal is to write API-software that works for existing extensions without extra additions per extension. I'm working on that and will present some ideas about it soon. The direction I'm exploring reuses not only the same model, but even the controller and adds a standardised view to that.

If we agree to take the practical road, we could define the steps and milestones we want to do in the near future. I'd like to work to some first practical results that will be useful asap.

[Herman Peeren]

Dear Kaosiso,

You are asking for "resource on Joomla web services for 2.5 version" and "a web service guide for 2.5", but this working group has just been started to create such web services. For, almost unbelievable but true, there are no webservices at all for Joomla! 2.5 yet. Some extensions use some webservices (or aspects of that) and we'll study them, but it is all very scattered over the Joomlasphere and there is totally nothing in the core.

Do you want to participate in this group to create those webservices for Joomla!? Maybe you could use the project you're working on as a test-case for things we are developing? Or maybe you could just give some specifications of the kind of things you want to achieve with a webservice. Anything like that would be of great support. A word of caution: although we want to deliver something useful as soon as possible, it could take some time, so if finishing your application is critical, you might be confronted with not reacing your goals in time. In the mean time: I would still be interested in hearing about your application.

Regards,
Herman

[Herman Peeren]

Dear Anaekwe Kaosiso,

You wrote me a personal email May 24 about using XMLRPC in Joomla 2.5. Sorry I didn't react earlier; it was on the Big Pile of TODO...
I hope you will respond to my earlier reply to you in this thread.

Best regards,
Herman Peeren

[Anaekwe Kaosiso]

Hello Herman,

Thanks for you response. I am ready to participate, as a test case using my application. The application is intended to offer similar CMS experience for Native mobile application.

While I really need to deliver this application as quick as possible, Nevertheless I wont mind running a  parallel integration that will ultimately produce the best result. By this I mean, I am willing to follow up with you core joomla integration outline, while experimenting with other adjunct solution  i can handle, I make my input as much as am able too.

Thanks 
Kaosiso

[Chris Davenport]

Something else to consider...

http://www.odata.org/

"The Open Data Protocol (OData) is a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today."

It looks interesting, although much more complicated than is strictly necessary, in my opinion.  But it might be worth considering.

If we do decide to support it, we would need to run it past the OSM legal team as there may be a potential patent risk from Microsoft.  Probably okay, but we would need to do "due diligence" checks before we could accept it into an official Joomla distribution.

Chris.

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/dldFMwb0OwIJ.

To post to this group, send an email to joomla-...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.

--

[Chris Davenport]

Not surprisingly there appear to be two goals to aim for and complicate the situation.

On the one hand we want to define an architecture that will be clean and efficient, delivering the maximum benefit to application developers for the least amount of additional code.  This should be possible at the Platform level and I'd be very keen to encourage thinking along the lines that Andrew is suggesting.  But it seems to me that this will be at best a long-term prospect for the CMS.

On the other hand, there is a clear need for some sort of solution to be implemented in the CMS as soon as possible, preferably in the early stages of the 3.x cycle and if possible even giving consideration to backporting it to 2.5.x if that is possible.

I don't see any easy way to reconcile these two goals.  So how about we just accept the situation and work on these ideas as two more or less separate projects?  Perhaps for the 3.x CMS cycle we work on a v1 API, while at the same time those interested in a more long-term solution can define a v2 API which could be incorporated into the CMS possibly during the 4.x cycle.  It might be possible to support both v1 and v2 APIs during 4.x with v1 being dropped in 5.0.

I don't see any reason why this working group should not look after both projects, although I agree with Andrew that the platform mailing list is more appropriate for discussions concerning web service support in the Platform.

Does that sound like a reasonable way forward?

Chris.

--
Chris Davenport
Joomla Production Leadership Team

[Andrew Eddie]

On 12 June 2012 04:11, Chris Davenport <chris.d...@joomla.org> wrote:
> Not surprisingly there appear to be two goals to aim for and complicate the
> situation.

Actually it doesn't have to be complicated at all and the goals share
a common path that forks at an appropriate point. Here's what I would
do.

A. In terms of code, I would create a new repo in github called
joomla-cms-api. I would more or less clone Stefan's project as the
starting point. While there are a few ways to think about deploying
it, I think the best way to approach it in the short term is to think
of this repo dropping into a {install}/api folder, just like the CMS
administrator application is in {install}/administrator. Being a
separate project allows you to handle web service specific issues
better and accept contribution more easily (please consider doing it
like we do on the Platform only with pull requests, not via patches
and a JoomlaCode tracker).

In terms of separation of work, I'd consider this step more Platform
than CMS … for now. We still have to iron out basic issues like
authentication and settle on a router strategy and that is all common
to any web services application that will be built on the Joomla
platform (and we really want to make Joomla an attractive choice for
writing such applications). It's also fulfilling Joomla's mission
which is very, very clear about the CMS not being the end of the line
(so lets drop the "the CMS is 99.99% of what the Platform is used for
anyway" nonsense please - that's a really narrow and out-dated view to
hold on to).

B. The next step is to consider designing the API itself. To that
end, I'd recommend jumping on Apigee's YouTube channel and watching
their many video's on the subject, and scanning the api-craft Google
group.

This step, and eventually step C are most definitely CMS centric so
I'd talk about them here. This is a big step and a long conversation.
You should start at least one new thread about it.

C. Finally, you want to think about the architecture in the long term
and be in the mindset that all the model-like code will end up in the
services repo (in other words, stop thinking like a CMS in terms of
business logic and data manipulation). This is obviously a big deal
and will take many cycles to plan and implement, but I'd like to sow
the seed now as it helps cement the position of the goal posts.

This is a conversation to start after you have a few basic web
services designed and are comfortable with how everything works. The
goal here is to ensure that you aren't duplicating code (mostly
models).

D. The other aspect is designing more services for Joomla (CMS or
other apps) to consume, like Facebook or whatever. We already have a
github API and any others can follow that paradigm; it's just a matter
of people writing code and contributing it to the Platform. This is a
space in the market that I believe the Joomla project can own.

Anyway, that's 'a' way to approach it.

Regards,
Andrew Eddie

[Herman Peeren]

 I think it is a good proposal from Chris to work parallel on 

• the short term goal of a working API as soon as possible, preferably even useful for CMS 2.5. Here are some immediate needs, motivating some people to do the work.
  
• the long term goal of a good API in the platform (and everything that comes from it). Here the fundamental work for future applications will be done. Andrew has done some proposals for a way to mainly work on this second goal.

It might be an idea to start our own "work"-mailinglist for this working group, for only a limited number of mailinglist-subscribers is interested in details of the subject. And then regularly do an update to the two main mailinglists (this one and the platform's) for the two aspects of the project. I am willing to do the regular reporting to the two  main mailing lists. 

I think the emphasis of this project at the moment is on the API-side, but consuming webservices in general can of course later be added.

[Andrew Eddie]

On 12 June 2012 15:40, Herman Peeren <herman...@gmail.com> wrote:
>  I think it is a good proposal from Chris to work parallel on
>
> the short term goal of a working API as soon as possible, preferably even
> useful for CMS 2.5. Here are some immediate needs, motivating some people to
> do the work.
> the long term goal of a good API in the platform (and everything that comes
> from it). Here the fundamental work for future applications will be done.
> Andrew has done some proposals for a way to mainly work on this second goal.

No, that's not correct. My assumption is that you will use the same
application architecture to deliver both short and long term goals for
the CMS. If you want to deliver web services within the CMS
(component??), that's fine, but I think you'll find it's a really,
really bad idea.

[Herman Peeren]

In the switch from SOAP to REST many people forgot about the benefits of WSDLs. Especially the possibility to automatically (!) build client-side apps for connecting with the server, based on this metadata-information, was useful. There will still be a need for a description of metadata, so logically protocols like this OData will emerge. Probably followed by some directory services that can be automatically discovered (like the UDDIs). In some aspects the future is a reinvention of the past.

[Herman Peeren]

I totally agree in the longer term and on a fundamental level, but right now we just have to resolve some needs for web services in the current CMS.

[Ashwin Date]

Hi Andrew,
The short term goal (of adding webservices in the CMS) is not looking at using a component for the webservices. The aim is to leverage existing stuff - either the controller or the format=json/xml etc to provide the webservices by building on top of that. The CMS end of things will definitely work towards a common goal of ultimately integrating maximum code with the platform. I think the 2 pronged approach should be good, with co-ordinated teams working towards the short & long term goals.

That said, it's important that we get the webservices in J! 2.5/3, that way we can have a lot more feedback and then filter out the best things for the platform.

Herman,
On the webservices description end, I agree that SOAP is definitely best from self-documentatino perspective, however we can design the REST services so that they can be self documenting as well. There are several libraries out there which can do this.

THis also opens up the question of supporting multiple types of services - SOAP/REST. Should we plan to support everything (in which case we'll need some kind of an adapter approach) or just REST ?


- Ashwin

[Herman Peeren]

I think it is clear that the way to go nowadays is REST. Thinking in resources suits the OOP paradigm better than Remote Procedure Calls, which is coming forth from procedural programming. BUT, as I always like to leave other possibilities open, I would like to look for possibilities to put a layer in between (what you probably  mean by an "adapter approach"). Also because Joomla!'s basic architecture is not RESTful (like for instance Nooku's is); so there always have to be some adaption. For some Joomla!-applications/extensions an RPC-style web service would be easier to accomplish.

I think for the short term goal (= getting things done asap), as you said, we will try to use what is there as much as possible, but maybe some adaptions are necessary. That could be in the form of a core-class-overriding-plugin (to adjust some basic class for the short term), as a separate component (just as an extra layer) or as a separate application (like we had with XMLRPC in 1.5). At the moment I'm playing a bit with the idea of using a component as an extra layer (to go from there to the original component). Let you know when I've got something to show.

Herman

[elin]

In terms of consumption actually people should remember we also have JGoogle, JFacebook (and J

Twitter, JLinkedin also by Diana) and JMediawiki projects this summer and all are making fantastic progress. I'm sure each of those developers would appreciate people cloning and providing feedback. I know that with the JGoogle project at least only a small portion of the more than 90 Googledata APIs will be included but we the structure of the class will make it easy for other developers to contribute specific classes they need or would have fun writing. 

https://github.com/aaronschmitz/joomla-platform/tree/JGoogle/libraries/joomla

https://github.com/dianaprajescu/joomla-platform/tree/JFacebook/libraries/joomla/

https://github.com/nprasath002/joomla-platform/tree/JMediaWiki/libraries/joomla

Anyone can really take on writing a similar API as a project.  It would be great to see Amazon, Yahoo etc (see http://framework.zend.com/about/overview for some ideas :)).  

If merged all of those are going to be usable in the CMS right away.

Elin

[Andrew Eddie]

On 12 June 2012 16:06, Ashwin Date <cool...@gmail.com> wrote:
> Hi Andrew,
> The short term goal (of adding webservices in the CMS) is not looking at
> using a component for the webservices. The aim is to leverage existing stuff
> - either the controller or the format=json/xml etc to provide the
> webservices by building on top of that.

You mean adding web services directly into the existing components?

(In case the answer is yes, I really hope you don't go down that road)

Regards,
Andrew Eddie

[Andrew Eddie]

On 12 June 2012 18:44, Herman Peeren <herman...@gmail.com> wrote:
> Also
> because Joomla!'s basic architecture is not RESTful

Um, actually/technically it is, but the API is limited to GET and POST methods.

[Herman Peeren]

On Tuesday, 12 June 2012 13:47:54 UTC+2, Andrew Eddie wrote:

You mean adding web services directly into the existing components?

I think the short term goal is to use as much as possible of the existing components, without adding code (or at least: adding as little code as possible) for the specific extensions. How far can we come making their already coded functionality available via web services? That serves a double goal: we fulfil a need (to connect Joomla-functionality with other apps than browsers asap) and we learn from it. It is just a first step some of us will take on this road. It doesn't prevent us or others from taking steps in other directions too.

We will probably do some steps in the wrong direction too. That is part of every creative process.

[Herman Peeren]

Sorry, I said it a bit too harsh. JTable and MVC indeed offer a CRUD-interface to basic objects, that is easily mappable to a REST-interface (for which it doesn't matter much if you implement HTTP PUT and DELETE methods directly or via overloaded POSTs or via another code).

I meant RESTful in the sense of thinking in resources (nouns) with a uniform interface instead of thinking in controllers with a task (verb). It is not a crystal clear matter of definition, but more a general way of thinking. I want to avoid hanging labels of good or bad on RESTful or not; I'll try to just notice.

[Andrew Eddie]

On 12 June 2012 23:59, Herman Peeren <herman...@gmail.com> wrote:
> I meant RESTful in the sense of thinking in resources (nouns) with a uniform
> interface instead of thinking in controllers with a task (verb). It is not a
> crystal clear matter of definition, but more a general way of thinking. I
> want to avoid hanging labels of good or bad on RESTful or not; I'll try to
> just notice.

I know what you mean. But then again, proposing to solve web services
by sticking a bandage on existing components isn't really achieving a
robust RESTful interface either.

A concern though is that extension developers are going to copy what
you do so if you choose the quick-and-dirty approach, you are going to
live with that decision for a long time.

It's going to take the same, possible less effort to design the system
properly from the start, so why not take that golden opportunity? The
bottom line is the CMS as it stands is not a good architecture to
deliver web services (CMS 5.0 running on the core platform, not
legacy, fully converted to use the auto-loader may be a different
story).

But that's just my take. Don't let me stop you doing it, but I just
see a large amount of effort going to waste for no good reason other
than to achieve a questionable quick fix. I also see many vendors
(Github, Facebook, etc) that have been in this space for much longer
and it behoves us to learn from the mistakes they have already made.

[ssnobben]

Well this does mean that we can connect Joomla with an open source
"rule engine" like OpenL Tablets et al :) ?

http://openl-tablets.sourceforge.net/documentation/faq#question3

http://openl-tablets.sourceforge.net/documentation/getting-started/getting-started-for-business-analyst

"OpenL Tablets Integration SOA Ready

Expose rules as efficient, scalable and standardized services using
Service Frontend."

[Herman Peeren]

Could you give a practical example of what you want to accomplish with that? Do you want to use the business rules in OpenL Tablets to regulate access to Joomla!-documents? As an alternative for or enhancement of Joomla!'s ACL?Or do you have something else in mind? -Herman

[Herman Peeren]

BTW, as you probably know, there are plans to go from the current ACL to business rules.
See for instance: https://github.com/eBaySF/joomla-platform/tree/content/libraries/joomla/authorisation
Maybe you envision such business rules in Joomla being managed via webservices with external software?

- Herman

[Herman Peeren]

The goals of this web services workgroup are: to provide connections to and from the functionality and content of a Joomla!-installation (via HTTP). What we want to accomplish first is simpler than what you propose; think of things like updating Joomla-content via a smartphone-app.

- Herman

On Friday, 15 June 2012 08:54:52 UTC+2, ssnobben wrote:

[Andrew Eddie]

On 15 June 2012 18:35, Herman Peeren <herman...@gmail.com> wrote:
> BTW, as you probably know, there are plans to go from the current ACL to
> business rules.
> See for instance:
> https://github.com/eBaySF/joomla-platform/tree/content/libraries/joomla/authorisation
> Maybe you envision such business rules in Joomla being managed via
> webservices with external software?

That's not really the intent, though it is a desirable side effect.
The authorisation package is simply an attempt to refactor the current
access control system into a strategy pattern. I probably should have
done it that way in the first place, but you know what they say about
hind sight.

[Andrew Eddie]

On 15 June 2012 18:46, Herman Peeren <herman...@gmail.com> wrote:
> The goals of this web services workgroup are: to provide connections to and
> from the functionality and content of a Joomla!-installation (via HTTP).
> What we want to accomplish first is simpler than what you propose; think of
> things like updating Joomla-content via a smartphone-app.

Ok, so for web service applications we need to start another working group?

[Herman Peeren]

On Saturday, 16 June 2012 01:45:23 UTC+2, Andrew Eddie wrote:

Ok, so for web service applications we need to start another working group?

Don't think so. Let's explore several roads but keep communication bunddled.

First: I don't know exactly what you mean by "web service applications". My interpretation of that expression is: client side applications, consuming Joomla!'s webservices. For instance native programs on a desktop, a smartphone or whatever. Or did you mean: Joomla! Platform applications? And if so: did you mean server-side (providing a more general API) or client-side (consuming webservices from other parties like Facebook etc)? For I think all those things are related and can all work together towards the bigger goal: get Joomla! connected. It's just another focus.

I should have said: "the goal of some people in this working group, including me, is...". For the road I'll be following is: first work on a "bandage" (thank you, that is the right word) to provide (server-side) webservices that can be used in a CMS 2.5 installation. I waited for 2 years with that and cannot wait another 2 years; unfortunately I need that bandage a.s.a.p. It will clearly be presented as a bandage and a temporary solution. It is also a learning track: for at the same time we (that is: me and whoever wants to take that same road) will work on basic framework/platform specifications, based on the practical work on the "bandage". These are two roads that are not opposite, but parallel. Both are server-side: provide an API to connect to Joomla! from the outside.

There are 2 other focusses, which are on the web service consumption (client) side:

• native applications on desktops or smartphones or whatever
  
• and consumption of other party's web services within Joomla (connecting Joomla with Twitter, Facebook, whatever)

I don't think it is handy now to split that up in different working groups; for information can better be bunddled than scattered. I just sketch the road that I will follow (although I would prefer a road to not need a "bandage" first, I unfortunately feel the need to focus on that first); whoever wants to join me on that road is welcome. In the mean time I'd like to keep in touch with people who prefer another road.

[ssnobben]

I was thinking of a open source ecco-system and integration with other
world class open source projects that people world wide uses so Joomla
( 3pds) dont need to invent the wheel again ( doing small business
limited solutions) instead using this SOA architecture for small and
mid sized org and companies. That would strengthen the future for
Joomla and also attract more Joomla users.

Other examples of projects that Joomla would gain to integrate with:
BPM ( http://www.processmaker.com/), BI ( http://www.pentaho.com/
already working integrated with ProcessMaker see this integration),
Rules ( OpenL Tablets, Drools- http://www.jboss.org/drools/) OpenX ad
system (even that they are hacked now
http://www.ficora.fi/en/index/viestintavirasto/uutiset/2012/P_23.html)
etc

So its not to use "business rules" within Joomla plattform itself is
to use other platforms to make Joomla ecco system ( the open source
ecco system) better. Thats why the integration with other platforms
are important for attracting other users that have those needs and
Joomla could be smart to be the front door (CMS) to that ecco system.
So rule engines(decision engine) take decisions and could be used in
many ways. This example uses rules/decision engine for marketing
http://avail.com/solutions/technology/ so a system like this could be
uses in many many ways..

Rgds

[Ashwin Date]

Hi snobben,

The whole purpose of the group is indeed exactly to achieve what you are saying. The group would be responsible for creating a framework that would expose REST webservices from Joomla so that it can be easily integrated with any other 3rd party application. However, the immediate focus is on creating an infrastructure for the webservices. Paralelly, the group is also focussing on creating a framework that will allow consumption of 3rd party services like linkedin, facebook, google etc.

When both the above are ready, it would be very easy to couple Joomla to the applications that you mention and many more.

If you have thoughts on how we can create a webservices framework for Joomla so that it will be easily possible to integrate with the services above, we're all ears to your suggestions. Shoot!


- Ashwin

[elin]

We actually also have a GSoC project on building a workflow engine and related apis and I'm sure that developer will be looking for feed back and   maybe even code contributors in a few weeks when there is a bit more code pushed to the repository.  

As I said though, anyone can write, share, and offer to contribute an api to consume from third party application. I'm not sure why some a of those developers are not sending pull requests if they have done or started this work already. Why do hacky when you could do it the right way and make Joomla better at the same time?

Elin

[Matias Aguirre]

There are some advance on this? Im trying to get REST working on the CMS but is a bit complex, i like to know if there are some clear objectives or next step.

What im trying to do is to adapt the Louis Landry REST libraries to an onAfterRender plugin (like server) and there read the headers to establish the communication. The idea to use this is to have secure access to content between Joomla's installations (for migrations or just copy/move data).

I can establish the communication using to the plugin "server" using this REST browser extension (that is very useful) and using a custom CLI client too, but i have some doubts with how to implement it between 2 CMS and how to implement the oauth_callback.

Tell me if I'm mistaken, what we need to have is:

• One component, to get the token
• One plugin or something that route the Rest requirements
• The libraries to do all
  

This is ok?

Regards

[Herman Peeren]

Hi Matias,

I'm working on this, but have not made much progress the last weeks (doing too many different projects at the same time, as usual). Just a short reaction on what you wrote:

• first of all: great if you want to work on this too! Using webservices for migrations sounds good to me; that is is nice practical example.
  
• accessing the resources and authentication/authorisation are 2 different things; work on those can be done seperately.
• OAuth is a possible and much used way of authorisation. There are other ways too. I'd like to keep different authorisation-schemes open. Maybe in the example of a migration it might not be the best way. A Basic login with username and password in an authorization-header, or maybe sometimes no authorisation at all (for instance when using a localhost for the migration), could be more efficient and sufficient. See for instance: http://hueniverse.com/2007/09/oauth-isnt-always-the-solution/ .
  
• my plan is to first work on a component that handles the request: checks the authorisation (or has that done by another part), provides routing to the original html-component and returns the http-response. An entry point for webservices. My main challenge, what I try to accomplish, is: try to avoid any special code for different components, but try to make it as general as possible, so that when this webservice-component is added you have a (RESTful or other) access to all existing components.That is why my focus is now on the routing and providing this "gateway" to the resources. I don't look at authorisation for the moment.
  

There is no clear plan for a next step in general for this working group in general. I personally have a next step (a component to provide webservice-access to existing components), some people prefer to focus on longer term and more thorough solutions (like with the GSoC REST-project), and maybe you want to first investigate the possibilities of OAuth and tokens. I'm convinced that in the end all those efforts will not be in vain but contribute to better webservice-connections for Joomla!.

Kind regards,
Herman Peeren

[Matias Aguirre]

Hi Herman,

Basic authorization seems to be a good way to do migrations, oAuth represent a more secure way but more difficult to implement in the short time, it needs lot things to test (i could make it work only in PLAINTEXT, RSA returns always a different key) so i like to know more about your idea about use component instead a plugin.

I know that you are thinking it only for 2.5+ but remember that i need to implement it on 1.5 too because i need to GET the content from there so is very strange for end users to install 2 components on 2 sites to make a migration. So, why we cant make a plugin that resolve the petitions and all the routing to the components?

Other thing to be consider is the URL, for example: We need to get one content using this URL http://www.example.org/content/59, how joomla know that this URL is a REST requirement and needs to be resolved with the com_rest component?

Sorry about my ignorance about routing but is more easy to ask :-)

Regards

[Herman Peeren]

The idea I'm working on now is a component, com_ws, and if you want to retrieve com_content id=59 you'd call www.example.org/ws/content/59 (you see: with the name of the webservice component in between, here ws, but could be api or whatever). So everything is routed to com_ws and from there we figure out what the rest of the route is (which also depends on the router.php of that specific component). The component could also implement RPC-style webservices, but let's first look at REST.

What I try to do is to keep the URL-part after /ws/ the same as it would be when "normally" entering the website with html. So in fact, you don't know anything about the id, but give a menu-alias and get some article back. It would be something like www.example.org/ws/menu_alias. I haven't thought how to exactly name a URL for module-output yet, although I had some (positive) experience with that some years ago, when I made Flash-templates with Joomla!-backends: I then retrieved menu's and content in XML-format. What made it very easy: I just retrieved content that was also publicly available via a html-website and didn't need any authentication. But if you are migrating a site, it is also no problem to provide a basic authentication with your superadmin username and password: it is something different when you want to give all kinds of programs and people restricted access to your site (but that's where we have ACL for in 2.5).

I'm mainly concentrating on the server-side now. So, that component has to be installed on the "sending" site (the one you migrate from). Of course there has to be some client software too, to store the retrieved content in the new site. So you always need 2 extensions (components or plugins or whatever): server-side and client-side. Bit confusing terminology: the client-side can be another web-server...

Hope my ideas can be of help for what you want to accomplish.

I see it all as temporary solutions, a "bandage" until we have webservices as a solid part of the framework/platform.

Ciao,
Herman

[Herman Peeren]

Hi Matias,

When you want to use webservices for migration of content you only need to GET (read) the content from the server-side, not write. Authorisation is always needed for writing, but for reading less restrictions are needed.

In the mean time I just wonder if a direct db-migration is not much easier... Why are you looking for webservices for content-migration? Is that the best tool? I'm curious for your thoughts about this.

Ciao,
Herman

[Matias Aguirre]

Herman,

The idea to implement webservices on migration comes because some big sites (really big.. like 300mb of database) causes lot of errors, timeouts problems and buffer/memory overflow if we run direct db migrations, so webservices can retrieve it a little more slower (i dont know how much slower) but is more stable (i think) because we can retrieve and migrate at the same time. That is the main idea, i really dont know if this is the better solution but we agreed with Ronni that we like to try.

The problem is that i not only need to retrieve content, i need to retrieve content, categories, modules, components, etc.. so this must to have some authentication.

What im doing now and its working for now is:

• Create a "server side" plugin, that is installed in the site that have the content to migrate
• The standard upgrade component installed on the new 2.5/3.0 site, that will to retrieve the data and migrate it

The plugin that im writting is running good for authentication but is missing the route to get the content/categories/component data.. i think that is not a very big work, my idea is to create differente classes called content.php categories.php modules.php components.php etc that do the differents retrieve methods.

I ll to show you some example when i have this working, maybe we can reuse some classes or some ideas.

Regards

[Matias Aguirre]

Herman,

For URL issue, im sorting this passing the type (content) and the id (59) into the headers too so the joomla router dont causes problems. I know that this is a bad solution but is fine for only migrating proposes.

Take care

[Matias Aguirre]

Hi all,

I have one plugin for joomla 1.5 that can be used for example of RESTful web services. This is an experimental plugin so please only use it for testing proposes.

The way to use it is to install the plugin and enable it. Then with the RESTClient extension, we can recreate the client side:

Method = GET
URL = {Your URL Here}

Authentication -> Basic Authentication = A valid username/password of your Joomla 1.5 installation
Headers -> Custom Headers -> type = user
Headers -> Custom Headers -> task = total

What should do this? This will to retrieve the total rows of your user table. If you like to get one particular user:

Authentication -> Basic Authentication = A valid username/password of your Joomla 1.5 installation
Headers -> Custom Headers -> type = user
Headers -> Custom Headers -> task = row
Headers -> Custom Headers -> id = 62

This will to retrieve the superadmin user in JSON format. For now, only the user table is supported.
I hope this is useful for someone.

Regards

On Wednesday, 4 July 2012 17:42:29 UTC-3, Herman Peeren wrote:

[Ashwin Date]

Hi Matias,

The architecture does look good, however the only catch that I see is that the current structure is 'table' driven. In case of Joomla a object can be represented by data collected from various tables. As an example if you have Jomsocial, then the user object would have information from his Jomsocial profile as well. We need to think how this can be addressed.

[Matias Aguirre]

Hi Ashwin,

Yes, the table structure is adapted only for jupgrade/migration proposes. It should be adapted to be more extensive and general.

Regards

[Ashwin]

I worked on a quite large mobile application recently and a major challenge that we had was that of uniform data structures. As an example, consider the following APIs for Jomsocial

API Group 1

- Member search

- List of Friends

- Group members

- Event attendees

API Group 2

- Group search

- List of My groups

All the above APIs are expected to return a list of groups/users. What is important is that the data for the individual users/groups in each of the APIs needs to be same datastructure. Depending on how the data is accessed the fields returned from the database would be different, which would lead to unpredictable fields in each api. To overcode this, we created data structure objects like APIDataObjectProfile, APIDataObjectProfileSmall, APIDataObjectGroup, APIDataObjectGroupSmall

When data is accessed using Jomsocial models or directly via database, it's always passed through the data objects so all the APIs from the same group have the same data structure.

The above is something that we should be considering when we build the architecture.

[Ashwin]

Hi Herman,
Did you have any luck working on the controller mods for achieving the webservices ?
Am planning to spend time in the next few days doing a prototype for com_users, so if we can catch up on Skype that will be great. If you can give your skype id, will add you to the webservices skype chat.

- Ashwin