Web accessible APIs

[Oli Griffiths]

Hey all

I'm needing to create a web accessible api for a component im creating. I need to be able to converse with the component securely and be able to send an retrieve data. The component will be hosted on a remote server and needs and api to insert/modify/delete/retrieve data and perform functions on that data. The component has a hosted frontend that users can use but I also want an api to I can integrate other joomla websites with it.

What is the best and future proof way of doing this.

I recall that xmlrpc is depreciated in 1.6 so is webservices a good way, perhaps soap? Can anyone point me in the right direction?

Also, what is the best way to get a list of the articles out of a joomla site? I've used an xmrpc plugin that achieved it but wondered if this is the best way.

Thanks

Oli

---

Contact Organic Development:
Telephone: 0845 869 7654
Web: http://www.organic-development.com
Email: in...@organic-development.com
Twitter: http://twitter.com/growwithorganic

---

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Organic Development Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone.
Please contact the sender if you believe you have received this email in error.

[Dan Lopez]

This still needs more work to integrate with components, but that is the plan. So please take a look at it:

http://github.com/plexicloud/Plexicloud-JWS

--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To post to this group, send an email to joomla-de...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-gene...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-general?hl=en-GB.

[Brad Gies]

I don't think I'd bother with SOAP. It's a lot of overhead for almost no
gain. You can access Joomla webservices via HTTPS and that should be
more than enough security for most purposes. You also don't need xmlrpc.

You can just use a normal Joomla component and call it with the
format=raw parameter. You just need to make sure your component is well
written to ensure that it only sends the data after the user is well
authenticated. Joomla 1.6 makes it a little easier because you can use
the normal MVC patterns with different formats, but it's achievable in
1.5 as well.

Louis had a pretty good article on how it's done in 1.6. Good enough
that I had no trouble figuring it out, but I don't have it bookmarked
any more. Google for "Joomla 1.6 Louis webservice xmlrpc" or something
like that and you should find it.

Sincerely,

Brad Gies
-----------------------------------------------------------------------
Bistro Bot - Bistro Blurb
http://bgies.com
http://bistroblurb.com
http://ihottonight.com
http://forcethetruth.com
-----------------------------------------------------------------------

Everything in moderation, including abstinence

Never doubt that a small group of thoughtful, committed people can
change the world. Indeed. It is the only thing that ever has - Margaret Mead

On 22/09/2010 11:36 AM, Oli Griffiths wrote:
>
> Hey all
>
> I'm needing to create a web accessible api for a component im
> creating. I need to be able to converse with the component securely
> and be able to send an retrieve data. The component will be hosted on
> a remote server and needs and api to insert/modify/delete/retrieve
> data and perform functions on that data. The component has a hosted
> frontend that users can use but I also want an api to I can integrate
> other joomla websites with it.
>
> What is the best and future proof way of doing this.
>
> I recall that xmlrpc is depreciated in 1.6 so is webservices a good
> way, perhaps soap? Can anyone point me in the right direction?
>
> Also, what is the best way to get a list of the articles out of a
> joomla site? I've used an xmrpc plugin that achieved it but wondered
> if this is the best way.
>
> Thanks
>
> Oli
>
>

> ------------------------------------------------------------------------

> Contact Organic Development:
> Telephone: 0845 869 7654
> Web: http://www.organic-development.com
> Email: in...@organic-development.com
> Twitter: http://twitter.com/growwithorganic
>

> ------------------------------------------------------------------------

> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Organic Development Ltd.
> If you are not the intended recipient of this email, you must neither
> take any action based upon its contents, nor copy or show it to anyone.
> Please contact the sender if you believe you have received this email
> in error.

[Oli Griffiths]

Thing is, I don't really want to make all the different views/controllers available to the api. I also don't really want to have to use post data to interact with it. I'd rather send the data as a payload of a single post variable rather than the entire post array.

I currently have a Json layer that can respond with a json string from the component if format=json is called. Is json an acceptable format for an api?

Oli

[Ian MacLennan]

On Thu, Sep 23, 2010 at 2:50 AM, Oli Griffiths <o...@organic-development.com> wrote:

I currently have a Json layer that can respond with a json string from the component if format=json is called. Is json an acceptable format for an api?

Oli

Yes.  Google 'restful json' and you will find lots of discussion on it.

Ian

[Herman Peeren]

On August 25 I wrote something related to this issue on the Nooku
mailinglist in this thread: http://groups.google.com/group/nooku-framework/browse_thread/thread/544eb7801d63c4eb/

As I don't know for sure if that list can be read if you are not a
member of that list (what you should certainly apply for if you're
interested in RESTful webservices for Joomla!), I'll repeat my main
posting in that thread here. I hope you'll find some inspiration in
it:

-----------------------------------------------------------------------
I'm working in the line of a controller extension, both for Joomla!
and Nooku. My extension gets the credentials from the user-input and
that can be from the HTTP-headers (for REST) but also from the POST-
body, e.g. from SOAP-headers (I want to use the same login for REST,
SOAP and XML-RPC). I try to reuse as much as possible from the
"normal" html-login: I think there essentially is not much difference
whether you login as a human being with a web-browser or as a
computer-
program with a webservice. I also want to use the token-generation and
-verification that is already natively available; don't reinvent the
wheel. It is work in progress and I hope to publish it soon. I'll tell
some more about my ideas about it; I'm open for other suggestions.

I've first experimented with a Joomla! system-plugin that logs in
onAfterRoute(). It works, but if I want to keep the normal html token-
verification from the login-page working (which is important to avoid
CSRF-attacks), then with my webservices I always have to go to the
server twice (like in a digest-authentication) to get the original
token. The plugin could also totally ignore the token-check, but I
don't want the normal html-login be anything less secure than it was.
At the moment, I've temporarily left the path of the plugin and am
working on the controller-extension (in some "spare" time).

It is all about code reuse. I see it this way: if an extension
(Joomla!, Nooku, whatever) does something with data, then in order to
(re)use the same thing as a webservice, only the controller and view
have to change (a bit); the model can stay exactly the same! For such
webservices you want to use (existing) data-manipulating extensions,
which are mostly in the administrator-side when we talk about a
Joomla!-website. In order to use those files you have to be logged in;
otherwise you will be redirected to the html-login-page. That is why I
looked at the plugin-possibility: to be able to directly address files
at the administrator-side.

The controller-extension I'm working on now, always is on the site-
side. It then uses the models from the administrator-side; you don't
have to be logged in for that. But to keep it secure, you have to
provide authetication and authorization; that is why I do the login in
the controller. I haven't thoroughly looked at it but I wondered if it
is completely secure the way it is used now in the Nooku-example
com_harbour, where the site-models use the admin-models (with
KFactory::map()) without any authentication needed? Would that not
open ways to use data-manipulation methods in the admin-model without
proper authorization...???

I think that for proper authorization an ACL-system would be
important. And again: that should be the same system for use from a
webbrowswer or via a webservice. I'm looking at the ACL that is
planned for Joomla! 1.6, but the specifications of that are not even
completely clear yet... Grrr. I think for a webapplication to be used
in a serious way, good authorization is basic. Maybe I'm biased by the
use of .NET too much, but it is an example where authentication and
authorization is the basis upon which the whole system is built.
Whereas in Joomla! a.o. the ACL is just coming in afterwards, as an
extra. In a way, the whole concept of a separate administrator-side is
based on another concept of authentication and authorization than ACL;
the two interfere sometimes.

I think the OAuth-project is very interesting: it provides a way to
use resources from a server in another application, without sharing
the credentials to login into that server (e.g. giving an application
access to some part of your Twitter-account, without sharing your
Twitter-credentials). But I wonder if that is the best solution when
you want to use webservices on your own site. And I mean "best
solution" in terms of code reuse (both for the models allready
available as for the existing Joomla!-login-methods). Let's find out.
-----------------------------------------------------------------------

The last posting in that thread was from Sept. 4, by Johan:

-----------------------------------------------------------------------
Yes, this would definitely be something to work on for 0.8. KResponse
and KHttp are packages we would need to improve to make this all
happen out of the box. Looking forward to your code and ideas.
-----------------------------------------------------------------------

BTW, you've probably seen my slides from J and Beyond about
webservices for Joomla!:
http://www.slideshare.net/HermanPeeren/webservices-connecting-joomla-with-other-programs-4399412.
There is even a video of what I said there:
http://jandbeyond.org/video/tuesday-june-1st-2010/item/241-webservices-connecting-joomla-with-other-programs-herman-peeren.html
( a bit messy as the session started later and some people dropped in
later). The aim of the presentation was an overview of who is doing
what allready now.

As for SOAP, I think it also depends on for what applications you want
to have your webservice available. For instance in Delphi, Flex/
Actionscript and certainly .NET-applications SOAP is built-in so you
hardly have to do anything more than give the wsdl-adress and you have
your application up and running. REST certainly is the hype & hope,
but I personally would like to provide standard connections of
different kinds from the framework.

[Brad Gies]

In my web services I ONLY return JSON, so my vote would be yes for it
:). JSON is lightweight, and because I have written both ends of the
webservice communications I already know the data types, so the only
checking I do is for security reasons.

I don't use JSON for posting, but there is no reason you couldn't.
Actually, it would be a good way to write one entry point into your
webservice to accept the JSON and decode it, and if you added a field
for the function you want called, the webservice could just do the
security checks, decode the JSON, check that field, and use a switch
statement to pass the JSON on to the correct function, and then return
the results. I really don't think it saves much code however, because
you still need to write all functions. I normally do it the other way. I
write a function that does the security checks, and then call it from
every function written for the web service, but that's just personal
preference. There is no right or wrong way, and both are good designs.

Actually, I should have mentioned that in J 1.5, I normally just do
everything in the controller without the model and views. The code runs
a bit faster, but for 1.6, I do use the MVC pattern.

Sincerely,

Brad Gies
-----------------------------------------------------------------------
Bistro Bot - Bistro Blurb
http://bgies.com
http://bistroblurb.com
http://ihottonight.com
http://forcethetruth.com
-----------------------------------------------------------------------

Everything in moderation, including abstinence

Never doubt that a small group of thoughtful, committed people can
change the world. Indeed. It is the only thing that ever has - Margaret Mead

[Oli Griffiths]

Thanks for all the info guys, ill look into it and see what is the best approach.

I just wonder if there is an "official" way of communicating from the outside world to joomla in 1.6?

Oli Griffiths
Development Director
Organic Development Ltd.

Contact Organic Development:
Telephone: 0845 869 7654
Web: http://www.organic-development.com
Email: in...@organic-development.com
Twitter: http://twitter.com/growwithorganic

[Herman Peeren]

On Sep 24, 9:48 am, Oli Griffiths <o...@organic-development.com>
wrote:>

> I just wonder if there is an "official" way of communicating from the outside world to joomla in 1.6?

No. We need to develop it.

The only thing that is done is the controller-redirection as outlined
in Louis' article at http://docs.joomla.org/Xml-rpc. Louis has started
writing a native Joomla! xmlrpc-library instead of relying on
phpxmlrpc, but this piece of code is not yet finished (I was looking
at it too, but also: lack of time...).

[Dan Lopez]

To be honest the XML-RPC code in joomla is not really even baked.

I helped code this http://github.com/plexicloud/Plexicloud-JWS which is installable and has an integration to Jomsocial already. We created it so that component developers can extend it and add to the library.

Take a look. I am going to continue dev on it post 1.6 launch and develop more documentation

Dan

--

[Herman Peeren]

On Sep 24, 4:35 pm, Dan Lopez <danlope...@gmail.com> wrote:
> To be honest the XML-RPC code in joomla is not really even baked.

There was indeed no xmlrpc-code published yet. On July 2 Louis has
sent me a 11KB zip with his raw xmlrpc-code. As said, I had no time
yet to develop it further (probably, he neither).

The JWS-initiative is very interesting. At the moment it generates an
API-key and secret instead of using joomla user/password. However they
do intend to modify this functionality in the next release by using
the joomla username and password. Nicholas (Akeeba) is working on a
JSON-based solution too (with his own authentication API). Rafael
(Dioscouri) is working with SOAP (which is natively available in
PHP5). Nooku is working with REST and has done quite a lot allready
(but not on authentication/authorization). I've been using some custom
solutions for data-exchange between Flash/Flex and Joomla!. In this
thread we see Oli looking in which direction he will work on a
solution. And there probably will be dozens of other initiatives. For
connecting Joomla! with other applications will become more and more
wanted.

What I would like, is a basic framework solution upon which we can all
build further, instead of all of us reinventing the same wheel. A
solution that uses methods for authentication and authorization that
can be used in all extensions throughout the CMS *and* with
webservices. A general, basic solution: that is exactly what a
framework is meant for. Authentication and authorization are basic
tasks and should be built very carefully: if it is done well, many
future security issues will be avoided. I was happy to read that JWS
was now also looking in the direction of using the Joomla! username
and password.

It might be a good idea to join forces.

[Dan Lopez]

Comments inline. Thanks for the feedback by the way...!

On Fri, Sep 24, 2010 at 10:54 AM, Herman Peeren <herman...@gmail.com> wrote:

On Sep 24, 4:35 pm, Dan Lopez <danlope...@gmail.com> wrote:
> To be honest the XML-RPC code in joomla is not really even baked.

There was indeed no xmlrpc-code published yet. On July 2 Louis has
sent me a 11KB zip with his raw xmlrpc-code. As said, I had no time
yet to develop it further (probably, he neither).

Yes. However, XML-RPC has many limitations, and currently does not interact outside joomla core articles. So you are stuck either building it out to your own specs with XML-RPC limitation that it itself has, or roll your own. We rolled our own and it works like a champ.
 

The JWS-initiative is very interesting. At the moment it generates an
API-key and secret instead of using joomla user/password. However they
do intend to modify this functionality in the next release by using
the joomla username and password.

This is correct. However, we have debated the use of api keys, and it seems it is the industry practice, and we see it extending an API externally for developers to access parts of your site. Our integration with Jomsocial points in this direction where a user requests an API key and receives a key and passcode. They use this to build say an iphone app on their own or a mash up. Or you can restrict this to only the owner of the site has a single key and integration to business intelligence, CRM or ERP system is rather straight forward.

 

Nicholas (Akeeba) is working on a
JSON-based solution too (with his own authentication API). Rafael
(Dioscouri) is working with SOAP (which is natively available in
PHP5).

SOAP is slow on a big site, and this is proven say with SugarCRM SOAP interface.
 

Nooku is working with REST and has done quite a lot allready
(but not on authentication/authorization).

Nooku REST is promising and the likes of Daniel Chapman and myself included think the Nooku framework is promising and should have been incorporated into core (at least something similar) for a long time.
 

I've been using some custom
solutions for data-exchange between Flash/Flex and Joomla!. In this
thread we see Oli looking in which direction he will work on a
solution. And there probably will be dozens of other initiatives. For
connecting Joomla! with other applications will become more and more
wanted.

What I would like, is a basic framework solution upon which we can all
build further, instead of all of us reinventing the same wheel. A
solution that uses methods for authentication and authorization that
can be  used in all extensions throughout the CMS *and* with
webservices. A general, basic solution: that is exactly what a
framework is meant for. Authentication and authorization are basic
tasks and should be built very carefully: if it is done well, many
future security issues will be avoided. I was happy to read that JWS
was now also looking in the direction of using the Joomla! username
and password.

Again, username and password is possible, but think Facebook App. You login, and request your own credentials. An admin can revoke your api key at anytime.

 

It might be a good idea to join forces.

I am all for it. I will be relaunching Plexicloud.com next week with a dev area to release more enterprise features and code enhancements (some / most GPL free , some GPL commercial with support), and may rely on github too. I am open to thoughts there too.



 

[Brad Gies]

Herman... I couldn't agree more. The authentication and authorization
should be in the Joomla core for all extensions/webservices to use.
Maybe also some kind of a basic (virtual/abstract?? not sure what the
PHP term is) class for webservice builders (maybe a descendant of
JController? ) to extend with the details of their protocol, although
I'm not sure it's needed because in 1.6 you can already use the
view.html.php, view.pfd.php, view.json.php patterns... so why not just
open that up to view.soap.php etc. or any file/mime type? Then the
controller just needs to call some method to do anything specific to
that protocol before (and maybe after) calling the view.

I'm not sure I'm being clear... but maybe all that is needed is a few
small extensions of JController, like JSOAPController..
JXMLRPCController.. JRESTController etc... You, Louis, Andrew etc. would
know more about that than I do :).

Sincerely,

Brad Gies
-----------------------------------------------------------------------
Bistro Bot - Bistro Blurb
http://bgies.com
http://bistroblurb.com
http://ihottonight.com
http://forcethetruth.com
-----------------------------------------------------------------------

Everything in moderation, including abstinence

Never doubt that a small group of thoughtful, committed people can
change the world. Indeed. It is the only thing that ever has - Margaret Mead

[Herman Peeren]

@Dan: very interesting. I'll investigate a bit more on API-keys. I
hope we will talk about it further in the future.

I've got 2 main points concerning authentication and authorization:

1. I don't see a real difference (yet) between logging into an admin-
side of a website via a browser and logging in as a webservice. As it
is not good to have different code for similar operations, I think it
is good to look for a common way to login. The use of API-keys is
coming forth from the need not to share your login-credentials but
still give other applications (limited) access to your data and data
manipulation. What are limitations and (dis)advantages of both
methods? I think this needs some investigation and I definitely want
to learn some more about this. Also see:
http://blog.sonoasystems.com/detail/do_you_need_api_keys_api_identity_vs._authorization/

2. I see some initiatives of people who make an identification (via an
API-key or via their own username/password). I might be wrong, but I
have the feeling that at the moment different people are coming with
their own custom solution and we might end up with several systems. We
might have a look if there is a framework-solution possible, that can
be used by everyone. That, and the fact that there is a login via
username and password allready available right now, makes me think
that the optimal solution is to use what is allready available, if
that is possible. Combine that with the new ACL-possibilities: how
does your API-user cooperate with rights given to groups and users?

I mostly see advantages and disadvantages in everything. That holds
for protocols (XML-RPC, SOAP, REST, your own) and formats (XML, JSON,
HTML, AMF, your own) too. I don't agree with the limitations you
sketch that would be inherent to XML-RPC. And I've seen some great
examples where the extensive metadata of SOAP really touched the
ideals from the starting-era of webservices: that applications can
communicate with each other, without the need of human interference. I
understand the ideas of REST to come to a limited "uniform" interface
and to avoid duplication of the verbs allready available in the HTTP-
protocol (like PUT and DELETE), but simpler is not always better:
metadata can be of great value and in that respect REST is a step back
IMHO. That is why I strive for multiple possibilities to connect
Joomla! to the outside world.

@Brad: Yep, that is exactly what I envision: extending the controller
to get the data that are being sent by the client, then use a common
model and get the output in some format via an extended view. In Nooku-
framework something similar is done too, with different controllers
and views (with a default), extending form an abstract view and
controller. I think authentication and authorization should be
provided in such an abstract controller as much as possible.

[Herman Peeren]

Joomla!: knitters and architects

There are two different approaches to development in a software
project: expanding existing code or restarting from scratch. Both have
their pros and cons.

It is good to use what has allready been done. So, if you want
something extra, you add it. This is the main approach that is used
for Joomla! 1.6 (and 1.7 etc.): we keep what we have as much as
possible and create something extra to expand it. I call it
"knitting". In itself nothing wrong with it. It is the best method to
get a quick result (which can still take some time anyway...). But
after several attachments a knitwork can become a patchwork. Then it's
time for a major revision.

When you start from scratch, having learned from what has been done in
the past, you can thoroughly lay out a plan, an overall structure, a
fundament. This is "architecture"! It takes more time than "knitting",
but the result is more future-proof. This is what is needed for
Joomla! 2.0.

In 1.5 and 1.6 we have mainly 3 systems of access control at the same
time:

1. We have a restricted administrator-folder. Files under that folder
can only be reached from outside if you are logged in (with the right
access level), otherwise you'll be redirected to the login-page.

2. you'll have to have the proper rights to create, edit or delete
content. It is mainly this part that has been expanded with the new
ACL in Joomla! 1.6.

3. Read-access, wether you can view something, is mostly arranged by
restricting menu-items and/or content to certain access-levels.

I see this as a patchwork and the only way it can be thoroughly solved
is by solid architecture. Now I understand, that that will not be done
in a minor release, as 1.6 is. It is something for Joomla! 2.0. I
plead to start the development of a major release concurrent to the
development of minor releases: it is a different thing. The
architectects can do their work while the knitters continue.

Now back to webservices. What I see in some approaches now, is that
developers create their own system for authentication and
authorization. So, apart from the 3 systems of access control we
allready have in Joomla! they make a fourth. And it is understandable:
you want your webservice up and running as soon as possible, clients
are waiting, so there is no time for solid architecture; take your
knitting needles and attach what you need.

Maybe we can get some people together and work out a good architecture
for Joomla! 2.0, with an emphasis on access control. That can be done,
as said, concurrently with "knitting" on for 1.6, 1.7, extensions,
webservices etc. But I have the vague feeling, that the patchwork of
existing code is so bad at the moment, that a reorganisation that
could be done in a major release would save more time than it costs;
in that case 2.0 could be released earlier than 1.7.