API Reference Documentation

[Chris Davenport]

I've been thinking about where we might want to go with the API documentation in the near, and not so near, future.  Although http://api.joomla.org has served us well over the years, the Joomla API documentation has long been in need of improvement.  In particular, our goal of supporting and encouraging user-contributed notes and code samples has proved somewhat elusive although some good progress has been made recently.  Since the API documentation is important for developers (obviously) and given our intention to launch a new developer website, I'd like to get some feedback from the rest of the team on where we want to go with the API documentation.

Where we are now
phpDocumentor is good at producing a comprehensively cross-referenced and good looking resource (although the code is a big pile of spaghetti), but it doesn't produce predictable URLs (because the URL includes the sub-package name and you can't deduce the sub-package name from the class name).

We also have manually-written pages starting from here: http://docs.joomla.org/Framework.  Although there is some useful information in there they suffer from being, well... manual.

Doxygen is simple and fast and produces predictable URLs, but doesn't produce quite such nice output.  However, Alex Bachmann has written Doxiki which can update the wiki directly from Doxygen output, although I would think twice about running it nightly as the wiki keeps a full version history and I can see that getting huge.  I need to do some research on whether it is possible to easily delete an entire namespace together with its version history as this would make things a lot cleaner.  You can see the output from Doxiki here: http://docs.joomla.org/API15:Framework and here: http://docs.joomla.org/API16:Framework

We also have an extension to the wiki installed which supports addition of comments and code samples.  A combination of namespacing and tagging means that comments are not overwritten when a fresh run of Doxiki is made.  Furthermore, a new version of Joomla will automatically carry forward the comments from earlier versions and tagging can be used to refine that.  With some more work we can get the output from Doxiki to look good and make it more obvious that users can add comments.

Goals

• Provide rich access to API information.
• Support for comments and user-contributed code samples (php.net does this particularly well).
• Predictable URLs for classes and methods.  This will make it possible for tools such as Eclipse to automatically retrieve API documentation pages.
• Must be able to cope with different Joomla versions.  That is, major and minor releases should have separate documentation areas.  Maintenance releases can overwrite.
• Some level of multi-lingual support must be available.
• Automate as much as possible (continuous integration, etc.)
  

Proposal
I'd like to propose the following for our API documentation going forward.

• Generate phpDocumentor output at developer.joomla.org/API/1.6 (okay, this clashes with the URLs suggested below so we'll need to resolve that issue) triggered by Hudson.  This is already being done.
• Embed links in the code itself (using @link) to predictable URLs.  Something like: @link http://developer.joomla.org/API/1.6/className/methodName.  This is a one-time task that is easy but time-consuming.  We won't need to change these except when a new major or minor version is started and then we can do a simple find and replace on them.  These URLs need to be highly stable over a long period of time so as to encourage developers to make use of them in tool integrations (eg. Eclipse).
• Use a component in http://developer.joomla.org to serve transcluded wiki pages from these predictable URLs.  So http://developer.joomla.org/API/1.6/className/methodName will transclude http://docs.joomla.org/API16:className/methodName.  I think this can be done using the wiki proxy component already developed, together with a little custom router.  Need to make sure that the comment extension will work correctly in this setup.
• Further develop the Doxiki code to generate output that makes it clearer that user-contributed notes are encouraged.  Run Doxiki every time a major or minor Joomla release is made.  Maybe more often if lots of changes are being made to the code between versions or if we can solve the problem of the wiki keeping full version history for the Doxiki namespace.
• Links to appropriate pages on http://api.joomla.org need to be entered manually into the wiki pages as the URLs are not predictable.  However, this can be done by adding the links in the See Also section of the doxiki-generated pages which works the same way as comments, so it only has to be done once and it doesn't get overwritten by later versions.
  

So in effect we end up with two resources: the output from phpDocumentor in sub-directories under the new developer website and the output from Doxygen in wiki content pages with commenting, but transparently also available in the new developer website.  Each of these resources contains cross-references to the other.  Plus, we pave the way for tools such as Eclipse to integrate cleanly into the documentation.  This arrangement also helps focus people on the developer site for this kind of information.

Thoughts?  Comments?  Ideas?

Regards,
Chris.

--
Chris Davenport
Joomla Leadership Team - Production Working Group
Joomla Documentation Coordinator

[Andrew Eddie]

First off, thanks Chris for the thought put into this. I know it's not
in the low-hanging fruit region to solve.

Maybe I'm not joining the dots properly but that sounds like an awful
lot of double handling.

Can't the commenting engine/service be incorporated in the
phpDocumentor template/output itself (we could even use Disqus or
something similar)?

Do the help integration tools in Eclipse work for more than one site?
As far as I'm aware you only get one choice so to put in a crafted
link for the Joomla API, you'd loose your PHP help (unless I'm
mistaken) - at least in phpEclipse (not sure on PDT). I'm just not
sure the extra @link tags need to be a priority (would indeed be a
heluva lot of work).

The only thing I'd add to the "Goals" is that the API docs should be searchable.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

> --
> You received this message because you are subscribed to the Google Groups
> "Joomla! Production Working Group Leadership" group.
> To post to this group, send email to joomla-wg-...@googlegroups.com.
> To unsubscribe from this group, send email to
> joomla-wg-produc...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/joomla-wg-production?hl=en.
>

[Louis Landry]

I want to start by saying that I think this is a great idea.  As most of you will know I am pretty big in favor of getting our API docs into a setup like the php.net documentation.  

This is a pretty good step towards that.  With respect to being able to search the documentation, if it is proxied from the wiki and that data is stored in a local database table we can easily index that and search it.  That is actually a pretty solid vote in the non-phpDocumentor route.

The other option I can see is to come up with a way to have phpDocumentor spit out data in some format where it can be imported into a Joomla content category/article structure and we can go from there.  I'm not sure that is a simple thing to do, but would be interested to hear your thoughts on something like that.

Either way I'm excited about the path, and I think Doxygen is probably our future... if we can wrangle all the output options.

- Louis

--
Development Coordinator
Joomla! ... because open source matters.
http://www.joomla.org

[Chris Davenport]

Hi Andrew,

Can't the commenting engine/service be incorporated in the
phpDocumentor template/output itself (we could even use Disqus or
something similar)?

In theory, yes.  However, there are a number of problems with phpDocumentor that make it less than ideal for our purposes:-

• probably the most important, in my view, is that it doesn't generate "predictable" URLs.
• we could use a third-party commenting service such as Disqus, but I would be concerned about having parts of our documentation hosted on a service that is not under our control.
• we could embed our own commenting system within the phpDocumentor output, but I don't think that is a trivial development task.
• adding commenting to phpDocumentor makes it harder to support exporting of the documentation in "static" formats, such as PDF or CHM.
  

Of course, all of these problems are potentially solvable, just not easily solvable, especially given the rather convoluted nature of the phpDocumentor codebase.

 

Do the help integration tools in Eclipse work for more than one site?
As far as I'm aware you only get one choice so to put in a crafted
link for the Joomla API, you'd loose your PHP help (unless I'm
mistaken) - at least in phpEclipse (not sure on PDT).

I wouldn't dream of replacing the PHP help in Eclipse!  No, what I had in mind was something more akin to what happens when you press F3 on a class name.  For example, I wrote a little EclipseMonkey script that binds to my F4 key so when I press F4 on a class name it opens up a web browser view with the class page from the wiki API reference.  So I can consult the documentation without having to step outside Eclipse.  By having a long-term stable API into the documentation, we can build this sort of functionality into any development tool we like.

I'm sure we can also look at generating an Eclipse help file containing the API documentation too.  That's a good idea and something I'll add to my to-do list (don't know much about Eclipse help files). :-)

 

I'm just not
sure the extra @link tags need to be a priority (would indeed be a
heluva lot of work).

Adding @link tags is certainly extra work and is not a high priority.  But it is something that can be done as a rolling background task.  I'm sure that we can find someone (or two?) from the wider community who would be willing to create a patch (or series of patches) to implement that.  If no-one else steps forward, I'm happy to tackle it myself.  Alternatively, I'm wondering if we might even use the Reflection API in PHP to automatically generate a patch.

 

The only thing I'd add to the "Goals" is that the API docs should be searchable.

Yes, you're right, search is an important goal.  Personally, I find that http://api.joomla.org and http://docs.joomla.org are both very well covered by search engines, but there is certainly scope for a "native" search capability.  The native search in the wiki is not ideal and doesn't help at all if you want to search on the developer site.  Within the context of the proposal, just caching wiki API pages to a database table would make it quite easy to add a search plugin to the developer site, provided the cache is primed periodically (credit to Louis for that idea).
 

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

Thanks Andrew, that's great feedback.

[Chris Davenport]

Hi Louis,

Regarding getting phpDocumentor to "spit out data in some format where it can be imported into a Joomla content category/article structure", yes that's certainly possible.  I think the best way to achieve that would be to write a custom "converter" in phpDocumentor which outputs some intermediate data format (possibly some XML, say), then write a little Joomla component to parse that and save it to the content table.

The only real problem with that approach is that writing phpDocumentor converters is not a trivial exercise.  Actually it would be a lot easier to use the output from Doxygen without any modifications and write a Joomla component to import that instead.  That's essentially what Doxiki does, except that Doxiki imports to the wiki instead of to Joomla content.

So would it be better to push the data into Joomla rather than the wiki?  Well, I might be a bit biased here, but I do believe that the wiki is a very powerful tool and having the API data hosted in the wiki opens up more possibilities than we would have if the data were external to the wiki.  For example, we can transclude parts of API reference pages (a class description or a syntax specification, say) or even contributed code samples, into other documentation pages.  We can also go the other way; a comment can transclude part or all of another wiki page, for example.  This ability to reuse documentation pages is one of reasons I love the wiki!

Of course, we could do both.  That is, we could use the output from Doxygen to import to both the wiki and to developer.joomla.org, but I would argue that there really isn't much point in such duplication when we already have a proxy component that can transparently pull data from the wiki into Joomla.

At the end of the day, there's lots of ways to "solve" the problem and they all have pros and cons.  We just need to settle on a strategy that we're all comfortable with.

Great feedback as always.  Thanks Louis.

Chris.

[Andrew Eddie]

Interesting, where do I find out more about Eclipse Monkey?

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

[Chris Davenport]

I came across when I installed the Aptana plugin, but I believe it can be installed separately.  See, for example, http://docs.aptana.com/docs/index.php/About_Eclipse_Monkey

Chris.