Library Management

155 views
Skip to first unread message

Jeremy Wilken

unread,
Feb 18, 2012, 3:33:37 PM2/18/12
to joomla-de...@googlegroups.com
I have been talking with Joe LeBlanc about this, and let us ignore the challenges for the moment and focus on the concept of managing libraries. This is also to avoid using plugins for tasks that are more properly suited for a library, which has been abused, to avoid duplication of code in extensions, and to provide a more standard and robust API for extensions.

We both feel the future of Joomla should be focused on being able to share common libraries, allow for extensions to have dependencies, and stop reinventing the wheel. If we look at Zend Framework, it has a lot of libraries that are optional, standalone packages (or have dependency on a couple others). 

Problems: Plugins should respond to events, which is all they should do. However for years they have been abused as libraries. We see a lot of helper folders, which are ambiguous classes that don't fit in MVC but provide some extra functionality that is needed, and these could very often be abstracted into libraries.

Example: We can also look at Joomla and see the captcha plugins, and see a use case. Currently only ReCaptcha is setup, but any number of captchas could be coded as plugins. We disagree they ought to be plugins, since captcha verification should act like JRequest::checkToken() does on requests and isn't evented. Rather, it would make sense for a Captcha library, with subclasses for specific types of captcha, to be made accessible for applications in their data-validation process. JRequest::checkCaptcha() for example, and I know JRequest is going away but its easier to write for the moment.

What would be practical, is if libraries could be optionally installed only when required. The core CMS uses Captcha, so it would be included. The CMS does not use Github, so it should not be included. I can think of dozens of libraries that could be developed for minor use cases (and developed by the community), and that could be used commonly throughout. oAuth, REST, Imagick, are just a few ideas. It could even be possible to generate additional JController subclasses for folks who have a different use case. We have the ability to manage libraries to some degree, and a challenge will be finalizing the specs for exactly how they should be managed with additional things like dependency requirements.

Then we would need a centralized library directory, that can be accessed during installation to provide the dependencies. Think gems for Ruby or PEAR/Pyrus for PHP. Our libraries in Joomla should be able to accomplish this too. I think the easier part will be to build the structure to provide for it, the more difficult aspect will to get the buy-in from Joomla developers and get them on board to use it.

Is this on the radar of the Platform team? I realize there are challenges, but first I want to focus on the need for this.

Louis Landry

unread,
Feb 18, 2012, 3:43:24 PM2/18/12
to joomla-de...@googlegroups.com
The short answer from my end Jeremy is that yes, this is on my radar at least.  I know that Ian, Andrew, Rob, Sam and I have all discussed it at varying times ... and I have perhaps forgotten others.

It may or may not be obvious but with the platform project we have been deliberately moving more in this direction.  There are questions about how dependency management is handled, there are questions about a great number of things actually... but this is certainly a direction that most everyone I've talked to is wanting to go.  My initial thought is to organize things at a package level, and have the packages be their own atomic units that can be installed.  I think we can achieve this fairly quickly and it would provide a great deal of flexibility around some of the challenges you've touched upon.

Since the beginning of this platform project concept I've said that the breadth of what will exist in this project won't be useful for the CMS ... and it should be up to the CMS to decide what packages it will include in it's distribution.  This would hold true for any application built on the platform.  Here's hoping we can make some great strides in that direction soon.

- Louis

Andrew Eddie

unread,
Feb 19, 2012, 6:37:24 PM2/19/12
to joomla-de...@googlegroups.com
Not really much more to add. It's certainly on the radar and will
take time as we continue to ween the Platform off the CMS. It's
something I'm keen to see rationalised.

Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla developers

Gary Glass

unread,
Feb 19, 2012, 7:29:45 PM2/19/12
to joomla-de...@googlegroups.com
So I'm curious as to whether you're thinking of a "pull" system which would make it easier for extensions to request common services or a push system that allows us to do dependency injection a la something like Spring? Or both? Or neither?

__________________________
Gary Glass

On Feb 19, 2012, at 18:37, Andrew Eddie <mamb...@gmail.com> wrote:

> Not really much more to add. It's certainly on the radar and will
> take time as we continue to ween the Platform off the CMS. It's
> something I'm keen to see rationalised.
>
> Regards,
> Andrew Eddie
> http://learn.theartofjoomla.com - training videos for Joomla developers
>
>
>
> On 19 February 2012 06:43, Louis Landry <louis...@gmail.com> wrote:
>> The short answer from my end Jeremy is that yes, this is on my radar at
>> least. I know that Ian, Andrew, Rob, Sam and I have all discussed it at
>> varying times ... and I have perhaps forgotten others.
>>
>> It may or may not be obvious but with the platform project we have been
>> deliberately moving more in this direction. There are questions about how
>> dependency management is handled, there are questions about a great number
>> of things actually... but this is certainly a direction that most everyone
>> I've talked to is wanting to go. My initial thought is to organize things
>> at a package level, and have the packages be their own atomic units that can
>> be installed. I think we can achieve this fairly quickly and it would
>> provide a great deal of flexibility around some of the challenges you've
>> touched upon.
>>
>> Since the beginning of this platform project concept I've said that the

>> breadth of what will exist in this project won't be useful for the CMS ....

Emerson da Rocha Luiz

unread,
Feb 19, 2012, 7:56:58 PM2/19/12
to joomla-de...@googlegroups.com
One additional param group on XML install of extensions, like

<dependences>
    <depence type="library" name="name" minversion="1.0" maxversion="" />
    <depence type="plugin" name="name2" minversion="1.0" maxversion="" /> 
</dependences>

And one way to add sources repositories should be sufficient.

emerson
--
Emerson da Rocha Luiz
+55 51 9881-9146  | MSN: emerson at webdesign.eng.br | GTalk: fititnt at Gmail | Skype: fititnt | http://www.fititnt.org | Twitter: @fititnt
Membro do JUGRS | Membro do JCoderBR

Gary Glass

unread,
Feb 19, 2012, 8:38:03 PM2/19/12
to joomla-de...@googlegroups.com, joomla-de...@googlegroups.com
I wonder if there would be value in declaring dependencies as interfaces (canonical names) rather than explicit types.

And also allow dependencies to be dynamically rather than statically fulfilled -- at runtime rather than install time -- i.e. dependency injection.

Or maybe I'm just restating what you already had in mind.

__________________________
Gary Glass

Jeremy Wilken

unread,
Feb 20, 2012, 9:41:22 PM2/20/12
to joomla-de...@googlegroups.com
I've been thinking about this a lot, because I want to tackle this in Square One. I think an entire rethink/overhaul of the current update and installer system is in order, and I am working on concepts and talking with several folks who have issues or are limited with the way it works currently. Dependency management is a major one, and my thoughts are along the lines of what Emerson has already provided. 

<dependencies> <!-- List of requirements to run -->
<dependency element="joomla" type="library" minversion="12.1" /> <!-- Dependency with some options -->
<dependency element="com_othercomponent" type="component" minversion="1.4" maxversion="12.3" manifesturl="http://example.com/othercomponent.xml" /> <!-- Dependency with all options -->
</dependencies>

The way I have it provides a url to grab the dependency, which can work around the need for a dependency repository in the short term, but could very likely keep extension devs from embracing the concept of one library per problem.

How open is the Platform to accepting a refactored library? I plan to do a full outline and explanation of the limitations and goals, in the coming weeks.

Andrew Eddie

unread,
Feb 20, 2012, 10:24:47 PM2/20/12
to joomla-de...@googlegroups.com
On 21 February 2012 12:41, Jeremy Wilken <gnom...@gnomeontherun.com> wrote:

> How open is the Platform to accepting a refactored library?

Always open. I guess the thing to think about is the Platform is
trying very hard to get off the "CMS" so changes to the installer
should be fairly low level and generic, and I think we'd move a lot of
the custom adapters to the CMS tree (would have to think more on that
one). An example of that is how would one install something via a web
services call (regardless of whether something is a CMS component, or
some hitherto unknown entity, heck Drupal could be connecting to the
API for all we care, the package for installing something should be
able to take care of all the base work "somehow"). The key is working
out what it's supposed to do so that would be great if you can do that
Jeremy.

To answer Gary's question, I think we are all looking at both pull
systems (PEAR, phar, etc) and composition. If you look at the changes
in the application package, you'll see how this is heading with
respect to composition. And yes, interfaces have an ever growing part
to play. You'll see a bit in the UCM as well as the database refactor
branch that is being tested at the moment.

Regards,
Andrew Eddie

Elin Waring

unread,
Feb 20, 2012, 10:52:19 PM2/20/12
to joomla-de...@googlegroups.com
This is really partly what my question about dependency tracking in the UCM is partly about. Some how those need to be tracked and documented even when they are beneath the surface (like the indirect assets dependency in the JContent package). Which means you need JTableNested too.  We actually need something like a graph with each of the -- not sure yet what unit, packages? Classes? -- libraries as nodes and then could use some connectivity calculations to find the dependencies.

I also think that's why to think about where to put sql meta files. Maybe it actually makes sense to put them in the Table package to the extent that makes sense. Again too it's important to understand the FKs in tables in the core (as well as possible places where at the moment there are common fields that are not mandated to have the same definitions. 

Elin

Gary Glass

unread,
Feb 21, 2012, 6:06:40 AM2/21/12
to joomla-de...@googlegroups.com, joomla-de...@googlegroups.com
This sounds a lot like Maven. There is a Maven for PHP project. I wonder if there's some useful work there that you could draw on.

__________________________
Gary Glass

Scott

unread,
Feb 21, 2012, 9:38:13 AM2/21/12
to joomla-de...@googlegroups.com
Take a look at Composer ( https://github.com/composer/composer ).  Symfony2 is moving to using this as its package/dependency mangement system for their 2.1 release.

Thanks,
Scott

Jeremy Wilken

unread,
Feb 22, 2012, 5:10:35 PM2/22/12
to joomla-de...@googlegroups.com
To be honest, I could care less if we use composition or pull systems, but composition probably won't always work. If I need access to a library in my component's template file, I'm not going to have a class for composition. You could tie it to the view I suppose, but I bet there are some good situations where it simply wouldn't work or not very elegantly. That isn't the core of what I wanted to bring up though, nor do I follow most of what Elin is saying as it relates to managing libraries. I don't know what UCM has to do with this discussion at the moment.

The challenge is two fold: 1) the code and technical process of managing libraries (not my main topic to discuss yet) and 2) the vision and process for how libraries should interact with the Platform and CMS. The second challenge needs to be considered before any technical process is designed for address the first. Look at the numerous 'helper' classes in core components, they are often repetitive and ambiguous. Do people agree those should be abstracted into libraries and made available? Should such things be part of the CMS or Platform library? Probably the CMS, since the Platform trying to be completely separate. How do we manage two libraries then? Should people be able to install supporting libraries into those libraries? How can we try to limit library duplication? What is the process for making a library and how should people find ways to share common dependencies? These are the kinds of questions to discuss first. I mentioned the current library for install/update likely needs an overhaul, but that is based on the assumption that the best plan of attack for this process requires us to rethink the way we do things now.

Composer is interesting, I believe Drupal is also considering taking it as well. Its not GPL, and its designed as a PHAR and I can't quite see how easy it would be to pluck out the pieces that we'd need (it has libraries for things the platform already has). Though it is a set of libraries available, and would limit needing to reinvent the wheel, I'm not sure without closer inspection how it would fit into Joomla and with the Joomla community.


Elin Waring

unread,
Feb 22, 2012, 8:05:10 PM2/22/12
to joomla-de...@googlegroups.com
I guess I didn't really understand the point of your post then, I thought it was about managing libraries and the possibility of not having to install all the libraries when you build an application. I mean it included a lot of other stuff too that was related to the CMS, but that was the first paragraph and subject line and that was what I was responding to.

This thread is on the platform list and is of course  thus not really  about the CMS (I assume you would have posted on the CMS list if talking about the CMS  was the goal) but about how all possible applications built on the platform will be able to work. So I take the question as whether the expectation is that they will always include the complete platform, build their library collections selectively by hand, or be able to selectively choose the relevant libraries only in some automated way that might even be integrated with IDEs.

My example of the UCM is this. Say I am building an application that uses JContent. I can't just take the content library and build my application. I mean, a. the code contribution itself falls into 3 separate packages all of which are needed  and b. it actually relies on session and cache handling which are separate libraries and  JUser and JForm. So it's not enough to say that you can just pluck one package out of the platform. If you pluck the content package you need to also pull the database package, the session and cache packages, the access library, the authorisation library and the user library and the forms library. Do you need the image library? No. I won't list out all the other things you don't need.   It would be nice to have a way of knowing exactly what those dependencies are was my point.

 The CMS is an example of a  highly complex set of three interacting platform applications that until the present has shipped with all the libraries. It might be the case that the CMS team decides for 3.0 to be more selective or they might decide to ship the whole thing, but that is a down stream decision not anything to do with the platform per se.

As an example application the CMS does raise some interesting challenges.

Imagine what happens if the CMS doesn't ship with essentially a full platform. It has an extension developer community of say several thousand people and if some of those people are building on the 12.1 versions of older libraries from the platform and some on the 11.4 and a webmaster--the end users of the cms (whereas developers are end users of the platform)-- tries to install both extensions that each attempt to install the relevant version of that library.  That's going to create problems that make multiple jquery versions look like a walk in the park. 

As of late February 2012 it makes sense to me for the CMS (or any application which is itself extendable) to say: this is CMS 2.5 and these are the supported versions of the platform libraries period. Overwrite them and you are on your own.  And what if the CMS bug fixes parts of the older libraries that don't even exist any more? You can't just pull from the 11.4 tag. Are we supposed to say to those people developing for 2.5 sorry no bug fixes the platform team has moved on? That's a huge ecosystem of people whose livelihoods that would be  totally disregarding just when huge parts of it are just now transitioning to using 2.5/11.4.

How the CMS team chooses to manage the true CMS libraries and the standards for accepting new CMS libraries that aren't simply migrating from the platform  is really a discussion  at the application level and really belong on the CMS list unless you think that there are things there that really belong in the platform. 

Elin

Jeremy Wilken

unread,
Feb 22, 2012, 9:50:42 PM2/22/12
to joomla-de...@googlegroups.com
@Elin - I look to the CMS as an example for the Platform. My example of the helper classes in the core is just an example. No matter how you slice it though, the CMS and Platform are still tied together in ways so its unfair to pretend they are completely separate. I know thats a goal, but its not the current reality. For this discussion I don't care if the CMS ships with only a portion of the Platform and those other issues raised, the question is how should it integrate and handle additional packages inserted into the library. 

The focus of my question is on how other applications (the CMS, or otherwise) should integrate their own libraries, and what weaknesses exist in the current system, what functionalities it should have, etc. The CMS has the ability to install libraries (though they are separate libraries, and not packages existing inside of the Platform), so its a practical way to see how this works now, vs how it could work. My initial post describes several situations where (using the CMS as an example) the current packages for installation/updating fall short of solving some problems.

Amy Stephen

unread,
Feb 22, 2012, 10:39:57 PM2/22/12
to joomla-de...@googlegroups.com

Agree w this direction.

Andrew Eddie

unread,
Feb 23, 2012, 1:51:41 AM2/23/12
to joomla-de...@googlegroups.com
Replying inline.

On 23 February 2012 08:10, Jeremy Wilken <gnom...@gnomeontherun.com> wrote:
> To be honest, I could care less if we use composition or pull systems, but
> composition probably won't always work. If I need access to a library in my
> component's template file, I'm not going to have a class for composition.

Well that depends on what you are doing but there are definitely parts
of the API that are not intended to be accessed from other parts of
the API. Funnily enough, I think template files are one of the most
abused areas of the developer world because anyone thinks they can do
anything in them … and they do. There is certainly recent work where
we've made it easier to access some data, and harder to access other
data (for example, we tend to default to private variables unless
there is a good case for making them protected, and on rare occasions
public). The net result is that it should be easy to do what you are
supposed to do, and hard to do what you are supposed to be doing in
any given area of the code.

> The challenge is two fold: 1) the code and technical process of managing
> libraries (not my main topic to discuss yet) and 2) the vision and process
> for how libraries should interact with the Platform and CMS.

I cannot stress this enough that you need to think about the Platform
as a basis for any application that someone wants to build on it.
Yes, it was born from the CMS, but it doesn't exist for the pleasure
of just the CMS. That's just one use case - there are many others.
The vision is already much wider than the CMS.

> The second
> challenge needs to be considered before any technical process is designed
> for address the first.

I don't think it figures into the equation at all because library
management applies equally to the CLI developer as it does to someone
writing a game services platform for the iPhone (i.e., no CMS
involvement).

> Look at the numerous 'helper' classes in core
> components, they are often repetitive and ambiguous. Do people agree those
> should be abstracted into libraries and made available?

It depends on which helper you are talking about. The current trend is
to use them less and get away from static usage.

> Should such things
> be part of the CMS or Platform library?

Again, it depends which helpers you are talking about. If you can
specify which ones I can give you some guidance on what might be done
with them.

> Probably the CMS, since the Platform
> trying to be completely separate.

That's not quite accurate. The Platform is trying to devolve itself
of CMS "specific" code that is only useful to someone working in the
CMS (JToolbar, in its current form, is a good example).

> How do we manage two libraries then?

It's already being handled quite nicely thanks to a recent change in
the auto-loader by Louis. See
http://developer.joomla.org/manual/ch01s04.html

You can include any number of libraries that you want that follow the
auto-loader convention (and there are ways around it if they don't
follow convention). However, if you want to add in something like
Zend Framework 2, you are probably on your own though.

> Should people be able to install supporting libraries into those libraries?
> How can we try to limit library duplication? What is the process for making
> a library and how should people find ways to share common dependencies?

I suggest to you that these questions can only be answered on a
case-by-case basis. If you give me an example, I can give you an
opinion about how it might be handled. Though, an umbrella answer is
that if all developers are thinking about what they can contribute
back to the platform core, it makes the discussion a lot easier.

> These are the kinds of questions to discuss first. I mentioned the current
> library for install/update likely needs an overhaul, but that is based on
> the assumption that the best plan of attack for this process requires us to
> rethink the way we do things now.

Do you mean in the sense of install/update like is done in the CMS or
install/update as a standalone platform not tied specifically to any
platform?

Regards,
Andrew Eddie

Emerson da Rocha Luiz

unread,
Feb 23, 2012, 3:56:14 AM2/23/12
to joomla-de...@googlegroups.com
If code will be on libraries/cms or libraries/joomla is not really a problem if it, at least, is in some of they.

And about CLI aplications, the JPlatform can maybe do not force use of database to handle where are repositories and cache of they. Since is not need for CLI aplications fix database, and does not need care about one beauty interface with opinion of others, should be a bit more easy. Even if not ready for CMS 3.0, could be release as undocumented feature that only works on CLI, and, because is not oficially released, could be change a lot before of CMS 3.5. 

But... well, 3.0 should come with more drastic changes than 3.5, so or we put this feature on 3.5, or be ***really*** sure that will be stable for 3.5. I'm not sure if this kind of feature tend to be  well tested by only bug squad members.

emerson

--
Emerson da Rocha Luiz
+55 51 9881-9146  | MSN: emerson at webdesign.eng.br | GTalk: fititnt at Gmail | Skype: fititnt | http://www.fititnt.org | Twitter: @fititnt
Membro do JUGRS | Membro do JCoderBR



Andrew Eddie

unread,
Feb 23, 2012, 4:36:18 AM2/23/12
to joomla-de...@googlegroups.com
On 23 February 2012 18:56, Emerson da Rocha Luiz

<eme...@webdesign.eng.br> wrote:
> If code will be on libraries/cms or libraries/joomla is not really a problem
> if it, at least, is in some of they.

Just to explain, the /libraries/cms folder in the Platform is
temporary. It's there to show the CMS what we moved so they can more
easily put it in their tree. Just before the release of 12.1 we'll
tag the repository and then remove the /libraries/cms folder (the tag
means anyone can go back and double-check what was removed), and then
the official (and lighter) version of 12.1 will be tagged and
released.

> And about CLI aplications, the JPlatform can maybe do not force use of
> database to handle where are repositories and cache of they. Since is not
> need for CLI aplications fix database, and does not need care about one
> beauty interface with opinion of others, should be a bit more easy. Even if
> not ready for CMS 3.0, could be release as undocumented feature that only
> works on CLI, and, because is not oficially released, could be change a lot
> before of CMS 3.5.

Actually, a lot of CLI's that I've written do support database work.
Yes, you can certainly look at the Joomla Platform as a basket of
undocumented features in whatever application is built on top of it -
that is, until we get the developer manual finished (not so subtle
hint for people to help!). That's the beauty of the separation of the
Platform from the CMS - new developer features can be added much more
quickly.

> But... well, 3.0 should come with more drastic changes than 3.5, so or we
> put this feature on 3.5, or be ***really*** sure that will be stable for
> 3.5. I'm not sure if this kind of feature tend to be  well tested by only
> bug squad members.

New platform features, with few exceptions, need to come with
automated tests. That doesn't mean there aren't bugs, but it makes the
code base a lot more reliable.

Thanks for your comments :)

Elin Waring

unread,
Feb 23, 2012, 8:16:06 AM2/23/12
to joomla-de...@googlegroups.com
@Andrew,

I was actually thinking about toolbar as an example in response to Jeremy, and have been thinking about it a lot. It seems to me that web applications in general do fairly often need a way to create a row of icons that do tasks. Obviously the current toolbar code is very specific and you've been using it as a prime example of what will be removed, but if you do want to  allow for rapid application development it seems to me like some toolbar-like-thing is really needed. So I was wondering how you are thinking about that. One idea I have had is using the examples repo or a new repo as a way to have some ready to go implemented building blocks for applications. An icon bar would be an example, although what really got me thinking about this is authentication and wondering if it is really necessary for every new to joomla developer to have to implement from scratch when all they really want to do is take advantage of jForm or Jimage in a pretty simple application.

Sorry for the semi hijack but  I do think it is related if we are thinking about how application developers interact with the platform.

Elin

Amy Stephen

unread,
Feb 23, 2012, 1:26:09 PM2/23/12
to joomla-de...@googlegroups.com

On Saturday, February 18, 2012 2:33:37 PM UTC-6, Jeremy Wilken wrote:
Example: We can also look at Joomla and see the captcha plugins, and see a use case. Currently only ReCaptcha is setup, but any number of captchas could be coded as plugins. We disagree they ought to be plugins, since captcha verification should act like JRequest::checkToken() does on requests and isn't evented. Rather, it would make sense for a Captcha library, with subclasses for specific types of captcha, to be made accessible for applications in their data-validation process. JRequest::checkCaptcha() for example, and I know JRequest is going away but its easier to write for the moment.



Molajo has a "services layer" for a couple of reasons. First, to isolate the application layer from change in the underlying framework, and second, to provide an integrated, simplified, stable interface to the API for frontend developers (in other words, bury some of the functions they are not likely going to need, bring forward what will be more helpful, and keep the API stable, even if subsystems are swapped out.)

Adding and removing services is simply a matter of defining another entry in the services XML file. During the initialization of the application, MolajoServices creates a static instance of these various service classes which are then made available to the application via a proxy class.

Developers can then use a similar approach to filtering: Services::Security()->filter('field', 'email') as is used for string translations: Services::Language()->translate('language-string') as is used to verify permissions for Services::Access()->authoriseTask('edit', 3).

This layer adds a cushion between the application and the framework that should help ease the impact of change and support more of a plug and play environment.

The code is pretty simple, basic. Mainly, I just wanted to share the concept that such a layer, combined with better isolation on the component elements that Louis indicated earlier, should give the platform team a little wiggle room to continue to advance this software while buying your users better protection from the difficult cost that comes from change.

If services are well organized, it should also make it easier to add services in a way that supports reuse and collaboration. Jeremy's example of captcha is a good one and his point about what plugins should and should not do is right on the money.

Certainly willing to share code and help if there is interest in this approach.

Links:
Services class - https://github.com/Molajo/Molajo/blob/core/applications/base/services.php
services.xml - https://github.com/Molajo/Molajo/blob/core/applications/options/services.xml
Proxy - https://github.com/Molajo/Molajo/blob/core/applications/includes/aliases.php#L64
 
Reply all
Reply to author
Forward
0 new messages