Phar-ing the platform

[Andrew Eddie]

Just dropping a note to say I've added some instructions on how to use the platform in a phar file.  It's really useful when you want to consider deploying an app in a standalone form (no need to get people to clone the platform repo or any of that jazz).  It's also advantageous in a project environment because you can have all your developers using the same base stack.

Anyway, have a look here: http://joomla.github.com/joomla-platform/?chapters/introduction.md

Regards,
Andrew Eddie

[Gary Mort]

Oh cool...  I was just looking into that, this will save me a lot of time.

In testing for namespaces I found somewhat interesting that while the "use" invocation is only specific to a single file, the "alias" method works across files.

IE, in a small test I added:

namespace joomla\patform\v12.4 

to a number of files

Then for the autoloader, I setup an oddball if/then clause... ie  based on the classic if classname is uppercase J and namespace is blank:

1) Try to load joomla\platform\v12.4\<classname> 

2) If joomla\platform\v12.4\<classname> exists then:

class_alias(\<classname>, \joomla\platform\v12.4\<classname>)

Note: since I created the autoload method inside a separate namespace, it is very very very important that to include the leading slash to specify namespace routing. 

I was stuck on a method to grab all existing Joomla files, the Directory Iterator was giving me some odd errors - so I put it on the backburner.  With your helpful addition and instructions, I can give it another shot... [my own personal goal here was to be able to use the latest Joomla platform while running on Joomla 2.5.   With judicious testing of the latest classes, I can define aliases only for the files which actually work.

[Gary Mort]

Wow....  this saved me a LOT of time.   So I went ahead and munged the code to allow it to be used for namespaces at a basic level.   See https://github.com/garyamort/packager for source.

I've added an xml attribute for the directory and file imports, "namespace".  Every time packager goes to import files, it will set the namespace parameter for the packager based on the current xml defined keyword - and if namespace is set it prepends namespace [namespace]; to the top of the file.  So it will not override an existing definition, it will just set a namespace if none is set.

Namespace folder contains some demo code to implement it.  It presumes a directory structure of:

\build

\src\lib --> link to joomla platform libraries

\src\libraries --> link to joomla platform libraries

\src\namespace: demo code

loader.php: munged JLoader class, simple string replacement of 'JLoader' with __CLASS__ since autoloadregister uses the string, not the namespaced class, to register the callback.  __CLASS__ will have the correct namespace when it is set or when it is not set.packager.xml : defines the names joomla\joomla\p12_4 just as a demo

import.stub.php - makes a few changes on bare function callbacks to add the namespace to the string - otherwise bad things happen.  Also adds a second prefix match to catch the namespaced classes...honestly this should be a namespace match but hey - it works!

sample_p12.4.php : at the moment just creates a couple of JObjects and dumps them to confirm the correct class was used.

There are some issues which I'll need to deal with.  The main one is that any calls to classes from within the Joomla platform[for example, calls to "throw RuntimeException" and to third party classes like "PHPMailer" - since their not namespaced[ie no slash at the start] - these calls tend to fail.

I made a hacked loader to get around this for now in the loader - but a more elegant solution is needed.

Also any registration of a callback method on a class is problematic - as their passed as strings.  For example, in JLoader::setup() there are calls such as:

if ($enableClasses)

{

// Register the class map based autoloader.

spl_autoload_register(array('JLoader', 'load'));

}

This FAILS since the class name is no longer JLoader but instead joomla/joomla/p12_4/JLoader.  This is relatively easy to fix by replacing it with __CLASS__, aka:

if ($enableClasses)

{

// Register the class map based autoloader.

spl_autoload_register(array(__CLASS__, 'load'));

}

Whats really fun about doing this is that it makes it relatively trivial write programs for Joomla! CMS v 2.5 while using the latest Platform code - so a component written now can be forward compatible with 3.0, 3.1, 3.2, and 3.5 - with only changes to the installer routines needed to get it to work.

Some mucking about to glue pieces together is needed - but still it's better than writing multiple versions - and it's easy to modify later.

For example:

class gmort\joomla\JCryptKey extends joomla\joomla\p12_2\JCryptkey {

}

Here I can use the latest crypt classes in the current 2.5 CMS, and when 3.5 is released and I'm ready to drop 2.5 support, a simple find/replace of joomla\joomla\p12_2\J to J will bring my code into alignment with whatever is installed by default with 3.5

Beautiful!

[Andrew Eddie]

No, no, no.  Don't you know it's our mission to make life as difficult as possible for downstream developers.  You are failing miserably ;)

Sounds great.  Let us know when you've ironed all the kinks out.

Regards,

Andrew Eddie

Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla developers

[Gary Mort]

In case anyone wants to play with this before I get back to it, there are 2 rather easily solved issues:

1) Certain functions[class_exists, call_user_func, spl_autoload_register, etc] can be given a class name as a string variable.  The issue is that when a class is defined inside a namespace, for example 

namespance joomla/joomla/platform the full classname is joomla/joomla/platform/classname not classname - so the string does not match the full classname.  

A solution, which I found on http://rommelsantor.com/clog/2011/04/10/php-5-3-dynamic-namespace-resolution/ is to define replacement functions in the namespace.  Unlike classes, function calls will go through the namespace first then global.  Ie if you call class_exists('JLoader') inside namespace joomla/joomla/platform PHP will try to run joomla/joomla/platform/class_exists then if that fails /class_exists

This means you can catch those offending function calls, check to see if the class variable is namespaced - and if it is not append your namespace to it.

2) Calls to classes in the global namespace from within the namespace fail.  As an example, using my library a call to $mailer=new JMail(); will fail because PHPMailer does not exist[more accurately, joomla/joomla/p12_4/PHPMailer does not exist.  /PHPMailer does].  

For that, an autoloader can be added to the END of the chain, not the beginning.  Any attempts at loading __NAMESPACE__/classname which has failed all other loaders should simply:

return class_alias($class, '/'.$classname, true);

class_alias with autoload set to true will automatically try to load the class - if it exists, it gets aliased to your fully qualified namespace class and it returns true.  If it does not exist, then it returns false.  

While I say the fix is simple, testing it out and such will take a couple hours so I have to do it "when I have time" - so I figured I'd mention it in case someone else wanted to actually use this solution before I get back to it.

[Mark Dexter]

Is this something we will be able to simplify going forward? Perhaps by building this in or something? Thanks. Mark

[Gary Mort]

Ok, got obsessed and went through and made a bunch of changes to my repo:

https://github.com/garyamort/packager

I made a good number of changes to the packager, so I'm not submitting this one as apull request to Louis just yet[though of course he's free to grab em if he wants them].

For the stubfile, I added the ability to define multiple stubfiles and give them each a namespace.  This lets me break up some things and still be functional.

I setup all the functions needed at least for now to mung classnames and such for the global namespace  - so no more custom JLoader needed.

I need to do the reverse now, as can be seen in my sample php application where created a JMail object fails because PHPMailer is not in the correct namespace.

[Louis Landry]

Yay!!! Someone else cares about Phar.  I think this is going to be of critical importance to us going forward.  Gary, with JWC just over, going back to work, and also moving house yesterday I am pretty stacked up right at the moment, but I'll take a look at your stuff ASAP.  Thanks for the interest and the work.

- Louis

[Donald Gilbert]

I for one think the effort here is great - but I gotta say, I don't really care for Phar. I would much rather see the platform move towards separating all the components out into their own repositories, add namespace support, make them installable via composer, add them to packagist.com, and then create a JoomlaPlatform base repository that, when installed, downloads everything. It makes it very nice for developers to cherry pick the libraries that they need, while allowing for great flexibility and growth of the platform. 

This is the approach that the Laravel framework is taking with the migration to it's next major version, Laravel 4. They created a new organization on GitHub that houses all the independent libraries - https://github.com/illuminate. Then, they have a 'foundation' that acts as the base of the framework, and imports all the required illuminate libraries into a cohesive whole. https://github.com/illuminate/foundation/blob/master/composer.json

[Gary Mort]

On Tuesday, November 20, 2012 12:02:34 PM UTC-5, Mark Dexter wrote:

Is this something we will be able to simplify going forward? Perhaps by building this in or something? Thanks. Mark

Building it in to what?  I don't really follow the question...so perhaps if I explain my goal?

 Because the platform is not namespaced currently, it is rather difficult to play some of the namespace tricks I'm using here.   To namespace the platform could be a backwards compatibility pain[though as I've discovered, if your willing to make judicious use of class_alias and the spl autoloader - you can catch attempts to access the global object and dynamically alias the namespaced object to the global root].

I would guess that the only reason this would be "built in" in the future would be if a lot of devs started using the sort of hack I've outlined here to change the underlying platform on running sites.

My goal is create a Joomla library extension...call it "Joomla Platform 12.2 Library"

When installed on a Joomla 2.5 website, it will consist of a single file:

libraries/platform/12.2/joomla.phar

It's a library, it doesn't magically upgrade the CMS to 12.2 - it just means other components can then use it.

Then in MY component controller, I would have[in theory]

namespace gary\joomla\component

require_once JPATH_LIBRARIES.'platform/12.2/joomla.phar';

class SiteconfigControllerConfig extends joomla\joomla\p12_2\JControllerLegacy

 

And from that point on, all my code will be using the latest platform classes and methods - despite running on Joomla 2.5

Any library setup needed to import the settings from 2.5/11.4 will be handled by the single line:

require_once JPATH_LIBRARIES.'platform/12.2/joomla.phar'

Then if I want to release a version of the same application for Joomla CMS 3.2/platform 13.1 in theory I just change it to:

namespace gary/joomla/component

class SiteconfigControllerConfig extends /JControllerLegacy

Which due to the long unique namespace I setup: joomla\joomla\p12_2 it's rather easy to run a global find and replace.

[Mark Dexter]

Hi Gary. Sorry if I wasn't clear. What I was trying to say is that making it super simple to add the platform to a project with one PHAR file (which as I understand it is analogous to a Java .jar file) sounds like a really great idea. I was trying to ask whether we needed to make changes inside the platform to make this as drop-dead simple as possible (again, as simple as adding a .jar file to a Java project). 

Mark

[Gary Mort]

On Tuesday, November 20, 2012 12:51:03 PM UTC-5, Donald Gilbert wrote:

I for one think the effort here is great - but I gotta say, I don't really care for Phar.

The main reason I care about Phar is security.

Everytime I've had to deal with a hacked site, it's always been a real pain to track down all the files affected.  If each member of the libraries folder was a single phar file, I could check the signatures for each library and be done.

Furthermore, because of the way phar files are used, it can make for faster site loads to place all the libraries there.  On linux at least, the phar file will be partially stored in memory - with the most frequently accessed parts in memory and the others on disk - so it saves a lot of disk lookups for the files.  

And mainly, for me, it's a relatively easy way for me to force a namespace onto all the Joomla classes I'm using without breaking anything.

[Gary Mort]

On Tuesday, November 20, 2012 1:17:16 PM UTC-5, Mark Dexter wrote:

Hi Gary. Sorry if I wasn't clear. What I was trying to say is that making it super simple to add the platform to a project with one PHAR file (which as I understand it is analogous to a Java .jar file) sounds like a really great idea. I was trying to ask whether we needed to make changes inside the platform to make this as drop-dead simple as possible (again, as simple as adding a .jar file to a Java project). 

Mark

Ah, then you should be fairly good to go with the original code.  My code just adds a namespace so it can avoid conflicts with other functions.  On the platform side, it really needs to be brought into a namespace to make this as simple as it is in the java world.  

To bring it into a namespace, ideally any calls to standard classes from the global namespace should be prefixed appropriately[ie add \ before the class name].

Including a set of functions such as https://github.com/garyamort/packager/blob/master/src/namespace/ns.utility.functions.stub.php    will deal with any issues in changing namespaces[assuming of course that you are using one namespace for the entire platform...if your assigning a namespace to each and every file it will be messy].

After playing with it for a bit, I should have a better idea of where things break for me and how they have to be changed.

[Donald Gilbert]

I would say it's rather trivial to run a recursive digest check on all files within the platform. However, I can see your point that comparing a single digest is visually much faster than comparing hundreds, but when it's done programmatically, it's a moot point IMHO

[Andrew Eddie]

Mark, it's as easy as this:

https://github.com/eddieajau/rahisi/blob/master/www/index.php#L26

The CMS could take the same approach, only using a phar with the legacy tree (and then we can remove all the `defined or die` lines from the platform code).

Regards,

Andrew Eddie

Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla developers

[Elin Waring]

Don,

I think people would be thrilled to see an implementation of dependency management when installing platform applications (including the CMS and CMS extensions).  However given that we don't even really have a published example right now that checks for the presence of an existing platform-as-a -whole tagged version before installing (i.e. different applications need different tags and some may even count on the CMS bug fixed LTS version)  not to mention testing for the presence of a specific package in a specific version, I think it is not realistic to say stop working on simpler solutions that will solve real problems in the short term.

Elin

[Donald Gilbert]

I didn't say stop working on the Phar solution. I just don't understand the problem that Phar is trying to solve.

From what I can discern (which may be wrong) the goal for packaging the platform as a phar is to be able to have a single instance of the platform codebase on a server which runs multiple platform applications. I don't fully understand the use case for this though. IMO, most users wouldn't find that very useful.

I can see that if a developer wanted to build multiple apps (web, command line, etc) on the platform for internal use within a company (eBay for example) it would be useful to have a single instance of the platform that powered all that code that would be easily updated. Or maybe for a service based company that provided a Joomla hosting / development environment.

But for a (what I call) standard web dev company (like what I work for), our clients all have their own hosting account already, or we set it up for them. It would be irrelevant if the platform was a phar or multiple files, since it gets installed on every account. As a phar, it would include packages that we may or may not use thus removing any perceived space saving benefit. To solve THAT problem, it would be much better for me to be able to specify exactly what dependencies I want to install in a json config file and run composer install (which handles all the dependency checking).

[Louis Landry]

There are several nice things about using Phar packages.  One (and one of the biggest things from a CMS perspective) would be the ability to update the entire platform with one file.  No more having to change hundreds, if not thousands, of files just to get that task done.  Extension developers could use this same process.  A component could package itself up as a Phar and be updated in the same matter.

In a high availability (clustered) environment Phar packages are also helpful.  Deploying an application by uploading a couple of Phar packages is much easier to validate than hundreds or thousands of class files.

You also have automatic file signing built in to Phar packages so you can avoid having unwanted changes to your code go unnoticed.  Additionally no one can hide a new file in your codebase.  Often times when sites are compromised there are shell scripts placed inside valid code folders as dot files or seemingly innocuously named files.  Not really an issue with Phar packages.

I'd encourage you to read through the readme at https://github.com/LouisLandry/packager.  There are several use cases for the packager application, not just packaging up the platform.  The original idea was to make it such that we can package the application with the platform packages that are needed.  The packager app itself is an example of this.  If you download it, joomla-packager.phar,

the application code only has the following packages added: application, registry, input, filter, event, log, object.  Additionally it will pull down and package those parts of the Joomla platform from the 12.1 release because that is what the manifest describes [https://github.com/LouisLandry/packager/blob/master/src/packager.xml].

Ultimately no one has to Phar anything, but I think it's a great option for us going forward.

- Louis

[Donald Gilbert]

Ok, I see there are some benefits to Phar than what I could come up with. Now that I think about it, composer itself is a phar command line app. And the package.xml files that that are required by joomla-packager.phar are very similar in structure to the composer.json files used by composer. The main difference would be that the packager downloads everything into a self-contained phar, while composer downloads everything into the vendor directory of your app.

The feature sets are a little different as well, and I'll need to dig into both to see them more clearly. But in the long run, it may be a worthy exercise to find the best of both worlds, and implement the best features of composer into the packager app. (Composer generated classmaps comes to mind, but that's only a benefit when we namespace the platform.)

[Gary Mort]

On Wednesday, November 21, 2012 1:23:13 PM UTC-5, Donald Gilbert wrote:

Ok, I see there are some benefits to Phar than what I could come up with. Now that I think about it, composer itself is a phar command line app. And the package.xml files that that are required by joomla-packager.phar are very similar in structure to the composer.json files used by composer. The main difference would be that the packager downloads everything into a self-contained phar, while composer downloads everything into the vendor directory of your app.

Not quite, composer CAN download everything into the vendor directory.  "Everything" might be a set of phar files from each vendor.

It will also do the reverse, and package everything up into a deployment package - but it's a bit weak there.

Packager on the other hand is a builder system.  And you can use it to build a composer distribution.  Ie define some files:

joomlaJoomla.xml, joomlaPHPMail.xml, joomlaSimplepie.xml

Each one of these files could be configured to package up a different section of libraries into a Phar file along with creating a composer.json file for each package - then submit them to packagist for downloading.

In short, if your goal is to reach a composer like distribution, packager is the way to get there as you can take get there in stages.  First setup a Joomla platform distribution, then a Joomla CMS, then start breaking up the pieces.

The limiting factor is most likely to be people willing to spend the time creating each subpackage....  I for one have no interest in it - for me this was a super convenient way to add namespaces to all the Joomla libraries without breaking an existing website.  Mainly because I discovered that while I can do:

use joomla\joomla\platform as jp;

I can't do:

use \ as jp;

[which was my initial goal and would have allowed me to check the JPlatform version and use either the installed libraries, or my own replacement set]

[Andrew Eddie]

On 22 November 2012 07:18, Gary Mort <jooml...@gary.mort.net> wrote:

Not quite, composer CAN download everything into the vendor directory.  "Everything" might be a set of phar files from each vendor.

I looked at composer and the problem with it is that it's up to the vendor to determine what downloads.  Maybe that's a phar, maybe it's the clean source, maybe it's the root of their github repo.  It also seems to be duplicating the effort that PEAR does to some degree.

Regards,

Andrew Eddie

[Donald Gilbert]

BUT it doesn't carry the stigma of what PEAR used to be.

PEAR is "old and busted" and composer is the "new hotness"

[Andrew Eddie]

On 22 November 2012 07:43, Donald Gilbert <dilber...@gmail.com> wrote:

BUT it doesn't carry the stigma of what PEAR used to be.

PEAR is "old and busted" and composer is the "new hotness"

Yeah, Apple's trying the same line trying to justify changing from the 30 pin to the 8 pin connector :P  

I don't think Composer is intended to be a deployment strategy; this is where Packager comes in.  To that end though, I think it's worth looking closely at what Packager and Composer do and get a clear picture of what their respective roles are (maybe one could even consider that Packager "extends" Composer).  An obvious point there is possibly looking at moving to JSON manifests rather than XML, and aligning them as closely as possible to what Composer does so you don't have to remember too many differences.

Regards,

Andrew Eddie

[Don]

I would be 100% for this approach. 

Sent from my iPhone

[Elin Waring]

The CMS already runs many separate applications on a single instance of the platform including three web applications and 4 CLI applications. Plus  longer term I think we all hope and expect that there will be many applications--e.g. CRMS, eCommerce applications, lots of things-- running alongside the CMS or next to each other totally apart  from the CMS.

I think you can phar or not whatever packages you want in a custom application. The original commit message even talked about packaging that way for extensions.

Elin

[Gary Mort]

On Wednesday, November 21, 2012 4:44:14 PM UTC-5, Donald Gilbert wrote:

BUT it doesn't carry the stigma of what PEAR used to be.

PEAR is "old and busted" and composer is the "new hotness"

The problem I have with PEAR is that the process of publishing a PEAR module is tedious and painful[hmmm, maybe like publishing a Joomla Extension?].

While anyone can setup their own PEAR server, there is no central repository for browsing them all[or rather, there are a couple and they require the vendor to register with them].

Packagist, the Composer equivalent of PEAR, still has the issue with the vendor needing to register each package.  But outside that, it does at least provide a single place for dependency management - and you can host your own Packagist server and provide the link to them for it.  And as I read the specs, while vendor/vendorname/packagename is the default directory setup - your able to override that when installing 

I've setup a repository for Joomla Packager recipes:

https://github.com/garyamort/joomla-packager-recipes

Right now it just has the one I'm interested in, namespacing.  However, I'll be adding 2 more:

bootstrap: a minimal Joomla framework install to create a Composer compatible install package of the bear minimum Joomla Platform install[JLoader, JImporter, JObject, JLog, and a few others].

Then I'll take another directory/set of objects and see about creating a second module....  If I can get 2 or 3 modules working then it's just a matter of someone deciding what a "package" should contain and creating it.