Reminder of the importance of testing 1.7

155 views
Skip to first unread message

elin

unread,
Jul 16, 2011, 3:12:32 PM7/16/11
to joomla-de...@googlegroups.com
Hi,

As you should know the release of 1.7 is scheduled for Tuesday and support for 1.6.x will end a month after that. That means that if you have not yet tested your extensions for 
a. compatibility with a simple upgrade of a user's site from 1.6.x to 1.7
b. new installation on 1.7
c. upgrade of an older extension release on 1.7 you should be doing it today. 
As you know, 1.7 has had an alpha, beta and RC release over the past 2 months in anticipation of this.

There are a small number of potential backward compatibility issues that you will want to check:

Most important I suspect will be the first, so I will highlight here:

JDatabaseQuery
JDatabaseQuery is now abstract due of the work done to support new database engines (Windows Azure and Microsoft SQL Server). This means you must use$db->getQuery(true); to instantiate a query as is the correct practice in Joomla 1.6.
But please make sure you have reviewed them all.

We have noticed that some extensions have hard coded version checks for == 1.6 so if you do that please make sure you have accounted for 1.7. 

Please report any issues to the CMS tracker.

Thanks for your help in making this what everyone hopes will be the smoothest Joomla release ever.

Elin

Nicholas K. Dionysopoulos

unread,
Jul 16, 2011, 3:58:00 PM7/16/11
to joomla-de...@googlegroups.com
Hi all,

Just a reminder to fellow developers who might be producing cross-version single installation packages for Joomla! 1.5/1.6/1.7. Here's a list of my most used tricks for cross-version compatibility (all of my software is currently Joomla! 1.5/1.6/1.7 compatible, using the same installation package).

In order to implement things differently under each Joomla! release, working around API changes, you can use the following code:

if(version_compare(JVERSION,'1.7.0','ge') {
// Joomla! 1.7 code here
} elseif(version_compare(JVERSION,'1.6.0','ge') {
// Joomla! 1.6 code here
} else {
// Joomla! 1.5 code here
}

or, if 1.7 specific code is not required:

if(version_compare(JVERSION,'1.6.0','ge') {
// Joomla! 1.6/1.7 code here
} else {
// Joomla! 1.5 code here
}

It seems like an overkill, but it has the following benefits:
  • Compatibility with all Joomla! versions using the same package, ergo using the same codebase, ergo less maintenance overhead
  • Once 1.8 is released and we drop support for earlier Joomla! releases, we can just grep our code for "if(version_compare(JVERSION," and remove all of the legacy code.
Moreover, I would like to remind you that you can mix <param> and <fieldset> tags in your XML files. Joomla! 1.5 will ignore <fieldset> tags, whereas Joomla! 1.6/1.7 will ignore <param> tags. One XML file is adequate for two versions.

You can always use JLanguage to load translation files from the com_myextension/language directory in Joomla! 1.5, simulating in large part the way Joomla! 1.6 works. You can also copy/paste the keys for menu item options from your sys.ini to your .ini translation file so that Joomla! 1.5 can find them. Since Joomla! 1.5 doesn't support the "_QQ_" trick, you can write HTML attributes in language strings like this:
COM_FOOBAR_SOME_KEY="<a href='http://www.example.com'>Just a test</a>"
that is, replace your double quotes with single quotes. It's still valid HTML, albeit it invalidates XHTML. Well, you can't always win, can you?

For people trying to implement custom JElements and custom JForm fields, let it be known that it possible to do so with a single PHP file, using conditional class inheritance.

If writing content plugins, do note that the name of the content preparation event has changed between 1.5 and 1.6/1.7. The parameters are the same, so you can just create am onContentPrepare method which calls the onPrepareContent and be done with it, with a slight performance penalty.

Danger, Will Robinson! If your manifest XML file states that the schema version is 1.5, there is a bug when performing upgrades under Joomla! 1.6 (haven't tested with 1.7). Your SQL will not run. Moreover, adding update SQL file definitions won't run either. The only workaround is to create a script.foobar.php file which uses an update() method to manually parse and execute your SQL files. Despite your package having version="1.5.0" in the root <install> tag, Joomla! 1.6/1.7 will execute the script file, as if it were a Joomla! 1.6 extension. The latest dev release of Admin Tools Core uses this trick to mitigate an issue I had with updates under Joomla! 1.6.

For all of my dirty cross version tricks, please take a look at how I have implemented things in Akeeba Release System, as I'm sure I have forgotten many of those tricks :) The only thing tentatively missing from it is support for cross database driver code, as I have yet to set up IIS and SQL Server on my secondary Windows machine.

Remember that "impossible" is merely something that we have not tried to implement yet. Given adequate motivation, everything is possible, including forward- and backward-compatible installation packages for our Joomla! extensions. Since reinventing the wheel hardly benefits anyone, please take a look at how I've solved the problems that you may be experiencing now. That's why the code is GPL, after all. Also, if you have a better alternative, please share it! Sharing is caring :)

Cheers,

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com
--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-general/-/VMWxcbOiWvwJ.
To post to this group, send an email to joomla-de...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-gene...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-general?hl=en-GB.

elin

unread,
Jul 16, 2011, 4:02:22 PM7/16/11
to joomla-de...@googlegroups.com
Nick,

As always, a fantastic post. Would you want to write that up for the wiki?

Elin

Nicholas K. Dionysopoulos

unread,
Jul 16, 2011, 4:05:01 PM7/16/11
to joomla-de...@googlegroups.com
Hi Elin,

Thank you! The information covers so many different topics, I am not sure which page on the Wiki it would fit in. Is there a page/section about writing cross-platform Joomla! extensions or should we probably start such a section/page? I ahve certainly no problem writing up this information in a fashion understandable even by near-newbie developers, with working examples :)

Cheers,

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com
--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-general/-/Gomj5KLw5ZgJ.

Prasit Gebsaap

unread,
Jul 16, 2011, 4:09:13 PM7/16/11
to joomla-de...@googlegroups.com
Hi,
 
I have a problem when I install my component (1.6 version) on Joomla! 1.7. It said that language files missing. I store it like this
 
 
language/admin/en-GB.com_jongman.ini
language/site/en-GB.com_jongman.ini
 
These package works for Joomla! 1.6.
 
What that I miss?
--
Prasit Gebsaap
 
Nonthaburi, Thailand
 

Mark Dexter

unread,
Jul 16, 2011, 4:19:06 PM7/16/11
to joomla-de...@googlegroups.com
@Nicholas: If you look here (http://docs.joomla.org/Developers) there is a section called Migration. Something like "Tips for Creating a Single Installation Package for 1.5/1.6/1.7" linked in that section might be a good place for it. Mark

Nicholas K. Dionysopoulos

unread,
Jul 17, 2011, 5:27:35 AM7/17/11
to joomla-de...@googlegroups.com
Thank you Mark!


Please review it and feel free to edit it :)

Best regards,

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com

Sam Moffatt

unread,
Jul 17, 2011, 10:39:05 AM7/17/11
to joomla-de...@googlegroups.com
On Sun, Jul 17, 2011 at 5:58 AM, Nicholas K. Dionysopoulos
<niko...@gmail.com> wrote:
> Danger, Will Robinson! If your manifest XML file states that the schema
> version is 1.5, there is a bug when performing upgrades under Joomla! 1.6
> (haven't tested with 1.7). Your SQL will not run. Moreover, adding update
> SQL file definitions won't run either. The only workaround is to create a
> script.foobar.php file which uses an update() method to manually parse and
> execute your SQL files. Despite your package having version="1.5.0" in the
> root <install> tag, Joomla! 1.6/1.7 will execute the script file, as if it
> were a Joomla! 1.6 extension. The latest dev release of Admin Tools Core
> uses this trick to mitigate an issue I had with updates under Joomla! 1.6.

Did you submit a bug report with a small reproducible test case? Most
unreported bugs tend to go unfixed. The version tag is ignored in 1.6
and was used in 1.5 to mark a 1.0 extension as requiring legacy
support. Where possible the changes to the installer I made were aimed
to be backwards compatible and also tolerant of 1.5 marked files so
I'm surprised you had issues.

A quick test of com_alpha from the test installer packages appears to
work fine with "version=1.5" in the <install> tag seems to work fine
against a 1.6.1 install I had sitting around. You need to keep in mind
that until an extension is installed it uses the "install" SQL file
not the ones specified in the updates and stores the number of the
highest update in the jos_schemas table so that when it does an
upgrade it runs the upgrade files between the last schema version it
saw until the latest version available. This is in part to provide
compatible operation with 1.5 style extensions (the install tag works
the same across both releases) while providing newer functionality
where possible (unfortunately a developer would have to implement the
upgrade functionality in 1.5 but 1.6 would handle it natively).

Check out the sample and have a play adding extra update files to see
what they do. They live here from the Joomla! SVN root:
/tests/_data/installer_packages

com_alpha is the primary testing though there are other packages that
exist to test other functionality as required.

Cheers,

Sam Moffatt
http://pasamio.id.au

Nicholas K. Dionysopoulos

unread,
Jul 17, 2011, 1:08:39 PM7/17/11
to joomla-de...@googlegroups.com

Hi Sam,

I did search for thus bug and I saw it was submitted a long time ago, albeit I am now on mobile and can't search for it.

My test case is simple. Install Akeeba Backup Core 3.2.7 on Joomla! 1.6.3 and install Akeeba Backup Core 3.3.b1 on top. The install.sql file, put in an update tag, doesn't run. Using the update tag doesn't work either. I had to apply the workaround to get it working. This has been the source of many bug reports in my forum.

Once I get back home I will try to create a proper test case, a bug report and paste the link here.

Nicholas K. Dionysopoulos, lead developer AkeebaBackup.com

> --
> You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.

Nick Savov

unread,
Jul 17, 2011, 5:43:44 PM7/17/11
to joomla-de...@googlegroups.com
@Nicholas Wow! Great work! I've been looking for information like this!
I'm very thankful that you put this together!

Cheers,
Nick

>> (mailto:niko...@gmail.com)> wrote:
>> > > Hi Elin,
>> > >
>> > > Thank you! The information covers so many different topics, I am not
>> sure which page on the Wiki it would fit in. Is there a page/section
>> about writing cross-platform Joomla! extensions or should we
>> probably start such a section/page? I ahve certainly no problem
>> writing up this information in a fashion understandable even by
>> near-newbie developers, with working examples :)
>> > >
>> > > Cheers,
>> > >
>> > > --
>> > > Nicholas K. Dionysopoulos

>> > > Lead Developer, AkeebaBackup.com (http://AkeebaBackup.com)
>> > > Web: http://www.AkeebaBackup.com (http://www.akeebabackup.com/)


>> > > Blog: http://www.dionysopoulos.me/blog
>> > >
>> > > On Saturday, 16 July 2011 at 23:02, elin wrote:
>> > >
>> > > > Nick,
>> > > >
>> > > > As always, a fantastic post. Would you want to write that up for
>> the wiki?
>> > > >
>> > > > Elin
>> > > >
>> > > > --
>> > > > You received this message because you are subscribed to the Google
>> Groups "Joomla! General Development" group.
>> > > > To view this discussion on the web, visit
>> https://groups.google.com/d/msg/joomla-dev-general/-/Gomj5KLw5ZgJ.
>> > > > To post to this group, send an email to
>> joomla-de...@googlegroups.com

>> (mailto:joomla-de...@googlegroups.com).


>> > > > To unsubscribe from this group, send email to
>> joomla-dev-gene...@googlegroups.com

>> (mailto:joomla-dev-gene...@googlegroups.com).


>> > > > For more options, visit this group at
>> http://groups.google.com/group/joomla-dev-general?hl=en-GB.
>> > >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups "Joomla! General Development" group.
>> > > To post to this group, send an email to
>> joomla-de...@googlegroups.com

>> (mailto:joomla-de...@googlegroups.com).


>> > > To unsubscribe from this group, send email to
>> joomla-dev-gene...@googlegroups.com

>> (mailto:joomla-dev-general%2Bunsu...@googlegroups.com).


>> > > For more options, visit this group at
>> http://groups.google.com/group/joomla-dev-general?hl=en-GB.
>> >
>> >
>> >
>> > --
>> > Prasit Gebsaap
>> >
>> > Nonthaburi, Thailand

>> > http://www.joomlant.com (http://www.joomlant.com/),
>> http://www.joomlant.org (http://www.joomlant.org/)
>> > http://ktdms.blogspot.com (http://ktdms.blogspot.com/),
>> http://lowcost-it.blogspot,com (http://lowcost-it.blogspot,com/)


>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Joomla! General Development" group.
>> > To post to this group, send an email to
>> joomla-de...@googlegroups.com

>> (mailto:joomla-de...@googlegroups.com).


>> > To unsubscribe from this group, send email to
>> joomla-dev-gene...@googlegroups.com

>> (mailto:joomla-dev-general%2Bunsu...@googlegroups.com).


>> > For more options, visit this group at
>> http://groups.google.com/group/joomla-dev-general?hl=en-GB.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Joomla! General Development" group.
>> To post to this group, send an email to
>> joomla-de...@googlegroups.com

>> (mailto:joomla-de...@googlegroups.com).


>> To unsubscribe from this group, send email to
>> joomla-dev-gene...@googlegroups.com

>> (mailto:joomla-dev-gene...@googlegroups.com).

Nicholas Dionysopoulos

unread,
Jul 18, 2011, 1:40:48 AM7/18/11
to joomla-de...@googlegroups.com
Hi Nick,

You're welcome :) I am very happy knowing that at least one person found it useful!

Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com

Web: https://www.akeebabackup.com
Blog: http://www.dionysopoulos.me/blog

Sent from my iPad. Please excuse my brevity.

Sam Moffatt

unread,
Jul 18, 2011, 3:10:29 AM7/18/11
to joomla-de...@googlegroups.com
Do you want to ping me on skype as well (pasamio) or GTalk and we'll see if we can go through it to work out where the bug is.

Cheers,

Nicholas K. Dionysopoulos

unread,
Jul 18, 2011, 5:10:07 AM7/18/11
to joomla-de...@googlegroups.com
Hi Sam,

I finally figured out what is going on here. Let me present an example so that you understand along with me what happened:

Version 1.0 of the component doesn't have an update tag in the manifest. This is logical for a developer to assume it's the correct thing, as version 1.0 doesn't have any updates applied yet (it's 1.0!).

Version 1.1 of the components adds the update tag and adds two SQL files in the updates directory. The first one (called 010000.sql) is empty, the second one (called 010100.sql - you see how I am following version conventions) contains the schema changes between 1.0 and 1.1. However, because of line 898 in libraries/joomla/installer/installer.php, the updates ARE NOT APPLIED, only the schema version number is placed in the database. This behaviour mislead me into believing that the updates do not work.

IMHO, the behaviour of the library at this point is completely broken for all practical intents and purposes. Let me explain why.

The vast majority of extensions predate Joomla! 1.6. As a result, none of them had an update tag, since that did not exist. Let's say that version 2.0 of my component was the first version to support Joomla! 1.6. Version 2.1 introduces a schema change. How am I supposed to apply it? I can't use the update tag because Joomla! will ignore it when users upgrade from 2.0 to any later release. I can't rely on the installation SQL file as Joomla! 1.6 does not run it. So I have to go the long way around.

There are two possible workarounds I can see:

1. If the manifest version is 1.5.0 run the installation SQL file even on extension updates

2. (recommended) If there is no schema version registered in the #__schemas table for the extension being updated, DO RUN the schema updates.

It is only logical for a developer to assume than the first version of his extension should not have an update tag (it's not like it's documented anywhere...). With the current behaviour, we are dooming an extension developer to not be able to update his schema unless he had the foresight to include an update tag since version 1.0 or if he gives egregious instructions to users like "install version 1.1 which does nothing, but is required by Joomla! 1.6 and later, then install version 1.2 or later to successfully update the component". As you know, users NEVER read nor follow instructions; they just assume that the developer is a moron and write flaming reviews. So, please, for the love of Joomla!, change this behaviour of the installer library so that it behaves in an intuitive manner.

Best regards,

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com

elin

unread,
Jul 18, 2011, 6:52:59 AM7/18/11
to joomla-de...@googlegroups.com
I do want to mention that we've had all of the best practices examples of this in the test _data/installer_packages folder for a little over a year but I am not that sure how many people know they are there. It's something that I've had on my list to put in the wiki. I've found those examples immensely helpful.

Elin

Nicholas K. Dionysopoulos

unread,
Jul 18, 2011, 7:09:19 AM7/18/11
to joomla-de...@googlegroups.com
Hi Elin,

I did see the examples and this is exactly where I found examples for the completely undocumented features of Joomla! 1.6's extensions installer, like the update repository definition, update repository XML format and, of course, the update tag. 

Now, let it be said that the tag is called <update> and it is evident from its name that you need to use it for component upgrades. There is nothing in these examples which would even remotely hint us developers that the update tag is required even for components which have never been installed on the site before. Based on its name, its a far cry to assume that we could even think that it is required in any other case except component updates, otherwise the updates would not work, forever.

See, we are developers, not fortune tellers. Documenting the pitfall in the API is the responsibility of the developer who introduced it in the API, not the developer who will bump on it. Moreover, the API should have a minimum level of intelligence. Punishing the developer for not including an update tag by making it impossible for him to update the schema forever is not exactly a sign of forward thinking or even a well thought out API. Just to give you perspective, the current behaviour of the API is like not allowing a user update the content of an article unless he has included the word "update" in the article's alias. Does this sound normal to you? To me it sounds nothing but absurd. Not to mention that it completely breaks backwards compatibility with Joomla! 1.5 which was supposed to be considered a bug, according to Andrew's words.

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com

On Monday, 18 July 2011 at 13:52, elin wrote:

I do want to mention that we've had all of the best practices examples of this in the test _data/installer_packages folder for a little over a year but I am not that sure how many people know they are there. It's something that I've had on my list to put in the wiki. I've found those examples immensely helpful.

Elin

--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-general/-/hC_6p8SXR5oJ.

Sam Moffatt

unread,
Jul 18, 2011, 7:49:42 AM7/18/11
to joomla-de...@googlegroups.com
On Mon, Jul 18, 2011 at 7:10 PM, Nicholas K. Dionysopoulos
<niko...@gmail.com> wrote:
> Hi Sam,

> I finally figured out what is going on here. Let me present an example so
> that you understand along with me what happened:
> Version 1.0 of the component doesn't have an update tag in the manifest.
> This is logical for a developer to assume it's the correct thing, as version
> 1.0 doesn't have any updates applied yet (it's 1.0!).
> Version 1.1 of the components adds the update tag and adds two SQL files in
> the updates directory. The first one (called 010000.sql) is empty, the
> second one (called 010100.sql - you see how I am following version
> conventions) contains the schema changes between 1.0 and 1.1. However,
> because of line 898 in libraries/joomla/installer/installer.php, the updates
> ARE NOT APPLIED, only the schema version number is placed in the database.
> This behaviour mislead me into believing that the updates do not work.
> IMHO, the behaviour of the library at this point is completely broken for
> all practical intents and purposes. Let me explain why.

This is a design feature. You can use actual version numbers in there,
Christophe put that code in to use version_compare so PHP formatted
version numbers work fine. I'm personally of the view that you should
go for dates in YYYYMMDD format because they are guaranteed increase
monotonically and if you start developing with a larger group of
people they can push changes into a file and commit it to SVN as it
happens.

It also means that if you do end up having to make more than one
change per day you can be a little more specific (add -1 or specify an
hour from the outset) and the system should pick it up fine. Again I'd
also like to suggest instead of tying things to version numbers that
you do smaller incremental changes. At some point I'd like to make a
developer tool that checks for updates and runs them as you work with
an SVN working copy.

I'll elaborate further why I think that the way the system works isn't
broken after your other comments.

> The vast majority of extensions predate Joomla! 1.6. As a result, none of
> them had an update tag, since that did not exist. Let's say that version 2.0
> of my component was the first version to support Joomla! 1.6. Version 2.1
> introduces a schema change. How am I supposed to apply it? I can't use the
> update tag because Joomla! will ignore it when users upgrade from 2.0 to any
> later release. I can't rely on the installation SQL file as Joomla! 1.6 does
> not run it. So I have to go the long way around.

I agree that the vast majority of extensions predate Joomla! 1.6, some
even predate Joomla! 1.5! However given this is the case, I agree it
is wrong to assume that the first release of an extension for Joomla!
1.6 would support all of the features of Joomla! 1.6 - indeed if
you're aiming for 1.5 support then initially it probably wouldn't.

> There are two possible workarounds I can see:
> 1. If the manifest version is 1.5.0 run the installation SQL file even on
> extension updates

I think this is something different and a bug in backwards
compatibility. Though I'd query adding support back in for a legacy
platform as 1.7 is being released. Given the next major release would
be another LTS release, I'd be interested in having a discussion about
adding an extra legacy feature from the previous LTS release.

> 2. (recommended) If there is no schema version registered in the #__schemas
> table for the extension being updated, DO RUN the schema updates.
> It is only logical for a developer to assume than the first version of his
> extension should not have an update tag (it's not like it's documented
> anywhere...). With the current behaviour, we are dooming an extension

Well it's been in the installer samples for an excessively long period
of time. The particular lines for update have been there since around
May 2010 and have for the first time been a coherent set of packages
which demonstrate the functionality of the installer system both for
the use of testing and as documented practical examples. The sample
packages existed around August 2008 or earlier so they're not
something new - just perhaps not well publicised.

Furthermore if you want some documentation about this check out the
"Developing a MVC Component for Joomla! 1.6" which since at least
November 2010 has covered the need to create a blank file from step
one for the update system. Again this is the great work of Christophe
Demko:
http://docs.joomla.org/Developing_a_Model-View-Controller_(MVC)_Component_for_Joomla!1.6_-_Part_01

> developer to not be able to update his schema unless he had the foresight to
> include an update tag since version 1.0 or if he gives egregious
> instructions to users like "install version 1.1 which does nothing, but is
> required by Joomla! 1.6 and later, then install version 1.2 or later to
> successfully update the component". As you know, users NEVER read nor follow
> instructions; they just assume that the developer is a moron and write
> flaming reviews. So, please, for the love of Joomla!, change this behaviour
> of the installer library so that it behaves in an intuitive manner.
> Best regards,

You make it sound like the extension developer, who has until now had
no supported way of updating their schema, is unable to do the job
themselves. Presently anyone wishing to update their schema has to do
it by hand and the system is predicated upon the simple fact that this
is already the case.

So lets presume that developer had a 1.5 extension which they've
written. If they're going to do schema changes then they're already at
the point that they have to handle this because 1.5 doesn't have that
functionality. They've released versions 1.5.0 through to 1.5.5 and
each had a schema change in them that they've handled. The 1.5 series
was also made compatible with 1.6 as well but they didn't want to take
advantage of any 1.6 features yet.

So at this point they've got code to do schema updates and want to
transition to 1.6 support only for the extension. 1.6 will once it
knows what version of the schema the extension is on apply any updates
however until that happens won't make any assumptions. I'm a great
believer of automated systems not making assumptions on my behalf when
we're dealing with data because of the irreparable damage it could
cause - though for your extension you can at least hope they had a
recent backup.

So in version 1.6.0 they don't have the foresight to include an
updates tag. In 1.6.1 they start shipping an update. So at this point
the system is going to record the highest schema number against the
extension and leave it at that. What you're proposing is that we go
through the each of the schema updates. This begs the question though
for this situation what schema version do you start applying from?

Presuming the user is installing from 1.6.0 to 1.6.1 then this is
easy, the 1.6.1 version should be applied. But it gets a bit more
complicated when you start taking into account that the user could be
on 1.5.0 through to 1.5.4 each with their own schema versions. At what
point do you start applying updates? Do you apply all of the changes
from 1.5.0 to 1.6.1? Or do you just apply 1.6.0 to 1.6.1?

However how will that work? The installer script runs after the
database processing has completed. That means that the 1.6.0 to 1.6.1
update would be applied on a 1.5.0 update. What happens if the 1.6.1
relies upon seeing the 1.6.0 database structure not the 1.5.x
structure?

So the system implements a rather simple rule: when in doubt leave it
to the person with the most amount of intelligence about what their
component state is and let them take care of it. In reality this turns
out to be a rather safe option as instead of the developer
unconsciously running into a pitfall, you've consciously handled the
situation without actually realising it.

Once the system knows what version of the schema you are running it
happily takes over. Until such time it doesn't take a stab in the dark
and randomly apply SQL statements against an unknown schema version.
For all it knows there needs to be changes to the underlying schema
before they are even relevant.

If anything the system is running as intended even through the
misunderstanding you've had of how it works. You're handling the
schema updates up until the point the system knows what version its on
and can safely take over control.

Hopefully you understand the much more cautious approach I've taken.
Instead of potentially and automatically trashing a schema based on an
assumption of a particular schema version that the system can't safely
make, it defers to when the developer has acknowledged they support it
and works from there. If anything you're a perfect example of this
actually working well.

elin

unread,
Jul 18, 2011, 7:57:41 AM7/18/11
to joomla-de...@googlegroups.com
Well it's been a "did you know" on the front page of community.Joomla.org for a a while  anyway http://community.joomla.org/featured-articles/did-you-know/456-extension-updates-made-easier.html. As far as I know it has been the recommended practice for ages. 

Elin

elin

unread,
Jul 18, 2011, 8:03:43 AM7/18/11
to joomla-de...@googlegroups.com
But even though that's about 1.5 I will totally agree that the more we document in more different ways the better, which is way it's great that you posted in the wiki. Because in the end like everything else docs are only as good as the docs we all write.  So if we all get into the routine of writing when we learn something or see something, we'll all collectively have better docs.

Elin

Nicholas K. Dionysopoulos

unread,
Jul 18, 2011, 8:45:24 AM7/18/11
to joomla-de...@googlegroups.com
Sam,

What you described is fine in an ideal world where users install all new releases of any given extension sequentially. However, the real world is a far cry from being ideal.

Akeeba Backup started supporting Joomla! 1.6.0 with version 3.1.3. In the meantime the following releases were published: 3.1.4, 3.1.5, 3.2.a1, 3.2.a2, 3.2.b1, 3.2.b2, 3.2, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.2.6, 3.2.7, 3.3.a1, 3.3.a2, 3.3.a3, 3.3.b1 and in some days 3.3. I need to support people upgrading from any of these releases to 3.3, 3.3.1 and so on. So, for the foreseeable future (since I didn't know I had to include a blank update file) I am stuck with the workaround. Given that I have seen numerous people still using Akeeba Backup 3.0.a2 (released 18 months ago), I expect that for the whole lifetime of Joomla! 1.8 I will have to provide the workaround. As a result, the SQL update feature of Joomla! 1.6.0 and later has become useless for me, since there is no way to guarantee that users will have installed Akeeba Backup 3.3 Stable which would be the first release to provide a blank update file. You see how I am stuck here? The workaround I had found was to check for the existence of tables when the user accesses the component's Control Panel (SQL overhead) and manually run them if they hadn't run, essentially duplicating my SQL code (maintenance nightmare). In order to avoid that, I have to create a script file which replicates core code and essentially simulates how things were done in Joomla! 1.5. Furthermore, I have to detect old schema versions in the installation script and apply changes manually - more so since many hosts don't even allow ALTER TABLE commands and I have to go through the long route of temporary tables.

So, based on the no-assumptions rule, I have to toss away a useful feature of Joomla! and do it by hand, replicating core code in my installation script file. Since the decision making is left to the most intelligent part of the installation process (that would be me, I assume :p) please at least give me a way to tell JInstaller something in the lines of "if you don't see any version number, run this SQL file, otherwise proceed with the updates tag".  Can we at least include that feature, or should I just forget all about Joomla! running SQL commands and do everything manually?

Sam Moffatt

unread,
Jul 18, 2011, 10:09:58 AM7/18/11
to joomla-de...@googlegroups.com
On Mon, Jul 18, 2011 at 10:45 PM, Nicholas K. Dionysopoulos
<niko...@gmail.com> wrote:
> Sam,
> What you described is fine in an ideal world where users install all new
> releases of any given extension sequentially. However, the real world is a
> far cry from being ideal.

Actually the example I was trying to demonstrate was precisely a
disjoint upgrade scenario where if the user has installed 1.5.0, what
schema version do you put in the install package for 1.6.1? Or do you
have a 1.5.0 to 1.6.1 package that users must install? You can't put
in the script to go from 1.5.0 to 1.5.1, 1.5.1 to 1.5.2, 1.5.2 to
1.5.3, 1.5.3 to 1.5.4, 1.5.4 to 1.6.0 and 1.6.0 to 1.6.1 in the 1.6.1
package and then run them all on update because if you do so then on a
base 1.5.4 install the 1.5.0 through to 1.5.3 scripts will be rerun
even though the database is already at the 1.5.4 schema release
already and don't need to be run.

I feel you haven't adequately thought out what would happen with such
a scheme on disparate versions being installed and perhaps realised
that your situation only works well when people install the first
version in the chain and no others before upgrading. Could you please
slowly re-read my earlier post and this post because I think you will
find that I very much am concerned about disjoint upgrades which is
precisely the situation that breaks blindly running SQL against a
database without the system knowing the version of the original
schema.

> Akeeba Backup started supporting Joomla! 1.6.0 with version 3.1.3. In the
> meantime the following releases were published: 3.1.4, 3.1.5, 3.2.a1,
> 3.2.a2, 3.2.b1, 3.2.b2, 3.2,
> 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.2.6, 3.2.7,
> 3.3.a1, 3.3.a2, 3.3.a3, 3.3.b1 and in some days 3.3. I need to support
> people upgrading from any of these releases to 3.3, 3.3.1 and so on. So, for

What you proposed is that the system blindly run the SQL update files.
So lets just for arguments sake say that your minor releases each have
minor schema update changes. Let's presume the user installed 3.1.3
before you tripped onto the update functionality with lets say 3.2 but
the user installs 3.3 without touching 3.2 where you started
supporting it. The installer doesn't know that you're running on a
3.1.3 schema before it runs the updates, it only knows what the last
schema version was for the last package it installed that had one.
Presuming you start using schema diffs in the supported manner from
3.2, where do you start your schema updates? At 3.1.3? But what about
someone who installed 3.1.4 before 3.2? Then the 3.1.4 update
potentially will be applied twice (might be a new table and select to
populate it from another table, I've done migrations like that before)
then could cause other problems based on the data structures in use.

I refuse to let a system blindly run random SQL statements when it
doesn't have an indication what the underlying schema version is
before it starts. That is a recipe for disaster and the start of an
inconsistent schema.

This is why the system will not activate until it knows what schema
version it is utilising. Once you get to that schema version at the
end of an extension install, it will happily accept that it is at that
version after install and take things from there assuming the
developer has through the install updated it to that release however
it will not make a guess that the schema version is in place until
that point.

As I asked in the previous email, at what point do we start keeping
schema versions and how do you handle with your mechanism a user
upgrading from 1.5.0 to 1.6.1 and 1.6.0 to 1.6.1? If the user isn't on
1.5.0 then running all of the schema changes from 1.5.0 to 1.6.1 is
incorrect because if perhaps they're on 1.6.0 then they don't need it
and if you only go from 1.6.0 to 1.6.1 then you still need to handle
getting the user to 1.6.0 to begin with.

> the foreseeable future (since I didn't know I had to include a blank update
> file) I am stuck with the workaround. Given that I have seen numerous people
> still using Akeeba Backup 3.0.a2 (released 18 months ago), I expect that for
> the whole lifetime of Joomla! 1.8 I will have to provide the workaround. As
> a result, the SQL update feature of Joomla! 1.6.0 and later has become
> useless for me, since there is no way to guarantee that users will have
> installed Akeeba Backup 3.3 Stable which would be the first release to
> provide a blank update file. You see how I am stuck here? The workaround I

I see how you are stuck. However you need to consider how you could
work out what schema version these folks are on and put it into the
system. Once the system knows what schema version they are on it will
act. But it won't act until that point. I can see how this is an issue
but even if you released 3.3 with the blank update file as your first
release, how would this help you getting from either 3.1 or 3.2?
Presuming even a change in 3.2, having an upgrade script from 3.1 to
3.2 to 3.3 in 3.3 at the first release would result in any 3.2
installs having the same change applied twice which may or may not
have undesirable outcomes. Presuming you got your 3.3 installs off on
the wrong foot and 3.4 contained a schema change again then what
happens for a 3.2 to 3.4 install? Or a 3.3 to 3.4 install if you have
the schema updates from 3.1 to 3.2, 3.2 to 3.3 and 3.3 to 3.4 there
and apply them all straight off the bat then you're going to be
reapplying the 3.1 to 3.2 and 3.2 to 3.3 updates.

> had found was to check for the existence of tables when the user accesses
> the component's Control Panel (SQL overhead) and manually run them if they
> hadn't run, essentially duplicating my SQL code (maintenance nightmare). In
> order to avoid that, I have to create a script file which replicates core
> code and essentially simulates how things were done in Joomla! 1.5.

You could also use JInstaller->parseSchemaUpdates with the XML
manifest and extension ID and it'll work that out for you once you
populate the #__schemas table with the version, but the trick is
working out the version to start from.

> Furthermore, I have to detect old schema versions in the installation script
> and apply changes manually - more so since many hosts don't even allow ALTER
> TABLE commands and I have to go through the long route of temporary tables.

You point out the problem that I have of automatically applying
updates: you need to detect the old schema version first! You have to
work out where you are before you can start applying changes right?
But Joomla! doesn't know where you are until you tell it! This is
precisely why I feel blindly applying SQL doesn't work as a solution
and you most definitely have to work out which schema version you're
on before you hand over control to Joomla!.

I hope you realise the problem you lament here as the author of the
schema for the extension is rather difficult for the generic
JInstallerComponent adapter to handle. You need to work out what
version you were on and tell Joomla! before it will take over control
otherwise all Joomla! is doing is taking a stab in the dark. If you as
the extension developer have a problem detecting what version of the
schema you are on already then how is Joomla! going to handle it?

> So, based on the no-assumptions rule, I have to toss away a useful feature
> of Joomla! and do it by hand, replicating core code in my installation
> script file. Since the decision making is left to the most intelligent part
> of the installation process (that would be me, I assume :p) please at least
> give me a way to tell JInstaller something in the lines of "if you don't see
> any version number, run this SQL file, otherwise proceed with the updates
> tag".  Can we at least include that feature, or should I just forget all
> about Joomla! running SQL commands and do everything manually?

So the fundamental problem is that you need to tell Joomla! what
schema version you are on before it will do anything. This is again,
as I keep stressing, very much part of the design of the system to
prevent it applying updates that are incorrect. As you lament it is a
pain to work out what version of the schema you are on if you don't
have a record of it previously and even more dangerous to shuffle data
around when you don't know at least what the schema should look like
(of course someone could change things from under you but at that
point they've voided any support IMHO).

In your preflight script put in the schema version into the database
table and then when it comes to run the schema updates it will pull
that number and safely apply them. Your preflight should upgrade the
underlying database structures to match what that schema version
should look like before it works or if you can apply SQL updates from
that point, push in the right schema version and let it go. Preflight
also runs prior to any copy or database operations (as opposed to the
old com_install or the replacement install hook) so you could use this
to probe the old XML manifest file for the version (normally the
manifest is copied last but if you have your manifest file listed in
the administrator files section it will be replaced when the rest of
the admin files are copied).

Alternatively you can utilise the install hook to run the database
upgrade until a certain point, populate the schemas table with the
correct base schema and then use parseSchemaUpdates to process the
files as well. Again you need to work out what version to start with
and push that into #__schemas before Joomla! will do anything. If you
do not have your XML manifest file listed in the manifest file itself
then you can also access the original manifest file here to work out
what version you're on.

Even if you want to keep using the ControlPanel mechanism you can
instantiate an instance of JInstaller (probably safer to grab
JInstallerComponent in case a later version of Joomla! marks
JInstaller abstract or something), set some values (the extension_root
path should be JPATH_COMPONENT_ADMINISTRATOR if you're using the
standard test component layout), load up your XML manifest file and
then use parseSchemaUpdates to run through them all.

Fundamentally at some point you need to let Joomla! know what version
of the schema it's trying to deal with. From there it will work out
the rest but it has no mechanism of working out what current version
of the schema you have installed beyond what it sees in #__schemas. As
the extension developer you're the one who knows what schema version
you're on and hopefully how to get from where it is now to where it
should be. If you've got SQL scripts for that then set the version in
#__schemas and let it sequentially apply them from that point. If
you've used a mix of PHP to do changes prior to that point, which some
people prefer, then get it to that point before you hand over to the
SQL update mechanism. Either way get it to a particular version, tell
Joomla! the version and it will sequentially apply the rest from
there.

Nicholas K. Dionysopoulos

unread,
Jul 18, 2011, 11:39:09 AM7/18/11
to joomla-de...@googlegroups.com
Hi Sam,

Ah! I see the logic behind this architecture now. And I now understand what you said, that the proper workaround is to use the preflight() method to populate the correct schema version. OK, now that makes perfect sense. I can use my smart PHP code to figure out which schema version is installed and save that information to the #__schemas table, so that I can let Joomla! 1.6/.7/.8/.whatever work out the upgrades. That seems to be the most hassle-free way to do that. I have to admit, I wasn't aware that preflight ran before the SQL files were run. Maybe this tip should be added in the wiki? It would save developers lots of trouble. If nobody else volunteers, I can always update it :)

Cheers,

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com
--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.

Nick Savov

unread,
Jul 18, 2011, 12:05:12 PM7/18/11
to joomla-de...@googlegroups.com
Sam and Nicholas, kudos to you both for being geniuses and for a great
discussion! Most of it was over my head, but I learned a few things here
and there.

It's nice knowing that people like you, Elin, and others are involved in
the Joomla project :)

Cheers,
Nick

>> (mailto:joomla-de...@googlegroups.com).


>> To unsubscribe from this group, send email to
>> joomla-dev-gene...@googlegroups.com

>> (mailto:joomla-dev-gene...@googlegroups.com).

Sam Moffatt

unread,
Jul 18, 2011, 8:12:49 PM7/18/11
to joomla-de...@googlegroups.com
I started on some documentation here but never got it finished:

If you're happy to update the wiki then that'd be great.

Effectively preflight runs as soon as possible after we've got the manifest file, worked out if we need to switch into an update mode from the install (e.g. if the extension is installed then the system transparently swaps into the update mode out of install so that the installation hooks work properly) and has worked out where the script file is and loaded it. At this point you could conceivably prevent the installation by calling the abort mechanism from the parent or prepare anything ahead of the file copy and database changes occurring. It is one of the new hooks in 1.6 to let you get your own code in as close to the installer process kicking off as I felt safe. You're also handed a copy of the installer object (from memory) and a param to say if the preflight is for install or update so that if there is a difference you can handle that. 

The install/update hooks run effectively where com_install ran which was after the files have been copied (except the manifest) and the database has been updated (except for the #__extensions or equivalent entry). 

Postflight, at least for components, runs after the #__extensions entry has been made/updated (install/update) and that the manifest file has been copied. At this point there should be no reason for the extension to fail. For those extensions utilising a redirect that broke with the shift from 1.5 to 1.6 where 1.5 com_installer doesn't do a redirect after install and now 1.6 com_installer does, you can instruct the installer in postflight to redirect to a custom extension installation screen from a component (again, see com_alpha's scriptfile for an example of this).

Both plugins and modules should also have these hooks as well, though their execution varies slightly.

Cheers,

Nicholas K. Dionysopoulos

unread,
Jul 19, 2011, 5:52:01 AM7/19/11
to joomla-de...@googlegroups.com
Hi Sam,

I updated the wiki page (http://docs.joomla.org/Extension_Installer/Installer_Hooks) with your insight.

I am going to perform a few experiments with the update tag and the preflight hook and share that back in the wiki.

Thank you for explaining how JInstaller works! It's greatly appreciated :)

-- 
Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com

Sam Moffatt

unread,
Jul 19, 2011, 10:48:15 AM7/19/11
to joomla-de...@googlegroups.com
Awesome, look forwards to seeing what you come up with. Complicated
3PD upgrades aren't really something I've much experience in so it
should make a great contribution.

Cheers,

Sam Moffatt
http://pasamio.id.au

On Tue, Jul 19, 2011 at 7:52 PM, Nicholas K. Dionysopoulos

Mike Smith

unread,
Jul 19, 2011, 11:05:15 AM7/19/11
to joomla-de...@googlegroups.com
Just installed the brand new 1.7 Stable - complete release onto a localhost and am being swamped with php errors:
Strict Standards: Accessing static property JCache::$_handler as non static 

All other test releases have worked correctly and without these error messages......


Mike

Mike Smith

unread,
Jul 19, 2011, 11:08:53 AM7/19/11
to joomla-de...@googlegroups.com
That was in the admincp, in the front end they are there as well, as is:
Strict Standards: Static function JDatabase::test() should not be abstract

Mike.

Omar Ramos

unread,
Jul 19, 2011, 11:22:00 AM7/19/11
to joomla-de...@googlegroups.com
It looks like this error is being caused by the following code in the JCache file:
        /**
* Get the cache storage handler
*
* @return  JCacheStorage   A JCacheStorage object
* @since   11.1
*/
public function &_getStorage()
{
if (!isset($this->_handler)) {
$this->_handler = JCacheStorage::getInstance($this->_options['storage'], $this->_options);
}
return $this->_handler;
}

Since _handler is now static and above it is being accessed like it is still a public property.

Omar Ramos

unread,
Jul 19, 2011, 11:36:12 AM7/19/11
to joomla-de...@googlegroups.com
As for the second one, my IDE is throwing up a message in the JDatabase file about "static function should not be abstract" too and this Stack Overflow thread appears to provide some more detail:

So it looks like we should not have the test() function declared as abstract (I'm wondering if we can simply remove the abstract keyword from JDatabase, but everything else can go ahead and stay the same)?

It doesn't look like the test() method gets used at all in the code base (JDatabaseMySQL and MySQLi don't even use it in their own constructors) so it seems like the error would only show up during E_STRICT compile time as you discovered.

Mike Smith

unread,
Jul 19, 2011, 11:42:27 AM7/19/11
to joomla-de...@googlegroups.com
Agree Omar.

Also I know this can be "fixed" by turning off the e_strict reporting, but this is NOT the solution as PHP will continue to move forward so the core code needs correcting so as to support these standards. If not the web will inevitably get swamped with people all experiencing these "problems".

Mike.

Omar Ramos

unread,
Jul 19, 2011, 11:52:09 AM7/19/11
to joomla-de...@googlegroups.com
For the cache side correcting it to:

/**
* Get the cache storage handler
*
* @return  JCacheStorage   A JCacheStorage object
* @since   11.1
*/
public function &_getStorage()
{
if (!isset(self::$_handler)) {
self::$_handler = JCacheStorage::getInstance($this->_options['storage'], $this->_options);
}
return self::$_handler;
}

Removes the strict warnings on my end.

For the JDatabase side we have two options...we either remove the following line completely:

abstract public static function test();

Since it is not valid, or convert it to a simple static function definition (albeit an empty, non abstract one):

public static function test() {}

And that should remove that other E_STRICT issue.
Reply all
Reply to author
Forward
0 new messages