Installation improvements for 1.7
Jeremy Wilken
I recently took charge of allowing extensions to contain multiple
manifest files for specific versions. This would allow a lot of power
to developers who don't want to distribute extensions in version
specific packages (such as myself).
Here is my basic plan. Most of the work will be done in the JInstaller
class as far as I can tell from a quick glance at the code.
- Look for xml files (currently it will recurse into subdirectories as
well I think)
- Ignore any that aren't installer types
- If multiple installation xml files are found, select the one that
meets or is closest to the version of Joomla being used without going
over. So if the manifest has a file for 1.5 and 1.6, and we install on
Joomla 1.7, the 1.6 should be attempted
- Discard any extra installation files
- Copy the xml file to the proper extension place, depending on what
extension type, and rename to conventions (component.xml,
mod_module.xml, plugin.xml)
Let's say you have a component. You have component.xml, which is a 1.5
installation manifest file. Then you can add component_1.6.xml, and
the installer will pick it up instead, not based on the filename but
on the contents of the xml using version="1.6" in the root tag. The
installer will copy the 1.6 version to the component folder and rename
it to component.xml and ignore the 1.5 version.
Feedback is welcomed.
Oladele
is that apart from the core elements of any application, the rest
should be left to the clients to decide. What I am talking about is
simple - Give the end user the ability to choose what they wanted to
install. Most components, extensions are not actually being used
through out the life cycle of an application. This way applications
will be leaner and faster instead of clogging them.
Dele
> --
> You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
> To post to this group, send an email to joomla-...@googlegroups.com.
> To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.
>
>
--
The information contained in this message may be CONFIDENTIAL and is
intended for the addressee only. Any unauthorized use, dissemination
of the information or copying of this message is prohibited. If you
are not the
addressee, please notify the sender immediately by return e-mail and
delete this message.
Thank you.
Sam Moffatt
1) Do you support later versions of the manifest? (e.g. a 1.7 manifest in 1.6)
2) How do you handle the situation where someone puts another manifest
for a different extension in their unpacked zip a few levels deep?
Sounds good otherwise.
Sam Moffatt
http://pasamio.id.au
Sam Moffatt
Can you open a new thread for this instead of hijacking an existing one?
Thanks!
Sam Moffatt
http://pasamio.id.au
Jeremy Wilken
My opinion is that a 1.7 manifest should fail on 1.6. Install files
can be adopted upwards, but not backwards. Without changing the xml
manifest organization, I think it makes most sense to give the
manifest version the minimum version of Joomla, and if a future
version requires a different install file, then including it with a
higher minimum version would be picked up by the installer.
I plan to recurse and parse all xml files, just to be safe. That way
if the install file is deeper than the root, they will be picked up.
So parse the whole unzipped directory, collect any xml files that have
a correct manifest declaration (install or extension for the root
node), and then make a judicious call on which to select based on
active version, and manifest versions available.
Dele, I'm not sure I understand why you are saying has to do with this
thread. If you can better explain I'm listening, but if its another
topic as Sam says, please start a new thread, thanks!
Jeremy
On 5 Mar, 21:33, Sam Moffatt <pasa...@gmail.com> wrote:
> Hi Oladele,
>
> Can you open a new thread for this instead of hijacking an existing one?
>
> Thanks!
>
Sam Moffatt
modules embedded in it and maybe a plugin. Worse still what if one of
the plugins is only needed for a particular version and is marked
accordingly - it would be the greatest version and picked over the top
of the root component. How would you handle this eventuality?
Sam Moffatt
http://pasamio.id.au
Jeremy Wilken
determinant of what additional extensions are installed? Say you have
a shim plugin for 1.6, but 1.7 it isn't needed. Then you would have
two manifest files that point to an install.php file (or just one
install and manifest file, where the install.php file runs a version
check during execution), which does the additional installations.
If I'm understanding correctly, that means the Joomla installer will
install what it is asked to do, and if something shouldn't be
installed but is passed to the installer, it seems tricky to set rules
for when that should fail. I'd rather leave it in the hands of the
developer to process the additional extensions during a component
installation.
So the version attribute in the root installation xml file node is the
minimum version. Support for a maximum version attribute would be the
logical way to solve this type of situation if we are open to adding
to the manifest. I was thinking of ways to avoid adding to the
manifest, but that doesn't mean it has to be so. I just think its
unnecessary, because if the manifest no longer will install in a newer
version, you will need a new one, which will be selected by the
installer regardless of a maximum version attribute.
Jeremy
On 5 Mar, 22:04, Sam Moffatt <pasa...@gmail.com> wrote:
> The thought is what happens if someone has a component with some
> modules embedded in it and maybe a plugin. Worse still what if one of
> the plugins is only needed for a particular version and is marked
> accordingly - it would be the greatest version and picked over the top
> of the root component. How would you handle this eventuality?
>
>
Sam Moffatt
plugin would because it's XML file would have the highest matching
version number and the component would smaller number because it could
reasonably be installed on a different version.
E.g.:
You're on Joomla! 1.7
The component is marked as 1.6 and you don't need a 1.7 manifest
The plugin is for 1.7 only and to prevent it installing in 1.6 you
flag it accordingly.
The plugin is sitting in a subdirectory of the component installer
(/admin/support/plgSystem17ExtraSupportShimThing).
The plugin has the highest matching version number and wins.
Sam Moffatt
http://pasamio.id.au
Jeremy Wilken
I think I understand what you are saying, if the package contains
multiple extensions in the main folder? I was not thinking of that, as
I zip each extension separately, and let the install.php file
unarchive and install each separately. That would be one way to
prevent this, but doesn't maintain current functionality well IMO.
This seems to be only a possible issue with components, as other
extensions do not have the ability to run an install script. So then
it follows that the installer should be smart enough to favor
component manifest files if they are found, ignoring others. Since
plugins/modules/templates don't install multiple parts, a component
file won't be found and it will just use the best matching version
attribute.
Does that solve the problem in your eyes too?
On 5 Mar, 22:34, Sam Moffatt <pasa...@gmail.com> wrote:
> The component would never get loaded under your model though. The
> plugin would because it's XML file would have the highest matching
> version number and the component would smaller number because it could
> reasonably be installed on a different version.
>
> E.g.:
> You're on Joomla! 1.7
> The component is marked as 1.6 and you don't need a 1.7 manifest
> The plugin is for 1.7 only and to prevent it installing in 1.6 you
> flag it accordingly.
> The plugin is sitting in a subdirectory of the component installer
> (/admin/support/plgSystem17ExtraSupportShimThing).
>
> The plugin has the highest matching version number and wins.
>
>
> On Sun, Mar 6, 2011 at 2:20 PM, Jeremy Wilken
Sam Moffatt
installer manifest that was the lowest in the tree. A manifest from
the root of the file (or an immediate subfolder) is more likely to be
the candidate than the one that is another few folders down. Those
manifests on the same level would then be considered against each
other leaving sublevels alone.
As of 1.6, plugins and modules can run script files and there was a
request that templates do the same that didn't sound like a bad idea
either (must admit I was originally skeptical).
Cheers,
Sam Moffatt
http://pasamio.id.au
Beat
bring memory and installation timeout issues, as some components have
heavy XML files that have nothing to do with the Joomla installer
(e.g. for JForms or other structural descriptions).
A bit of naming convention order (e.g. ".joomla.xml" file extension
and maybe name_of_extension.joomla.xml filename) and hierarchy order
could help avoiding to scan huge sets of files and heavy XML files.
Which OTHER xml files to scan could be a directive inside the main XML
file.
Just another 2 cents to the subject. ;-)
Best Regards,
Beat
http://www.joomlapolis.com/
Sam Moffatt
parse files until we find a valid installer file. If we ignore files
we find in subfolders after we've found at least one valid XML file
then we save ourselves parsing those files as well while still
enabling multiple install files at the same level.
Cheers,
Sam Moffatt
http://pasamio.id.au
Jeremy Wilken
should stop recursing directories as soon as the first install file is
found. That means the only requirement is that install files have to
be in the same directory for different versions, which sounds pretty
logical to me. So taking the example again with a 1.7 installation,
1.6 component, and 1.7 plugin, since historically the install takes
the first valid install file it finds to use, we can use that logic.
The first folder that a valid install file is found in (preferably the
root), is the last folder to be searched. So this means 1.5 would
still pick the first install file it finds, and the scenario you put
forward would not be an issue then.
@Beat - There is technically no restriction on the name of the xml
file for the installation. However, its pretty clear from my
experiences that just about all extensions use either
extensionname.xml or in some cases manifest.xml for their install
files. The issue is I want to maintain backwards compatibility in
conventions, so requiring a specific naming convention threatens that,
but it is certainly a worthwhile notion. It might not be an issue for
99% of extensions, but thats a guess rather than something I can say
for sure.
On Mar 7, 6:38 am, Sam Moffatt <pasa...@gmail.com> wrote:
> As an aside we currently do a recursive scan for all XML files and
> parse files until we find a valid installer file. If we ignore files
> we find in subfolders after we've found at least one valid XML file
> then we save ourselves parsing those files as well while still
> enabling multiple install files at the same level.
>
> Cheers,
>
Sam Moffatt
Sam Moffatt
http://pasamio.id.au
Beat
purposes:
1) installation
2) description of configuration params / fields
Joomla 1.6 has introduced a major incompatibility by renaming params
to fields, making the description of configuration params/fields
incompatible with previous joomla versions, which were compatible.
And for "2)", joomla is pretty strict on filename it is looking for.
So now that a way to solve the installation issue has been found,
Would it be elegant to use same method to find the right XML file for
configuration params/fields as for installation too, instead of a
"hardcoded" name ?
> Sounds good! Look forward to seeing it in action :)
>
Sam Moffatt
> One thing to keep in mind is that most joomla XML files have 2
> purposes:
>
> 1) installation
> 2) description of configuration params / fields
Keeping in mind params in the installation file are only used as
defaults as of Joomla! 1.5; com_config refers to config.xml instead.
For a good tutorial on how component params work in 1.5, check out
this article on the wiki:
http://docs.joomla.org/Component_parameters
The details are for 1.5 not 1.6 however it is mostly applicable with
the exception of the changes in the XML format and the fact that in
1.6 JParameter has been removed (you can use JRegistry as an
equivalent with loadJSON and loadINI for 1.6 and 1.5 respectively; the
automatic form generation isn't a part of JForm by default however
that issue is covered elsewhere on these lists).
>
> Joomla 1.6 has introduced a major incompatibility by renaming params
> to fields, making the description of configuration params/fields
> incompatible with previous joomla versions, which were compatible.
Yes, the params have changed format since they were introduced around
2004 (perhaps earlier, not sure). This is part of the new JForm system
which also enables tabbed configuration in components for grouping
plus provides hooks to programmatically manipulate and extend forms.
Converting across many of my params was a matter of doing
s/<params/<fieldsets/ and adding the extra attributes to the field set
and then s/<param/<field/ to get the fields across. If you had custom
JParam elements then it is a bit harder.
Joomla! 1.6 also swapped from using INI in the database to using JSON
to get a performance boost particularly on sites that are param heavy
(e.g. large amounts of modules particularly with lots of params or a
large number of plugins again with large numbers of params). This also
applies to a lesser extent with component params and template params
since they're usually only loaded once on a page load. Anything else
using params will have similar improvements when switching from INI to
JSON.
>
> And for "2)", joomla is pretty strict on filename it is looking for.
>
> So now that a way to solve the installation issue has been found,
> Would it be elegant to use same method to find the right XML file for
> configuration params/fields as for installation too, instead of a
> "hardcoded" name ?
The only reason we're hunting is that during installation we don't
enforce a particular location for the installer of either having it in
the root, a subfolder or really anywhere in the directory structure -
let alone the fact we don't necessarily know what we're installing
ahead of time. Changing would potentially break more BC and I believe
we're trying to avoid that. This will add support for multiple files
for the installer to enable flexibility there with different releases.
I had personally hoped for them to be as backwards compatible as
possible though others have made changes that made that impossible
unfortunately. I'm hopeful we won't have that happen much more going
forwards but having support for multiple install files enables other
stuff like different SQL statements per version, different script
files per version and different update sites.
What is the advantage of having this live in a random file? It would
add extra complexity to com_config to have it now hunt for files as
well the installer whose reason is the fact that it doesn't know what
you've just handed it and is digging around to guess what it should
install. At the moment the config.xml file will work with 1.5 params
and 1.6 fieldsets from the testing I've just done. They're duplicated
which sucks but they do work side by side.
Matias Griese
of the first components (among CB and some others) where we have tried
to build only one installation package for J1.5 and J1.6.
Because of some incompatibilities or inconsistencies between J1.5 and
J1.6 manifest files, we came into this same conclusion -- the best way
would probably be to allow 2 or more manifest files for different
versions of Joomla.
Kunena 1.6.4 is already using two manifest files and relaying that
J1.6 version gets loaded first and ignored in J1.5. This has been done
by naming of the files (for Windows) and by file creation order inside
zip (for Unix). This seems to be the perfect workaround for the time
being.
Modules and plug-ins are much harder to deal with, mostly because of
they only accept one filename for the manifest file.
My suggestion how to make multiple manifests to work is very simple
and needs only minimal amount of changes:
Do the manifest search as now, but instead of returning the first
match, change the logic to pick up the best match from the matching
directory. After this the file should be copied into standard name:
kunena.j16.xml becomes kunena.xml (named after extension name) to make
modules and plug-ins to work. This also means that we need to find the
file only once -- in live installation there would be no need to find
the component manifest anymore.
The harder question is how to get Joomla 1.5 manifests to comply the
new standards better:
http://docs.joomla.org/Developing_a_Model-View-Controller_Component_-_Part_1#Wrapping_It_All_Up_-_Creating_the_hello.xml_File
The worst issue is extension name -- which used to be something like
"Kunena" or "System - Kunena Plugin". These names will be saved into
extension table instead of using the new naming: com_kunena or
plg_system_kunena. By not using the new naming conventions any code
relaying on the right names will break up. This happens with the
language strings and I've seen some other side effects, too.
Jeremy Wilken
Installer scans for xml files, completely recursive. This is standard
behavior. The first valid manifest file that is found is checked
against the current version of Joomla. If they match, it is set as the
manifest file and used. If it is a valid install file, but not
matching the same version, it is put into a possible list of
manifests. Then, the current directory of the first matched install
file is the only directory further processed (to prevent loading and
scanning lots of extra xml files). After any additional possible
installer files are found, the closest version (without going over) is
selected and set as the manifest.
I also added a check if no files match the current version, for
example in cases when someone tries to install an extension for 1.8 or
greater on 1.7.
Things that need to be worked out:
- Since multiple manifest files can be used, the installer needs to be
smart about renaming the manifest files depending on which extension
type (for installer maintainability stuff), this way it won't matter
what you name the files, so long as they are all located in the same
folder. I am considering adding extra functionality to favor files
named in a specific way, but I need to look at that more. Right now
the installer does not care what the manifest name is (at least not
during the installation phase, the manifest has nomenclature once its
installed yes) so this makes most sense to me.
- The extension name is an issue, but not something that can be fixed
in 1.6 so we have a problem there. I propose to add a new tag called
<identifier> and return the <name> tag to the 1.5 functionality, but
this won't work in 1.6 and I don't know what can be done about that.
Jeremy Wilken
Jeremy Wilken
elin
Jeremy Wilken
Ron Severdia
Jeremy Wilken
Mark Dexter
The initial timeline we proposed was based on the idea that the platform would be merged back into the CMS in May. This process took longer than initially planned, and that made the inclusion of features dependent on the platform more problematic.
This was also our first time through a six-month release, and I don't think anyone expected we would have all of the process perfectly thought out or that things would work without issues. Hopefully for the next release, the process of managing changes in the platform and coordinating them with the CMS will be better worked out.
There is always a risk that any coding work we do will not get included in the code base. This is true for bug fixes and new features. It can be disappointing when it happens, especially when it is due to circumstances outside our control.
In any case, thank you for your efforts. Mark
It was out of sync because the platform had moved and was being developed privately for some time, and only 'official' after I had coded and committed my changes. It was set to pending, and then reset because of the sync issues, which were being done privately and without forewarning. Also the initial timeline would be merges in early May, as I was told by Mark. The issue is thus, it was coded, ready by the initial timeline and in the correct place, but it was ignored. All of those details were changed privately without warning, and therefore it is my fault that my feature wasn't put in? This is not the response I would hope to receive from trying to contribute, and in fact having done so to the guidelines of the time.I have already said I'm not interested in working on it further, so if anyone wants to take my patch or branch from SVN, feel free to re-implement the feature.
--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/joomla-dev-cms/-/Z0DDCvEK9V0J.
Andrew Eddie
remember that we are dealing with volunteers giving their time, and
sometimes that time doesn't fit in with "your" (the royal your)
schedule or Joomla's for that matter. There have been a lot of
changes taking place within the project over the last twelve months,
and more to come, and unfortunately, like this case, some people's
work is going to fall through the cracks or set at a lower priority.
It's nobody's fault, nobody needs to be "blamed" and I certainly hope
there isn't a suggestion that it's been deliberate act. The best we
can do is accept everyone had the best intentions at heart and work
together to make sure these instances are minimised in the future.
This was always going to be a tough release and I think those
involved, at whatever level of commitment, need to observe the virtues
of tolerance and patience when things don't quite go right the first
time round. Not everyone got what they wanted for a whole gamete of
individual reasons and nobody planned to make anyone's life miserable.
Several other branches didn't get in but I know people have taken
that on the chin and are working hard to do their best to get it in
1.8 or the platform.
As already mentioned, I can sympathise with the disappointment because
I've been there myself many times and know how it feels, but you pick
yourself for "next time" and I think under the circumstances everyone
involved did the best they could for this release, which, honestly,
has surpassed my expectations. This first time round was never going
to be perfect, but let's keep the banta professional and keep our
expectations of what others will do "for the love of it" realistic.
Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla 1.6 developers
elin
Amy Stephen
Andrew Eddie
promise), to better clarify my attitude and position (which some
people are taking the wrong way).
I can't speak for the PLT, but I can say offer on behalf of the
platform maintainers, and all those wonderful people that sent in pull
request, an apology for inconvenience caused to developers during the
last two months of the 1.7 release cycle where the platform was in the
process of being merged. On the whole, it's gone smoothly but cases,
like Jeremy's, have occurred which is unfortunate, unplanned and
generally undesirable. Platform and CMS maintainers should be talking
soon about this, and many other issues that have presented
post-separation - but don't let this stop people throwing ideas out
there as well.
I do hope everyone can be patient during this time and all work
together as we work things out.
Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla 1.6 developers
> --
> You received this message because you are subscribed to the Google Groups
> "Joomla! CMS Development" group.
> To view this discussion on the web, visit
> https://groups.google.com/d/msg/joomla-dev-cms/-/7QvuIm_ov8UJ.
