RFC Drop support for MS SQL and Postgres

[brian teeman]

The reality is that we don't have the volunteers with expertise with either of these databases. As a result bug reports remain for a very long time rendering Joomla unusable with either of these two DBs for a long time now.

In addition I wonder how many extensions even work with anything other than mysql? I suspect its very few. Can we really say that Joomla supports MS Sql and Postgres if you cant install the majority of extensions and core functionality is completely broken.

I would like to suggest that we change our supported platform to state that MS SQL and Postgres are experimental and unsupported or even better take the brave step and say "we tried to support them but there isnt enough  demand so we are dropping them"

It's good to fail. It means you are trying hard - the challenge is to acknowledge failure and move on from it.

[Johan Janssens]

+1 here. 

In the past years since support was made available we have only received 2 at requests at Joomlatools from customers who has an interest to run our extensions on MSSQL. Never received any inquiries about Postgres. 

A database driver for MSSQL and Postgres could still be offered as an installable library through Composer for those developers that need it and be maintained by the community.

Johan

[Hannes Papenberg]

+1

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

[Mark Dexter]

+1

[Parth Lawate]

Not an expert on this.. But i have seen a lot of MYSQL bashing happening around where Postgres scores . So might make sense to keep that one ?  ON extensions we do get cases with Joomla running on Windows servers but MS SQL has been extremely rare .. 

[David Hurley]

+1 Great suggestion.

[brian teeman]

As an example it has not been possible to create ANY content with Joomla under postgres since at least Joomla 3.2.2 2014-02-06

[brian teeman]

I tested installing the first 10 extensions as listed in the Install from Web plugin (which itself will not install without errors)

Kunena - FAIL

JCE - FAIL

ALLVideos - FAIL

Ozio Gallery - SUCCESS

JEvents - FAIL

Community Builder - FAIL

Simple Image Gallery - SUCCESS

Akeeba Backup - SUCCESS

[rolandd]

I am a big fan of this and fully support the move.

Unmaintained and broken code is better to be removed.

+100 :)

[dgt41]

+1 from me as well

[Valentin Despa]

This is my opinion just about PostgreSQL, which I have only used in a testing environment and submitted and tested some patches.  But I understand that Microsoft SQL is suffering from the same disease. 

I wrote a couple of time about the issue with PostgreSQL in Joomla: Can I use PostgreSQL with Joomla 3.3, Request for testing: PostgreSQL for Joomla 3.3 and others...

I actually tried to get Joomla to work decently with this database as part of my contribution to the Joomla project. While technically it would not be a very difficult thing to make it work, we are lacking some know-how on the best approach and unfortunately the community interest almost zero. A couple of people (probably no more than 10 over time) sporadically submitting or testing a patch. Fixing major bugs (like SQL errors saving articles, tags etc) takes months. And it is a matter of time till a poorly thought query in MySQL, breaks in PostgreSQL. I actually have never seen PostgreSQL working without an error.

Practically I am wondering if anybody is using it at all, as the interest is very low and the product actually does not work! Even numerous postings on this group, attracted very little contributors. I would honestly like to hear from somebody running a live Joomla with this database.

After many many months of struggling, we as a community are not capable of delivering a decent support for this database. We should admit it, dump the whole thing and move on. So +1.

Probably a first step would be on installation to add a clear warning that this database is not recommended. It will save everybody some time.

[Robert]

+1 drop it

let us do one thing good as 10 half way good

[Leo Lammerink]

+1 drop it ;-)

Leo

On Tue, Sep 30, 2014 at 9:22 PM, Robert <rde...@googlemail.com> wrote:

+1 drop it

let us do one thing good as 10 half way good

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

[Vic Drover]

May I vote? If so, +1 :)

Cheers,

Victor Drover
Founder and CEO, Anything Digital LLC (BBB Accredited)
Co-founder, Watchful.li & jInbound.com
262-309-4140
Facebook: AnythingDigital | watchfulli | JInbound
Twitter: @AnythingDig | @watchfulli | @JoomlaInbound

[Ole Ottosen (ot2sen)]

+1 Good idea

[David-Andrew]

Do we even need any more votes?

Just in case:

+1

[George Wilson]

I'm gonna say +1 for MS SQL but -1 for Postgres. Back at the beginning of the 3.x series we did actually have a small but significant number of people interested in using Postgres. I think the reason we had more and more people stop was because as previously mentioned we in core did little to stop the bugs creeping in and few patches made it through.

If we did a code sprint (or even just a evening where a handful of us sat down) I think we could easily get Postgres in a supported position again. If we push this hard I don't think it will be too hard to get more support on board from extension Dev's because currently why support this thing when Joomla itself doesn't?

To keep support up I suggest we ensure we run the selenium test suite against Postgres before every release. That gives us an easy way of ensuring functionality is the same across databases. Eventually I believe the plan is to integrate the selenium suite with travis anyhow through (There's already an ongoing effort for this on com_localise which is successful will probably be bought forward into core)

Finally why Postgres and not Ms SQL? Because Ms SQL has required extra hacks in various core controllers that just isn't needed with PostgreSQL - I think the majority of issues in Postgres so far seem to originate in the driver rather than in the queries themselves.

I'm not saying it's a quick fix and I can easily see why we could drop support for these two easily. But just personally think we have a real advantage over our competitors here and it would be a shame to see us drop it!

Kind Regards,
George

[Sergio Manzi]

I'm unsure.

From a personal and practical point of view I would vote +1000, if allowed to. I've always used a LAMP/WAMP stack for Joomla and I have no need for Postgres and/or SQL Server at this time and the foreseeable future.

On the other hand it seems to be quite a major "retreat" and break of BC.

Probably 4.0 would be a better timing for that.

Is there anybody out there really using Joomla 3 in a Postgres/SQL server production environment? Their voices should have a greater statistical weight. Even more if they are committed to maintain "their" DB compatibility.

smz

[Fedir]

what about just move to PDO....

Середа, 1 жовтня 2014 р. 02:26:35 UTC+3 користувач Sergio Manzi написав:

[brian teeman]

On Wednesday, 1 October 2014 00:26:35 UTC+1, Sergio Manzi wrote:

On the other hand it seems to be quite a major "retreat" and break of BC.

As i said   it has not been possible to create ANY content with Joomla under postgres since at least Joomla 3.2.2 2014-02-06 

[Sergio Manzi]

Yes, Brian, that was clear enough.

This may also mean that there is a huge need to fix  things up, *if* there is a real request from J3+Postgres (and/or SQL Server) users.

If, on the contrary, there is nobody out there asking for it, well, we can just let the ball drop.

Fedir's suggestion to move to PHP Data Objects sounds interesting too. Will there be inefficiencies? This is anyway definitely something best left for 4.0 as we should also ask 3rd party developer to do the same (I think).

[brian teeman]

On Wednesday, 1 October 2014 09:41:24 UTC+1, Sergio Manzi wrote:

Yes, Brian, that was clear enough.

This may also mean that there is a huge need to fix  things up, *if* there is a real request from J3+Postgres (and/or SQL Server) users.

See valentin's earlier comment. Also note that a fix has been waiting for this specific issue (and others) for months. Have there been thousands of users reporting it? No. clearly there arent many postgres users. If you compare that to the number of users reporting the issues with 3.3.4 in just 24 hours the answer is clear.

[Sergio Manzi]

Fair enough... :-)

[Yann Loosli]

Hi everyone,

I was told by Alex from SourceCoast about this thread which is surely of concern to me.
We are using Postgres with joomla for a couple of months in our project.
It is surely buggy. But yet it's there.
We've moved on this database because we are making a webgame engine component for joomla and our past experience on mysql is that it can quickly get overload by the amount of data generated by a game. It's running smoother under postgres.

The two major issues we've run in so far are : the "group by" option is most of the time incomplete for postgres and the sql dump install are not provided for postgres in extensions.
But if developers use the database driver, things can run.

The final question is : why didn't we submit our corrections ? Well - it's rather simple : we made some corrections inside joomla because it was blocking us - not for the sake of joomla (which is bad, I guess :-D ). Tell how to proceed and we'll try to find a couple of minuts to send these corrections.

PS : I am not a programmer myself, so don't ask me too much tech details, I wouldn't be able to answer :-/

PPS : my vote : -10000000000000000000000000000000000000. At least :-p

[Yann Loosli]

-1.

We are using postgres with joomla because this database is greater for our needs.

Basically, it's quite true that nowadays joomla postgres flavored version is totally unusable out of the box.
There are two main reasons :
1°) the GROUP BY in the core components of joomla is not adapted to postgres requirement. As it is used everywhere, joomla is broken in every single feature, from user management to articles and so on.
2°) the SQL dump installation files in extension are rarely provided, even when the code respect the db driver syntaxes.

Also it is true that it is not ready for production, postgres support is not aiming the same public than Mysql.

For the one who might wonder why we didn't submit our corrections, this is rather simple : we didn't even think about it because what we have in our heads is our project, not joomla. Yeah, I know, it's bad. We'll have a look on how we can have our part of job for joomla done.

So, a big -1.

[Valentin Despa]

I don't know where Postgres users are hiding, but this would be a good place to get some input.

Thanks @Yann for dropping a comment on this topic.  It is certainly a nice thing to hear from somebody who is actually using this database! I am just curious what Joomla version are you running and what is the cost (in hours) when doing an update.

Thanks for offering some patches to fix things. But this is not our issue. Let me put in with other words. This is not about submitting a couple of patches! This is not about doing a sprint and fixing some things here and there. Here is the math: one person submits a patch, two need to test it. Bug gets fixed. This issue is that those 3 people are very hard to find (or missing almost all the time). Bugs occur all the time. Each release should be tested by at least one person, to hopefully avoid braking things. So we are talking about long term involvement in making Postgres work with Joomla. 

We need a dedicated group of at least 3 people using in production Postgres, submitting and testing patches. Does anybody outside the JBS volunteer to be part of such a group?

I know that PostgreSQL is gaining in popularity as a database. And I would love to see it work properly in Joomla.

[Leo Lammerink]

With all respect but why are you (seemingly Postgres users) coming out only now? As mentioned since J3.2.2. we have not been able to create any content nor do (most of the extensions) work with this database. Maintaining this "just for a few" is costly in manpower if we can find the people capable of doing do and we better use the resources for the bigger case.

Johan Janssens wrote and I quote to make sure:

A database driver for MSSQL and Postgres could still be offered as an installable library through Composer for those developers that need it and be maintained by the community.

And I like the last piece " maintained by the community if wanted"

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

[Hannes Papenberg]

Would it help you if you ran Joomla from MySQL and had a postgres driver to store/access your game data? I doubt that anybody wants to remove the postgres driver itself...

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

To post to this group, send an email to joomla-...@googlegroups.com.

[Omar Ramos]

For me, it's never been about using these other database drivers to run Joomla on top of that particular database type (I'm perfectly happy to run Joomla on MySQL and I don't feel very strongly about us really supporting Joomla for the other platforms necessarily).

For me, the other database drivers (primarily the Oracle one, which I needed so I built it) have been mainly there so they could be used by a developer who was building something custom for their organization so they'd have the ability to retrieve data from the other database (Oracle, MSSQL, Postgres) easily using the typical Joomla database methods should they have that need.

-Omar

To post to this group, send email to joomla-...@googlegroups.com.

[Matt Thomas]

That's a great point Omar, thanks. Worth taking into consideration when discussing this.

[Michael Babker]

The drivers themselves have practical use cases beyond the CMS and the actual code for them should continue to be distributed as part of the full Joomla suite.  However, in terms of an application truly being multi-database compatible, it really needs to be built ground up with that support and not hacked in as it really feels like is the case with the CMS' support of SQL Server and PostgreSQL.  Even I stepped back from trying to work with those databases; the amount of effort was just too much and it really felt like there was next to no gain considering those who might be using the CMS in those environments are a very quiet minority.  If we can't get some folks dedicated to getting things up to speed and maintaining it, then we should drop the application's support for those platforms.  I'd say the same if we were discussing IIS or Nginx support as well.

[Tim Plummer]

I personally use MS SQL for a couple of Intranet sites and internal applications. These are not running the current version of Joomla due to stability, but I would like to upgrade them in future.

I also support MS SQL database type in many of my extensions. From an extension developers point of view, there is not much extra effort required to support additional database types. That said, I agree with the previous comments that there is not much demand from users for MS SQL support. I don't use PostgreSQL at all, but certainly would look into adding support for my extensions if there was some demand.

I understand the additional effort required to maintain these additional database drivers, but if you want Joomla to be more attractive to the corporate market then they will want to link it with their other systems, many of which run on MS SQL or Oracle.

If the decision was made to remove these additional database drivers from the core, I'd definitely still like to see them as an optional package for those that choose to still use them. I would hate to see us loose this functionality completely.

[aquilarubra]

I am a happy user with Postgre. When I was looking for a CMS, I was only considering those that support Postgre. I had horrific experiences with Mysql and would never use it for a serious website.

The point is that, if you want to keep Joomla for the small sites, Mysql is just fine. If you want bigger and professional sites, only Postgre gives you the confidence and response times you need. So, the question really is, will Joomla just stay a toy or will it be a five star reliable CMS?

I know, Postgre support is still poor in Joomla. I have a list of fixes that I run every time I upgrade and everything works as a charm. Currently, I use version 3.3.3, with no problem to add content, etc.

My feeling is that there are only a few small details (bugs) that prevent flawless Postgre integration. The core is solid and is thought well.

About plugins, I would not put a limit on that. I mostly use the core functionality, with just a few plugins, just because I don't need more. But it is up to the developers to test with Postgre. I see no problem if they develop Mysql only plugins, and I have no need to use them. I would rather develop my custom plugins myself if there is nothing around. So, it depends on the developer *if* they want to expand their user base. This does not mean that Joomla has to drop Postgre support just because most developers are not developing for Postgre.

Moreover, the differences between Mysql and Postgre are minimal, as far as Joomla is concerned. So, really little effort is needed to keep things up to date.

I don't have the time to participate in the development, but if somebody wants, I have no problem to share my list of fixes to make everything work.

[brian teeman]

So we now have two users who happily use postgres AFTER applying their own fixes. What do we have to do to get those people to contirubute their fixes. If they arent willing to do that on a regular basis then is it really the experience we want to give people. "Sure you can use postgres but you will have to fix a lot of things every time there is a release and you are completely on your own with that" - is that really the impression we want to be giving. Not everyone who tries to use Joomla with postgres will have the skill, time, will to edit the code to make it work.

[Aquila Rubra]

As I said, fixes are really few. It would take so little time to have
Joomla work out of the box. It is silly to do 99% of work and then
drop it because you don't want to complete the remaining 1%.

As of myself, I routinely upgrade Joomla. Thus, I can easily send the
fixes. I don't have the time to participate in the development, so I
would just post my list of issues/fixes.

Postgre is just a little stricter than Mysql. If Joomla works with
Postgre, it definitely works with Mysql too (which is less strict).
So, compliance with Postgre is an insurance that the code is well
written. The few fixes I have are just poorly written sql.

> --
> You received this message because you are subscribed to a topic in the

> Google Groups "Joomla! CMS Development" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[brian teeman]

It is not about Joomla not wanting to do the 1% - we currently dont have people with the skills and those that do have the skills (like yourself) keep the fixes to yourselves.

Sorry if I sound frustrated but you are happy to take but not to give

[Aquila Rubra]

That's not true. I said, I can post regularly the fixes. I can send
them to this list if you wish, I have a file with all steps and
details.

I use Joomla with multilanguage support and sef. So, I had to fix
Joomla in some areas that not everybody uses. But, as I said, fixes
are simple and few.

[Aquila Rubra]

By the way, I found this forum only because a developer of Postgre
plugins for Joomla told me about this thread. I am sure there are many
other users.

[George Wilson]

Aquila - can you please post the file with the steps and details and I'll submit the PR. With Valentin and Brian that's 2 testers there plus if they're your fixes I'll try and get a test in as well (I had postgres set up ages ago on my old laptop but will do it again on my main PC when back from JDay UK)

Kind Regards,

George

To post to this group, send an email to joomla-...@googlegroups.com.

[Aquila Rubra]

Here is the file. Not everything is related to Postgre and with my
latest update (to 3.3.3) some things seemed solved. So you have to
check yourselves against the latest release. But the steps are clearly
detailed.

[Yann Loosli]

Well, unfortunately, no, because the game engine is strongly linked to the core of joomla... so we need that postgres runs all through joomla core features.

[George Wilson]

Thankyou so much for this. I have a Joomla Day this weekend as I say so I may not have time until the end of the weekend - but will do my best to get as many of these in asap for Joomla 3.4

Kind Regards,

George

[Valentin Despa]

@aquilarubra First of all the approach of keeping fixes for yourself, as free as you are to do whatever you like with your time, is not only sort of a selfish but also damaging for Joomla as a community and putting the whole Postgres support in danger. Joomla cannot exist out of pure air. Somebody needs to put in that 1%... This is how open source works. Sometimes things get fixed by others and you can sit back and relax. Sometimes they don't.  And you need to step in, if you care about Joomla. 

I am sure that the combined effort of fixing Joomla after each update can be more wisely used by fixing and testing the core in the first place! Com'on, this is not rocket science. It is just good business to be involved in the open source communities from where you use the software to build your business. 

This discussion is not aimed against Postgres and MSSQL users. We are you trying to draw a conclusion, looking at what happened in the last year or more probably.

Realistically at this point you have JBS members with a strong MySQL background deciding on fixes for a database they don't use or know very well (no offense to anyone). So obviously without people with Postgres expertise, with real websites running Joomla, things cannot move forward. Secondly the JBS is anyway outnumbered and the effort to get Postgres to work is huge. And obviously when you put the effort in to look into issues, to propose a fix and for 6 months nobody gives a f*** than the signal is as clear as possible. 

"I don't have the time to participate in the development" - is it better for your business if Joomla is to completely drop support for Postgres?

Thank you for your fixes. I know they are trivial. But they also need testers not just fixes. Without dedicated people, Postgres will (out of the box) still be unusable in a couple of months, leading to the same discussion. 

[Aquila Rubra]

Valentin, what I can I have always given. I saw here the posts of many
young people. I am older than you, have kids and a personal life and
difficulties that others might not have. Like many simple users, I
have downloaded Joomla (which with Drupal is said to be one of the
most used CMS's) and used it. I had no knowledge that there were few
developers working with Postgre. Indeed, most of my fixes I found with
some searches online. Thus, I could infer that they were well known.
It did not even come to my mind to submit bugs because I thought
somebody already did that. And anyway, without offending anyone, I
always found Mambo-Joomla very buggy in general (even with Mysql,
which I was using in the past). But since it is so widespread, I
expected a good community to help resolve issues.

As I will be continuing to update Joomla, I have no problem to post
bugs and solutions. I always keep all steps detailed, as in the file I
attached. If you don't want Postgre support to fall into oblivion, you
should ask yourself what's wrong about the marketing of Joomla.
Perhaps, you should give better evidence that you need
feedback/support for Postgre, or whatever it is. It does not come
natural to people to step in, if they are not invited. And of course,
if you just drop support, you will never find more users that could
use Postgre.

And as final note, if you write Postgre compliant SQL, it will work
with Mysql. The opposite is not true. Thus, if you drop Postgre
support, you will just allow in Joomla more poor SQL code (see GROUP
BY clauses). At the end, all faults come out, so you will just have
more work to do in the future. If the SQL syntax is well written,
things will just work out of the box with Postgre.

I could suggest: in the routines that are in charge to execute the
queries, implement a check for proper SQL syntax. If the SQL command
does not follow a strict standard or lacks something, even if it could
be acceptable for Mysql, raise an error and display it. This way, no
developer would write non compliant SQL and it will always work both
on Mysql and Postgre. Even if you implement just a GROUP BY check (all
items in the SELECT must be in the GROUP BY also), you overcome 95% of
Postgre issues and you have a proper syntax.

[brian teeman]

On Thursday, 2 October 2014 11:24:41 UTC+1, aquilarubra wrote:

I saw here the posts of many
young people. I am older than you, have kids and a personal life and
difficulties that others might not have.

That is not true and irrelevant 

[Aquila Rubra]

This solution would also force plugin developers to fix the
defective/improper queries and make perhaps 80% of plugins immediately
compliant with both Mysql and Postgre. Or, just not to break
everything, you could just issue a red warning of non-compliance. You
can know exactly what is wrong (which query, what file, etc.), so the
users themselves would have no problem to fix things.

[Matt Thomas]

Aquila,

Thank you for taking the time to join this conversation and sharing your findings. That is very helpful. It is especially beneficial to know that writing compliant Postgres works well with MySQL.

I may, in fact, have a project coming up where we'd likely be using Postgres. I may just go ahead and look at starting to use it locally and see if I can help improve Postgres support in Joomla.

@All

Given this, Tim and Omar's accounts, I'm not sure if I'm ready to drop support for MS SQL and Postgres. I'm fine with with stating that they are experimental / unsupported / incomplete at this point in time, or even not supported by core extensions and meant to be used for interacting with external databases.

Best,

Matt Thomas
203.632.9322
http://betweenbrain.com/

Sent from mobile. Please pardon any typos or brevity.

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send an email to joomla-...@googlegroups.com.

[Aquila Rubra]

Well, Postgre is known as the opensource Oracle. I tested both Mysql
and Postgre with a 1.5Gb table. While Mysql was taking almost 2
minutes (!) for a simple SELECT, Postgre was taking 6-8 seconds.

Postgre has all the advanced features one might think of, but Joomla
only uses the basic SQL functionality. Thus, it is practically
identical to Mysql. The only difference is that Postgre is more
unforgiving: you must use a formally correct (academic) syntax.

> You received this message because you are subscribed to a topic in the

> Google Groups "Joomla! CMS Development" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> joomla-dev-cm...@googlegroups.com.
> To post to this group, send email to joomla-...@googlegroups.com.

[Leo Lammerink]

Ok so we now have 2 confirmed users and one PLT-member who might (!) want to use it and now the suggestion is to keep it alive by Matt based on 2,5 possible users out of how many thousands? Are we real here?

+1000 Drop

Leo

[Russ Winter]

it seems to me, that the choice is being made as the discussion has progressed;

Basically, there is interest in further developing a Multi DB support capability and that there is (in commercial terms) a "Business Desire" for it, however there is limited skills and resources to cover it fully at this time and it would appear that PG is more supportable and closely related to existing mindset than MSSQL.

So, my "Business Process Logic" says, describe them both as "Experimental" and make them "optional" for early adopters, with a plan of attack being that of, one-first, learn from it, develop the skills, and then approach the second.

Kick off a "Sub-Project" (Branch, I think you might call it) for PG first, the folks here having expressed an interest in PG support assist, where possible with developing that branch, with those other interested parties in learning more about PG, chipping in where and when feasible.  Once the branch is more stable, maintainable and mainstream supportable it can be merged in to production.

Once PG support is mainstream, some lessons have been learnt, further community support is established for MSSQL, then the same/similar process is implemented.

The benefits are that those in need have their requirements satisfied, the folks more involved in the hard work of developing and maintaining have a phased approach to learning, implementing and building and of course, Joomla! acquires additional audience growth and can further interest and attract PG and MSSQL skills over time.

just my 2c worth,

--
Regards,
Russ Winter

hotmango, Web and Print

To post to this group, send email to joomla-...@googlegroups.com.

[Mark Dexter]

Whatever we decide about official support, imo we should endeavor to keep the core code database neutral and fix any reported issues with non-MySQL dbs. For example, although MySQL has some handy non-standard features (GROUP_CONCAT for example) we should not use these in core. And we should fix the non-SQL-standard GROUP BY issues that cause errors in other dbs.

That way we keep our options open and don't deliberately make it harder to support other db's going forward. My .02. 

[Matt Thomas]

I couldn't agree more Mark.

Best,

Matt Thomas
203.632.9322
http://betweenbrain.com/

[Aquila Rubra]

The sub-branch is a good idea I also thought. However, Postgre fixes require to adjust core calls that must be tested anyway agaibst Mysql (group by clauses). There should be no issue with the two, but why keep roads separate if, anyway, at one point you have to merge them?

An experimental flag is more appropriate, but not a separate branch.

[Ronni Christiansen]

It seems like there is new people arising to the ocation and offering to help to maintain support for Postgre and MS SQL.

So i would go with trying to empower them to become a part of the ongoing efforts to be diverse in terms of DB.

I am sure that there is BIG Companies interested in MS SQL and ditto a lot of people prefering Postgre so if there is volunteers lets not remove it.

/Ronni

[brian teeman]

To be clear in my original post I suggested TWO options

I would like to suggest that we change our supported platform to state that MS SQL and Postgres are experimental and unsupported or even better take the brave step and say "we tried to support them but there isnt enough  demand so we are dropping them"

[Aquila Rubra]

Just in case I would like to join the Postgre developers, can you give
me a quick instruction how to do that?

[Valentin Despa]

Short introduction for aquira and others interested:

Requirements:

- github account

- know how to make a PR on github

- all the bug tracking / fixing activity happens on issues.joomla.org

Some documentation is available on docs.joomla.org.

http://docs.joomla.org/Portal:Joomla!_Code_Contributors

Get help on contributing from a local user group (if available) or go to almost any Joomla Day event.

Possible things you can do:

1. Search for existing issues. Look into them, confirm if needed, review the proposed fix and test it. Two testers are needed for each fix. So each bug needs the involvement of at least 3 persons.

2. If you found a new, unfixed and unreported issue, open a new issue, providing detailed explanation on how to replicate. Even if you have more issues found, try to split them into standalone things to make testing, replication easier.

----------------

I am wondering if it is not possible to reach more Postgres /MSSQL users and inviting to contributing by using the post-installation messages and pointing to a wiki article or something.

[Ronni Christiansen]

Maybe some of our good PLT members could help to setup 2 specific teams for Postsgr and MS SQL support and give people a place to meet? :)

--

[sovainfo]

Vote against both options.

Primairily because I don't see any benefits to do so. On the contrary it will hurt this community.

Consider the new product packaging in an experimental stage and not mature enough to propose separate packages for MS SQL and PostgreSQL. The core product is using too many old implementations that doesn't allow for this. Don't think these issues should be offloaded to those that are going to be responsible for those packages.

Wouldn't call these drivers a failure, suggest to look at the current bugfixing policy. The problems mentioned are not exclusive to the databases in this topic. Many no-brainer fixes have found their way into the product, many are still waiting.

[Leo Lammerink]

Sovainfo (Marcel)  +1

@ Ronny, why did these people not requested the support before this thread was opened by Brian? We never had the need posted for the past 2 years by any of these 'users' and now sudddenly we need to accomodate a very small group (!) of 'alien' db-users? Makes no sense to me!

I still suggest we focus on our main issues and not on a very, very group of active users!

Leo

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

[Aquila Rubra]

That's the proof that it's not worth to share fixes. Go ahead. I think I might do a Joomla fork. Maybe there will finally be a working Joomla.

You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[George Wilson]

No. It is worth doing the fixes. I've set up Postgres on my laptop earlier and am starting to work through your fixes. Again thank you very much for submitting them. It DOES make a difference

Kind Regards,

George

[sovainfo]

Don't consider this RFC very usefull. It only proves there are only a few interested in MS SQL and PostgreSQL.
We know who they are, the only new one here was pointed by a friend to this discussion. So, it made it to only one user.
Guess you will get more reactions of users on other channels than this one.

Provided support on the forum for many users that don't go near the developer site. They do find what is needed to solve their issues. Provided bugfixes there as well as JC/issues.joomla.org/Github. The current bugfix policy is not appropriate. Hopefully product packaging is going to spread the load to other people that take interest in those areas.

[alikon]

for my gsoc project experience when the problem arise

we have decided to focus tests only on mysql 

so +1 to experimental till we have people working on it

FYI

this week with the approval from PLT a new working group 

has been created the "sql optimization working group"

Main goals

We are focused to improve the Joomla database performance

we are looking for volunteers so just contact me

[Aquila Rubra]

I found this note also, if it could be useful:

Joomla with PostgreSQL update:

-Find the PostgreSQL update script (typically in
administrator/components/com_admin/sql/updates/postgresql)
and modify the following:
1. '0000-00-00 00:00:00' should be '1970-00-00 00:00:00'
2. Create table clauses need the correct owner. Add this command after
the create clause:
ALTER TABLE <tablename> OWNER TO owner_name

[Sergio Manzi]

IMHO, this seems to be a perfect occasion to perform not only optimization but a clean-up too!

smz

[Johan Janssens]

Guys and girls,

Let me tell you a little store about four people named Everybody, Somebody, Anybody, and Nobody.There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that because it was Everybody's job. Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn't do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done.

-- Anon

This little story is called decision paralysis and this thread is a great example. An often-observed pattern in user groups and community initiatives is that there is lots of good-will and ideas, which get discussed, but after all the discussion, nothing happens. The pros and cons are hashed out, and something approaching consensus occurs, and yet no action results. This is a case of too many cooks spoiling the broth - no-one takes responsibility for making things happen.

So here is an attempt to get come to a real actionable result.

Where do we stand

Brian initiated a RFC and 27 people have replied to the question : "Drop support for MS SQL and Postgres"

- Pro : 13 people have confirmed they are in favour of this

- Con : 1 person has expressed he is against 

Other people in this thread have commented but not expressed a clear yes/no opinion. Based on votes and comments we can say there are no strong arguments to keep the drivers in the core.  

What else do we know

- 99,99% of Joomla installations run on a shared hosts which only supports MySQL

- Almost no Joomla extensions support Postgres or MS SQL 

- Joomla users have no demand for MS SQL and Postgres

- The MS SQL and Postgres implementation is broken

- The development team has insufficient knowledge and time to work on fixing this

Clear cut conclusion

Conclusion here is as clear cut as things can get. Our users don't need it, they don't use it and it doesn't work. Which means this code is just bloat that is taking time away from tasks that matter a lot more.

How to move forward

The question is we need to keep the code, fix the code, ... etc is not relevant here. Joomla is build to be extendible. Not everything needs to be part of the 'core'. This is the true power of being a free software project, people can modify(extend) it. It's one of the 4 freedoms giving by the GPL license we all hold dearly.

Joomla core should provide and support those features that most of it's user base needs. For Joomla that's a simple straight out of the box LAMP stack. Nothing more. Anything else can be provided as an extension but doesn't need to be part of and maintained by the project. We have 8500 extensions to proof this model works. 

In the early days it was not possible to install libraries, today the Joomla installer supports installing a wide variety of different extension types. Including libraries, and single files. Nothing prevents the community to create and maintain their own database drivers for database engines not supported out of the box by the core. 

We all know the Joomla installer isn't ideal for installing developer libraries. This is where composer comes in. With the help of Composer those drivers can easily be made available to other developers. A Composer installer already exists that can install any type of extension in both Joomla 2.5/3.x and composer support is also coming in 3.4. 

Action steps to take 

1. Remove the Postgres and MS SQL drivers from the core codebase

2. Move them into individual repos on Github under the Joomla account

3. Open up a request to find a maintainer for each, make the maintainer committer (write access) on the repo. 

Those people in this thread who have fixes can submit  PR requests. The responsibility of keeping the drivers working will now lie with the maintainer and community. If no one steps up to be the maintainer of the libraries, the libraries are removed. Simple, easy, straight forward empowerment of community. 

Now. Who is going to make this happen ? Leadership team members, a hands-up please! 

Johan

[Ronni K. G. Christiansen]

Based on your summarization - id agree on the Action steps.

Id expect PLT to to take this up and moveo n it.

Den onsdag den 8. oktober 2014 03.52.09 UTC+2 skrev Johan Janssens:

Guys and girls,

[Leo Lammerink]

Agree with action steps proposed. PLT pls act as such?

[Bakual]

The nice thing about Joomla is that anyone can do it. There is no reason why PLT has to do it, and in fact I sincerly hope PLT will NOT do this work. All it needs is someone proposing a PR.

If PLT were to do that, then we end up being still responsible for the code. There would be absolutely no gain for us, in contrary we just have additional repos which need additional release management. Thus I rather have the driver in our core than having it to maintain in a separate repo.

Our current real problem and work comes from providing the SQL files (install, update, sample data) for all three database types. These are not easy to create and hard to test. PRs usually only have the changes for the MySQL type and usually Michael ends up doing the others. I would love to get rid of those files.

[brian teeman]

On Wednesday, 8 October 2014 07:45:44 UTC+1, Bakual wrote:

Our current real problem and work comes from providing the SQL files (install, update, sample data) for all three database types. These are not easy to create and hard to test. PRs usually only have the changes for the MySQL type and usually Michael ends up doing the others. I would love to get rid of those files.

There already is a PR to remove the testing sample data - its been there for a while 

[Johan Janssens]

Hi Thomas, 

It seems my post wasn't 100% clear.  I'll further clarify. 

History :  

The database layer we designed back in the 1.5 days was build to be extendible and offer support for different drivers. Not because we wanted Joomla to run on Postgres or MS SQL out of the box, but because we wanted to offer flexibility to developers to use different drivers in their (custom) components.

Key points :

-  The PLT doesn't have to do all the work. There are just things  that require a leadership member or someone with the proper access and permissions to do. Creating repos for example, or merging PR's. 

- I think we clearly demonstrated by now that the Postgres and MS SQL drivers do not belong in the core of joomla-cms'. They are not used, don't work and are not maintained.

- Additionally, if you keep them in the core the whole core needs to be tested against them, SQL install files created, every component tested etc. As you and others already pointed out, we don't have the people nor the time for this.

A few people in this thread have expressed interest in both drivers. To quote Yann :

We've moved on this database because we are making a webgame engine component for joomla and our past experience on mysql is that it can quickly get overload by the amount of data generated by a game. It's running smoother under postgres

This is a good example of how Posgres can be used. Developers will choose Postgress of MySQL for performance reasons or choose MS SQL over MySQL because they need to connect to a legacy database or a different system. This doesn't mean 'the core' needs to be able to run out of the box on MySQL or Postgres. 

You can build a custom component that uses Postgres as Yann is doing, or you can build a model that connect to a MS SQL database to get data out and use that. All perfectly possible.

So in short : 

- Maintain the drivers as part of the core is hard and complex, as the whole core needs to work with them and be tested on them. Moving them out the core still allows people to use them for building custom components but doesn't mean the whole core needs to be tested to be compatible with them anymore. 

- Based on the facted that people have expressed their interest in both drivers we are proposing a solution to solve the maintenance problem to core as outlined by Brian and at the same time give us, the community the ability to fix and use  drivers for custom development solutions. 

This a win/win for everyone. The core no longer needs to deal with the maintenance of the drivers as they are no longer a core feature, the community can more easily fix. Everyone happy. 

So to avoid any confusion, and for further clarification, above action steps with concrete assignments who needs to do what :

Action steps to take 

1. Remove the Postgres and MS SQL drivers from the joomla-cms repo on Github 

Assignment : Pull request any community collaborator can do this.  Code merge needs to be done by a member of the joomla-cms repo on Gtihub with write access.

2. Move drivers into individual repos on Github under the Joomla account 

Assignment : Repo needs to be setup by the owner of the Joomla organisation on Github. PR requests can be made by any community collaborator.

3. Open up a request to find a maintainer for each, make the maintainer committer (write access) on the repo. 

Assignment : Request and assignment process can be run by the community (we already have 2 people in this list who said they have patches ready), the actual assignment of the maintainer, adding him to the repo as member can only be done by the owner of the Joomla organisation on Github. 

Overview of assignments

1. Where do we need leadership team help for

- We need someone with access to setup the two repos, one for the postgres driver and one for the mssql driver. 

- We need someone with access  to assign a maintainer to each repo 

- We need someone with access to merge our pull request into the joomla-cms repo

2. What will we do ourselves

- We will submit PR's to remove the drivers from the core

- Will will submit PR's to fix the drivers, make them installable and maintain them further.

I hope that clarifies. Now I'll ask my question again. Who is going to make this happen ? Leadership team members, a hands-up please! 

Thanks!

Johan

[Michael Babker]

Just for a bit of clarity, the actual database drivers are working efficiently (sure there are a few bugs in all of them related to differences in the databases that haven't been caught; we even recently merged in a patch for the unused SQLite driver), it's the application level implementation that is plagued with failures.  So I would *NOT* remove the actual database driver code from the libraries folder.  Rather, existing installations should be alerted to the experimental like nature of their setups and advised that support may end in the future based on the issues and new installations be limited in being able to install with these databases.

[Sergio Manzi]

I totally agree with Michael!

Also, as already said, lets just fix (while optimizing...) our queries to be standard SQL compliant and probably everybody will be happy...

[brian teeman]

How about something really simple

1. Change the language file strings from "PostgreSQL" to "PostgreSQL - Unsupported" or "PostgreSQL - Experimental"

2. Remove the sample data files from the non-mysql installs.

That way NO code needs to be changed only language strings. We dont need to maintain the other sample data sets and users will be more aware about the true status of the non-mysql database support

That should take about 2 minutes to do

[Aquila Rubra]

Moving Postgre support to a module is not feasible. It only shows lack of understanding of how a database works. It needs to stay in the core. Fine if you want to flag it experimental.

I will point out that the last install was very smooth for me. I don't see so many problems except the few bugs that I submitted to this list. If something is mature and works, why talking about removing it?

If you want to solve the issue at its basis, move everything to ODBC. So everyone can use whatever database he wants. There would be no problem to maintain multiple databases. Of course, SQL needs to be well written...

If you have issues in converting install and data files from Mysql to Postgre, there is a number of tools that perform automatic conversion. It would be interesting to include a similar script in Joomla and finetune it to Joomla. With little effort, every problem solved.

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[Johan Janssens]

Decision paralysis ... I rest my case. 

[Sergio Manzi]

It can also be called "lack of consensus"...

--

[Sergio Manzi]

Perfect! (for the time being...)

I like "Experimental" more than "Unsupported"

[Aquila Rubra]

I think nobody will object about the experimental flag.

However, as I posted in the past, what works for Postgre works for Mysql. Mostly, it just needs proper GROUP BY clauses and it works out of the box. So, why concentrate so much to remove Postgre if it allows you to do an integrity/consistency check? If it works on Postgre, simply the code is well written. It should be the standard. If you only concentrate on Mysql, poor code will just increase.

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[Valentin Despa]

While Johan's theory of externalising things sounds good, practically things are not so loosely coupled as one would think. I don't see a straightforward way to separate Postgres from the core.

Actually the issues are mixed with the implementation as Michel correctly stated. This is not a small part of the system you can just separate and develop on your own. You have installation and update SQL files on one side. And you have some application issues with bad designed queries on the other side.

The drivers themselves do a decent job, there would be little gain in externalising them and there is not much to maintain. Yes, the database driver could be more smart (technically more STRICT) and complain when GROUP BY is missing when it should be there.

As Brian suggested, changing the language files and removing the sample data would be an easy but clear step forward: admitting that Postgres and MSSQL are not production ready and will probably be removed at a later point. Also it is important to notify existing users that the support for this databases will end in the future. 

[Sergio Manzi]

Agreed, but what about replacing the "will" with "may" in the last two sentences?

[Aquila Rubra]

If you want to underline the uncertain future of other databases support, it is best just to remove it. I would not continue using an uncertain product. However, as I said, I might do a Joomla Postgre fork, with a separate development.

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/VZNVSf4IfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[Sergio Manzi]

While I understand your frustration, I do not think a fork (I mean, a definitive "split") would be the best solution, nor for you, nor for the whole community.

Of course it is a possibility and a choice *you* should make, but probably the best solution can be to indeed make your fork but then contribute back raising issues and proposing pull request for the main repository.

My 2 cents...

Sergio

[Allrude]

Jupp the people, what if, but why not, and ... are also in all of these discussions, 

it is easier to agree on a parliament for Belgium then this

thanks for trying Johan

Op woensdag 8 oktober 2014 14:58:09 UTC+2 schreef Johan Janssens:

[Matt Thomas]

It is certainly not a wasted effort. It gave us much to talk about in today's meeting and we are discussing it internally. Expect some news once we have an agreement and formulate a plan to move forward.

Best,

Matt Thomas

203.632.9322
http://betweenbrain.com/

--

[brian teeman]

That is great news Matt 

Thanks 

Brian

[Valentin Despa]

I think you underestimate the effort needed just to make a fork. Trust me, investing a couple of hours to get these basic bugs fixed is much easier. 

Speaking of contributions, in just a week from now we are having a Pizza, Pizza! Bugs, Bugs! event (October 17th, 2014). Everybody is invited. It would be a great opportunity to bring Postgres back to life. 

http://developer.joomla.org/pbf

If anybody wants to attend but has no idea on how to get started fixing bugs, I can give you a hand.