Joomla! scheduled articles and Varnish/nginx

273 views
Skip to first unread message

Daniel Kanchev

unread,
Apr 17, 2014, 6:41:57 AM4/17/14
to joomla-...@googlegroups.com
Hello,

Sorry if this is not the correct group for my question/suggestion (feel free to move it to another group)!

I am sure that you're all well aware that web hosts nowadays use various caching technologies to speed up websites and handle more visitors. Here at Siteground we also use our custom reverse proxy caching solution which is based on Varnish.

By default Joomla! does not support reverse proxy servers and HTTP caching. To overcome this issue and properly invalidate the cache we created our own caching plugin for Joomla! - it connects to the Varnish cache and it invalidates cache entries when a user creates a new article/menu item or something else. This plugin works for 99% of the Joomla! situations.

There is a problem with Joomla! scheduled articles and Varnish/nginx servers. The issue is that I can create a new article and then set it to be published after 2 days but in this case we don't clear the cache just before the article is posted because there is no action to attach our plugin to. Joomla! uses the following scheme - every time you access a Joomla! site the CMS checks the publish_up field and determines if an article should be displayed or not. To do this the CMS queries the database (even if you have no articles scheduled for posting/removal from the site). I see two issues with this setup:

1. The current solution is not Varnish/nginx friendly and it is almost impossible to invalidate the cache right before the article/module should be posted and enabled on the site. This is caused by the fact that you need a real visitor to open the site and the index.php script to be executed (the PHP script is not executed at all when full page caching is used).

2. The current setup is not optimized because the database is queries every single time a user opens the site and even if there are no scheduled articles. The SELECT query should be modified to check the publish_up field field only if scheduled article posting is enabled (or at least this is how I see things). Image that you have 250K articles and you don't use automatic scheduling - then you simply don't need the additional CPU overhead caused by the SELECT queries. There are many solutions to this issue - even MySQL triggers can be used to change a flag and then articles to be displayed by a cron job.

What are your thoughts on this matter. Please feel free to offer different solutions :)


Kind Regards,
Daniel Kanchev
Web Apps Engineer & Performance/Security Guru
SiteGround Hosting Services

brian teeman

unread,
Apr 18, 2014, 5:33:58 AM4/18/14
to joomla-...@googlegroups.com
Daniel

I totally get where you are coming from but I just dont see how it can work in practice (and without changing existing expected behaviour of Joomla)

Global switch to support timed posts
For example on my blog I have only ever once had a post set to publish in the future - so I would presumably have set up my site not to support this when i built it 2 years ago and I can guarantee I would have forgotten all about disabling this when it came to the post. 

Hourly cron
What if I wanted to publish at a specific time?

Maybe someone has a cleverer solution that doesnt require me (the user) to think

Daniel Kanchev

unread,
Apr 18, 2014, 7:58:17 AM4/18/14
to joomla-...@googlegroups.com
Hi Brian,

10x for your comments :) Here is what I think:


On Friday, April 18, 2014 12:33:58 PM UTC+3, brian teeman wrote:
Daniel

I totally get where you are coming from but I just dont see how it can work in practice (and without changing existing expected behaviour of Joomla)

Global switch to support timed posts
For example on my blog I have only ever once had a post set to publish in the future - so I would presumably have set up my site not to support this when i built it 2 years ago and I can guarantee I would have forgotten all about disabling this when it came to the post. 

- This can be automated - the global settings can be disabled by default and enabled when you set an article to be posted at a specific date/time. After that when the article is posted (or removed) the setting will be changed again.
 

Hourly cron
What if I wanted to publish at a specific time?

I agree that this is a possible issue. That is why I believe that the best way to resolve the problem is to create a Joomla! internal scheduler (similar to the Magento cron system) which will be able to post articles exactly when they need to be posted. I am sure that Jisse Reitsma from Yireo will be able to provide more details about a similar Joomla! scheduler.
 

Maybe someone has a cleverer solution that doesnt require me (the user) to think

I fully agree that the user shouldn't be bothered at all. A pseudo-cron can be used to achieve this.

Thomas PAPIN

unread,
Apr 18, 2014, 8:08:49 AM4/18/14
to joomla-dev-cms
A solution could be :

1 ) Create a system plugin than will do something on each request OR NOT (plugin option)
- If executed on each request, simulate a "cron" with an option like do something every 1 minute , 1 hour, 1 day.
- If not executed (means that there is a cron task attached) => this will work with full page cache
3) If it's time to do something, check publish state , up/down date  and change state of content if needed
4) Trig an event to call some other  "plugins" (to clean varnish cache)
5) simplify SELECT queries.

>> when i built it 2 years ago and I can guarantee I would have forgotten all about disabling this when it came to the post.

If the global option is disabled, the "publish_up and publish_down fields" should be hidden (with a message: "To enable future publication please activate the global option or "my system plugin".

-----------------------------------------------------------------------------------
Please note, we could do something more more generic.

With a "Cron Task" Component.
Lot of components needs some "cron tasks" to send notification email, to unpublish, etc...
We could have a way to centralize all these cron tasks (with a specific schedule for each task) with plugin systems.

There could be 3 ways to execute these cron tasks (user decision)
- (a CLI application, the best for me) which will trig the plugin etc..
- a Web Cron
- on each request. (By default)




--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.

Thomas PAPIN

unread,
Apr 18, 2014, 8:10:07 AM4/18/14
to joomla-dev-cms
Daniel,

+1 for Joomla! internal scheduler :-), it's nearly what I mean in my email.

brian teeman

unread,
Apr 18, 2014, 9:50:18 AM4/18/14
to joomla-...@googlegroups.com
Don't forget with joomla were not just talking about as articles. We can also sent modules to be published at certain times and what about content that's always date sensitive such as in an events or calendar component. These are just some of the reasons that the system page cache plugin is not only disabled by default but pretty much a hidden feature.
Joomla isnt WordPress, were talking about a lot more than blog posts when it comes to time based publishing

Roberto Segura

unread,
Apr 18, 2014, 9:58:44 AM4/18/14
to joomla-...@googlegroups.com
I told to Michael about my idea for this a couple of weeks ago.

The idea is quite simple: a cron file executed that triggers specific events for example onCronJobStarted, onCronJobFinished.

Magento already uses it. On your hosting you create a cronjob to that cron file and done. We can also have a backup lazy cron system to relay as basic solution for those who don't manage to do it.

We would have a serie of plugins on plugins/cron whose purpose is only manage cron based things. We can also rely on plugins/system and just add the triggers.

With that the system has infinite uses, not only Daniel's issue.

Matt Thomas

unread,
Apr 18, 2014, 10:05:16 AM4/18/14
to joomla-...@googlegroups.com
That sounds very cool Roberto. I can already think of at least a half-dozen existing implementations where that would have been extremely useful.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

Dimitris

unread,
Apr 18, 2014, 10:09:15 AM4/18/14
to joomla-...@googlegroups.com


On Thursday, April 17, 2014 1:41:57 PM UTC+3, Daniel Kanchev wrote:

By default Joomla! does not support reverse proxy servers and HTTP caching. To overcome this issue and properly invalidate the cache we created our own caching plugin for Joomla! - it connects to the Varnish cache and it invalidates cache entries when a user creates a new article/menu item or something else. This plugin works for 99% of the Joomla! situations.

 Daniel
I am very curious on your approach, do you mind sharing some code?

Thanx 

Dimitris
 

Daniel Kanchev

unread,
Apr 18, 2014, 10:50:54 AM4/18/14
to joomla-...@googlegroups.com
Roberto,

This is exactly my idea :) The situation that I described is just one of the things that will be resolved. I really think that the Magento scheduler is a good start and it can be modified, so that it will work for Joomla! sites.

Daniel Kanchev

unread,
Apr 18, 2014, 10:54:06 AM4/18/14
to joomla-...@googlegroups.com
Dimitris,

I am afraid that we use proprietary Varnish software packages and I cannot provide the code. However, the Joomla! plugin that we use is accessible and you can download it and check the code:

http://download.siteground.com/jSGCache.zip

Dimitris

unread,
Apr 18, 2014, 10:59:08 AM4/18/14
to joomla-...@googlegroups.com
In case you're interested here is my way of using varnish to detect mobile/tablet/iphone and pass it to joomla. Saves some cpu :)

Jisse Reitsma

unread,
May 20, 2014, 5:40:51 AM5/20/14
to joomla-...@googlegroups.com
@Daniel, I had a good laugh with this. A potential customer came by a couple of days asking for functionality to allow my EmailScheduler extension to be used to mail users when a new article was published. However, because an article is not changed state when the article simply reaches its publish_up date, there is no plugin event that could trigger a mail. So now I bumped into this thread and found you mentioning my name :) lol

@all, I've published a plugin that addresses this very problem we are facing: https://github.com/yireo/plg_system_pendingstate/ The plugin simply introduces a new state "pending" which then allows other plugins (like Daniels Varnish plugin) to use the onContentChangeState event to do their stuff.

This works all pretty ok, except for the issues mentioned in the repos README file. I hope to be working soon on these issues, but any help is welcome. The main thing that prevents it from being a very generic solution is that only articles with state 0 or 1 are shown by default in the Joomla! backend. But if an user knows this, it is very usable. I'm actually going to implement it now on my Yireo site ;)

Regards,
Jisse

Reply all
Reply to author
Forward
0 new messages