Django CMS and Varnish

[Imran M Yousuf]

Hi,

I am fairly new to Django CMS, but am experienced in developing couple of large CMS itself and numerous websites using them. I have been basically scouting for a CMS that is well suited to be merged and blended into Web Application and Django CMS seems to be intended to work that way.

From a website point of view, one of the easiest way to optimize is to cache non-changing or less-frequently (relatively speaking) changing/evolving contents at markup level (i.e. saving CPU cycles to generate the markup on every request even if the object used are cached) works wonders when scaling an application. This is where ESI and Varnish becomes pretty interesting.

Now from my exploration of use cases, tutorials and a demo setup it seems to be Django is designed to be extremely modular but I could not find a way by design to generate fragments of a page or link the fragments etc. Furthermore, I can cache dynamic data (session dependent or frequently changing/evolving data) by not putting them in markup and rather later doing a AJAX call; but I am having inefficiency in terms of having to purge the whole page if a placeholder's plugins have changed; or the DTO linked to one/more plugin instance across pages change etc. Google did not help me much on this topic other than discouraging :).

So I have been wondering if anyone before me has attempted it; if you have please share your experience and route of work. If not, is it possible to pass request parameters or query parameters so that based on their values we change what the page template generates; also the challenge this adds is that, we somehow need to make these URLs 'known' to a system so that if dependent value(s) change we can purge it from Varnish Cache; I would like to learn of some ideas on it.

Thank you,

Imran

[Philipp Zedler]

Hi Imran,

I have no direct answer to your question, but a hint about where to get one.

The django CMS is only a very small subset of Django providing certain features for organizing pages. I think that your question has no special relation to the django CMS, but is a more general one concerning the django project as a whole. The django project has its website here
https://www.djangoproject.com/
And it has mailing lists with much more traffic than this one. You find them here:
https://docs.djangoproject.com/en/1.6/internals/mailing-lists/#django-developers

Good luck!
Philipp

--
You received this message because you are subscribed to the Google Groups "django-cms-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-cms-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Philipp Zedler
Developer für Django und Python
Am langen Rain 12
39031 Bruneck
Italien
++39 / 324 / 77 345 96
http://www.bruneck-berlin.de

[Imran M Yousuf]

Hi Philipp,

Thank you for your response. The reason I posted stuffs here is what I need is more at Django CMS level, e.g.,

1. Can I get Page URL on the webpage at the time of when page is updated?

2. Can I listen to Page update events?

3. If I want to generate fragements of each page how best to approach it? Examples of fragments would be -

  a. Generate fragments for each Placeholder only - also will need to get URL for these fragments

  b. Generate fragment for each plugin instance within a Placeholder - also will need to get URL for these fragments

Do you think these are possible in Django CMS? Or if I am missing something and these are core to Django can you please explain how?

Thank you,

Imran

Imran M Yousuf
Entrepreneur & CEO
Smart IT Engineering Ltd.
Dhaka, Bangladesh
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
           +880-1746119494

[Patrick Lauber]

On 19.03.2014, at 16:20, Imran M Yousuf <imyo...@gmail.com> wrote:

> Hi Philipp,
>
> Thank you for your response. The reason I posted stuffs here is what I need is more at Django CMS level, e.g.,
>
> 1. Can I get Page URL on the webpage at the time of when page is updated?

have a look at django signals. the Page model and the Title model are market dirty if something changes.

> 2. Can I listen to Page update events?

What is the end goal that you want to achieve?

> 3. If I want to generate fragements of each page how best to approach it? Examples of fragments would be -
> a. Generate fragments for each Placeholder only - also will need to get URL for these fragments
> b. Generate fragment for each plugin instance within a Placeholder - also will need to get URL for these fragments

What is the eng goal that you want to achieve with this?

[Imran M Yousuf]

On Wed, Mar 19, 2014 at 11:17 PM, Patrick Lauber <di...@treepy.com> wrote:
>
>
> On 19.03.2014, at 16:20, Imran M Yousuf <imyo...@gmail.com> wrote:
>
> > Hi Philipp,
> >
> > Thank you for your response. The reason I posted stuffs here is what I need is more at Django CMS level, e.g.,
> >
> > 1. Can I get Page URL on the webpage at the time of when page is updated?
>
> have a look at django signals. the Page model and the Title model are market dirty if something changes.

Ok, will do.

>
>
> > 2. Can I listen to Page update events?
>
> What is the end goal that you want to achieve?
>

End Goal is to support Varnish Cache using ESI fragments; I tried to
elaborate on it as follows.

> > 3. If I want to generate fragements of each page how best to approach it? Examples of fragments would be -
> > a. Generate fragments for each Placeholder only - also will need to get URL for these fragments
> > b. Generate fragment for each plugin instance within a Placeholder - also will need to get URL for these fragments
>
> What is the eng goal that you want to achieve with this?

What I want to achieve is use Varnish Cache; so if we can generate
fragments then we can add a configuration whether we want to render
our templates as ESI fragments or not. So here is what I had in mind -
1. We identify placeholders in the template
2. Instead of replacing the placeholder with HTML we rather replace it
with the Placeholder URI in a ESI tag
3. In the Placeholder HTML we expose each Plugin instance as ESI tags
4. When a Plugin instance changes and a page is "Published" we PURGE
the Plugin Instance URI from Varnish cache while retaining rest of the
placeholder and/or other placeholders
5. When a new Plugin Instance is added or one is removed to/from a
Placeholder we PURGE the Placeholder URI and its older Plugin Instance
URI

Similarly, I have intention of supporting the same with the Django
template; but that would only make sense if we can achieve the support
in Django CMS.

Thank you,

Imran

[Patrick Lauber]

On 19.03.2014, at 19:28, Imran M Yousuf <imyo...@gmail.com> wrote:

> On Wed, Mar 19, 2014 at 11:17 PM, Patrick Lauber <di...@treepy.com> wrote:
>>
>>
>> On 19.03.2014, at 16:20, Imran M Yousuf <imyo...@gmail.com> wrote:
>>
>>> Hi Philipp,
>>>
>>> Thank you for your response. The reason I posted stuffs here is what I need is more at Django CMS level, e.g.,
>>>
>>> 1. Can I get Page URL on the webpage at the time of when page is updated?
>>
>> have a look at django signals. the Page model and the Title model are market dirty if something changes.
>
> Ok, will do.
>
>>
>>
>>> 2. Can I listen to Page update events?
>>
>> What is the end goal that you want to achieve?
>>
>
> End Goal is to support Varnish Cache using ESI fragments; I tried to
> elaborate on it as follows.
>
>>> 3. If I want to generate fragements of each page how best to approach it? Examples of fragments would be -
>>> a. Generate fragments for each Placeholder only - also will need to get URL for these fragments
>>> b. Generate fragment for each plugin instance within a Placeholder - also will need to get URL for these fragments
>>
>> What is the eng goal that you want to achieve with this?
>
> What I want to achieve is use Varnish Cache; so if we can generate
> fragments then we can add a configuration whether we want to render
> our templates as ESI fragments or not. So here is what I had in mind -
> 1. We identify placeholders in the template

- placeholders are rendered via templatetags.
- placeholders have a unique ID
- i think it would be difficult to mark placeholders automatically at the moment without code changes
- a manual template solution may be possible

> 2. Instead of replacing the placeholder with HTML we rather replace it
> with the Placeholder URI in a ESI tag
> 3. In the Placeholder HTML we expose each Plugin instance as ESI tags

this could be automated with a plugin processor but would be difficult with inline plugins (plugins inside text plugins … for example link plugin inside text)

> 4. When a Plugin instance changes and a page is "Published" we PURGE
> the Plugin Instance URI from Varnish cache while retaining rest of the
> placeholder and/or other placeholders

placeholder output is already cached (when no dynamic plugins are present) with django.core.cache and the result is retrievable from memcache

> 5. When a new Plugin Instance is added or one is removed to/from a
> Placeholder we PURGE the Placeholder URI and its older Plugin Instance
> URI

Why no just render the whole page again if something changes and cache it again?

[Imran M Yousuf]

Hi Patrick,

On Thu, Mar 20, 2014 at 2:34 PM, Patrick Lauber <di...@treepy.com> wrote:
>
> On 19.03.2014, at 19:28, Imran M Yousuf <imyo...@gmail.com> wrote:
>
>> On Wed, Mar 19, 2014 at 11:17 PM, Patrick Lauber <di...@treepy.com> wrote:
>>>
>>>
>>> On 19.03.2014, at 16:20, Imran M Yousuf <imyo...@gmail.com> wrote:

>>> <snip />

>>>> 3. If I want to generate fragements of each page how best to approach it? Examples of fragments would be -
>>>> a. Generate fragments for each Placeholder only - also will need to get URL for these fragments
>>>> b. Generate fragment for each plugin instance within a Placeholder - also will need to get URL for these fragments
>>>
>>> What is the eng goal that you want to achieve with this?
>>
>> What I want to achieve is use Varnish Cache; so if we can generate
>> fragments then we can add a configuration whether we want to render
>> our templates as ESI fragments or not. So here is what I had in mind -
>> 1. We identify placeholders in the template
>
> - placeholders are rendered via templatetags.
> - placeholders have a unique ID
> - i think it would be difficult to mark placeholders automatically at the moment without code changes
> - a manual template solution may be possible
>

Can't we change anything in the template tag where depending on
request URI we decide whether to output a URI that would output only
the URI to the detailed output or the detail output itself? Since
placeholders have a unique ID could we do something like -
{page-uri} # Provide the ESI tag with the following URI
{page-uri}?placeholder={placeholder-id} # Provide the output for the placeholder

>
>> 2. Instead of replacing the placeholder with HTML we rather replace it
>> with the Placeholder URI in a ESI tag
>> 3. In the Placeholder HTML we expose each Plugin instance as ESI tags
>

> this could be automated with a plugin processor but would be difficult with inline plugins (plugins inside text plugins ... for example link plugin inside text)
>

Firstly, for simplicity initially let us consider only the top level
plugin instances and ignore the nested plugins; that means, if nested
plugin value changes then we assume the top-level plugin changed too.

>> 4. When a Plugin instance changes and a page is "Published" we PURGE
>> the Plugin Instance URI from Varnish cache while retaining rest of the
>> placeholder and/or other placeholders
>
> placeholder output is already cached (when no dynamic plugins are present) with django.core.cache and the result is retrievable from memcache
>
>> 5. When a new Plugin Instance is added or one is removed to/from a
>> Placeholder we PURGE the Placeholder URI and its older Plugin Instance
>> URI
>
> Why no just render the whole page again if something changes and cache it again?
>

The idea is in a page with lots of plugins and placeholders we only
concern ourselves with "portions" that changes; needing to cache the
full page is a brutforce approach that is the last option IMHO; I
would only consider it if doing what I laid out above is months of
effort or not doable at all.

Thank you,

Imran

>> <snip />


--
Imran M Yousuf

[Patrick Lauber]

One problem you will face is that plugins can insert css and js files into the head depending if they are there or not.
So the head will be a combination of js and css files from ALL the placeholders of the page.
They also can react to the context of the template they are in. So if you want to render only one placeholder you would need
to capture the context they are in somehow.
We already cache the placeholders and their JS and CSS output to head. And only if a plugin changes do we rerender
the whole placeholder. Maybe some of that functionality could be reused.

Just for clarification. In 3.0 for not logged in users there will be only 2 calls to memcache after a page is cached. No db queries and we measured about 20-40ms response times.
A plugin can define if it is dynamic and can bust the placeholder and page cache.

What kind of application/website are you guys planning?

[Imran M Yousuf]

On Thu, Mar 20, 2014 at 7:37 PM, Patrick Lauber <di...@treepy.com> wrote:
> <snip />

>>
>> Can't we change anything in the template tag where depending on
>> request URI we decide whether to output a URI that would output only
>> the URI to the detailed output or the detail output itself? Since
>> placeholders have a unique ID could we do something like -
>> {page-uri} # Provide the ESI tag with the following URI
>> {page-uri}?placeholder={placeholder-id} # Provide the output for the placeholder
>
> One problem you will face is that plugins can insert css and js files into the head depending if they are there or not.
> So the head will be a combination of js and css files from ALL the placeholders of the page.
> They also can react to the context of the template they are in. So if you want to render only one placeholder you would need
> to capture the context they are in somehow.
> We already cache the placeholders and their JS and CSS output to head. And only if a plugin changes do we rerender
> the whole placeholder. Maybe some of that functionality could be reused.
>

Yes it is true that plugins may add CSS and JS and that is in fact one
thing I found problematic since multiple instance of the plugin within
or across placeholders cause the JS and CSS to be added multiple times
to the <HEAD> element. I personally did not like it and would prefer
plugins' to document the JS/CSS they would want included in the <HEAD>
in documentation over including them in the template.

So I would not worry about that for this/our use case in particular
but in general this would be needed to be added as a "things to note".

> Just for clarification. In 3.0 for not logged in users there will be only 2 calls to memcache after a page is cached. No db queries and we measured about 20-40ms response times.
> A plugin can define if it is dynamic and can bust the placeholder and page cache.

This is actually a very good example you bring up; what we are trying
to develop is a sport aggregation website (and hope to re-use most of
it across different sports) where we will host sports specific blogs,
match results summaries, player statistics, videos, presentations of
plays, coaching materials etc. The hope page will comprise of Slide
Show of images from recent matches, player statistics (computed), team
statistics (computed), Events calendar, Editor's Pick of - plays,
blogs, matches etc.

Now lets consider we updated a match result which would only impact
the recent matches area, e.g., that placeholder would be 10~15% of
pages data; now the plugin basically depends on a model 'Match' so
when that cases we will look up for all instances of MatchPlugin,
RecentMatchesPlugin etc. associated with the 'Match' updated and PURGE
them from cache; now if we are to purge all the pages instead of the
plugin instances effected only, that would cause much larger amount of
data getting purged unnecessarily; similar would be the case for
player statistics.

So partial content purging would enable us (and the community) to
benefit from purging only what changed and leaving the rest unchanged.

>
> What kind of application/website are you guys planning?
>

As mentioned above we are looking to come with a set of websites for
different sports community and generate content community specific for
non-profit purpose. We also intend to extend it for some other social
cause. In addition I come from a CMS background for news industry am
trying to learn a new CMS to explore its strong points and to be
honest Kudos to the team for such a wonderful product/project; I am
really liking it. We are also evaluating it as a platform (among other
CMSs) for mult-site hosting solution that is community driven.

In fact describing the above scenario I am even more convinced
Varnish/ESI and Django CMS would be a extremely benevolent combo. If
you feel that the community will benefit in general I am more than
inclined to chip in with the time for developing, testing and using it
:).

>> <snip />

Thank you,

[Patrick Lauber]

On 20.03.2014, at 19:59, Imran M Yousuf <imyo...@gmail.com> wrote:

> On Thu, Mar 20, 2014 at 7:37 PM, Patrick Lauber <di...@treepy.com> wrote:
>> <snip />
>>>
>>> Can't we change anything in the template tag where depending on
>>> request URI we decide whether to output a URI that would output only
>>> the URI to the detailed output or the detail output itself? Since
>>> placeholders have a unique ID could we do something like -
>>> {page-uri} # Provide the ESI tag with the following URI
>>> {page-uri}?placeholder={placeholder-id} # Provide the output for the placeholder
>>
>> One problem you will face is that plugins can insert css and js files into the head depending if they are there or not.
>> So the head will be a combination of js and css files from ALL the placeholders of the page.
>> They also can react to the context of the template they are in. So if you want to render only one placeholder you would need
>> to capture the context they are in somehow.
>> We already cache the placeholders and their JS and CSS output to head. And only if a plugin changes do we rerender
>> the whole placeholder. Maybe some of that functionality could be reused.
>>
>
> Yes it is true that plugins may add CSS and JS and that is in fact one
> thing I found problematic since multiple instance of the plugin within
> or across placeholders cause the JS and CSS to be added multiple times
> to the <HEAD> element. I personally did not like it and would prefer
> plugins' to document the JS/CSS they would want included in the <HEAD>
> in documentation over including them in the template.

JS and CSS files should not be added multiple times but should be unique. If this happens this is a bug.

>
> So I would not worry about that for this/our use case in particular
> but in general this would be needed to be added as a "things to note".
>
>> Just for clarification. In 3.0 for not logged in users there will be only 2 calls to memcache after a page is cached. No db queries and we measured about 20-40ms response times.
>> A plugin can define if it is dynamic and can bust the placeholder and page cache.
>
> This is actually a very good example you bring up; what we are trying
> to develop is a sport aggregation website (and hope to re-use most of
> it across different sports) where we will host sports specific blogs,
> match results summaries, player statistics, videos, presentations of
> plays, coaching materials etc. The hope page will comprise of Slide
> Show of images from recent matches, player statistics (computed), team
> statistics (computed), Events calendar, Editor's Pick of - plays,
> blogs, matches etc.
>
> Now lets consider we updated a match result which would only impact
> the recent matches area, e.g., that placeholder would be 10~15% of
> pages data; now the plugin basically depends on a model 'Match' so
> when that cases we will look up for all instances of MatchPlugin,
> RecentMatchesPlugin etc. associated with the 'Match' updated and PURGE
> them from cache; now if we are to purge all the pages instead of the
> plugin instances effected only, that would cause much larger amount of
> data getting purged unnecessarily; similar would be the case for
> player statistics.
>
> So partial content purging would enable us (and the community) to
> benefit from purging only what changed and leaving the rest unchanged.

Ok i understand your problem space now way better.

>
>>
>> What kind of application/website are you guys planning?
>>
>
> As mentioned above we are looking to come with a set of websites for
> different sports community and generate content community specific for
> non-profit purpose. We also intend to extend it for some other social
> cause. In addition I come from a CMS background for news industry am
> trying to learn a new CMS to explore its strong points and to be
> honest Kudos to the team for such a wonderful product/project; I am
> really liking it. We are also evaluating it as a platform (among other
> CMSs) for mult-site hosting solution that is community driven.
>
> In fact describing the above scenario I am even more convinced
> Varnish/ESI and Django CMS would be a extremely benevolent combo. If
> you feel that the community will benefit in general I am more than
> inclined to chip in with the time for developing, testing and using it
> :).

I think greater support for varnish and ESI is surely benevolent to the community and if
you guys need any help let me know.

>
>>> <snip />
>
> Thank you,
>
> --
> Imran M Yousuf
> Twitter: @imyousuf - http://twitter.com/imyousuf
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>