Re: gae-cms.com - A brand new CMS based on Google App Engine via Python

[timh]

Is there a live demo somewhere ?

On Wednesday, June 13, 2012 12:06:59 AM UTC+8, GAE-CMS wrote:

I am the creator of a brand new CMS called GAE-CMS.  It is based on Google App Engine written in Python 2.7.  It is free and open source (GPL).  It has been designed from the ground up for speed and usability.  As well it is very easy for designers to modify every aspect of the theme and apply different themes to different sections of the website.  It is developed in a very modular way with different types of content able to inherit from the base content class and define how it behaves and its views.  I believe it is a fresh new take on CMSs and believe it could surpass some of the current great CMSs if enough developers get behind it.  This is an open call to developers to check out our project, install it, use it and if you believe in it then contribute back and join our leadership team to help guide the vision of this project.  Visit http://www.gae-cms.com for more information and access to the code.

[GAE-CMS]

Hi Tim, I'm afraid the only demo right now is the one actually running the website http://www.gae-cms.com.  You may have been tricked into thinking that it's actually Google Code but that is the theme I cloned for the website to make it a more seamless experience.  You can replace this theme however, and try out different themes using the form on the top right of the site.  To try out the administrative features including creating pages, adding content, creating themes, etc you will have to check it out from source, change the appspot id in the app.yaml file to one that you manage, and deploy it.

[Jay]

When I try to run locally I get

path not accessible: './themes/.svn/templates'

[GAE-CMS]

Hi Jay, thanks for responding and attempting to install GAE-CMS. As far as I know, the latest App Engine SDK ignores .svn and other "files in an RCS revision control directory" during deployment by default.  I have developed and tested only on the last two versions of App Engine and have not faced this problem before so can you try with the latest version?  If you are in fact using the latest version, please respond and I'll try to get to the bottom of it.  Thank you.

[GAE-CMS]

Neat, if you want to submit a fix, I can give you developer permissions. Email me privately. Yes I am for now but am open for developers and leaders as I mentioned.

[Bill Graham]

I am very interested in this project.  What functionality have you included thus far.  Do you have roles/permissions and Access Control management?  What about multi-level menu management?  What about threaded discussion comments?  Is there the concept for users to be able to extend the framework via module plug-ins, etc.?

I may be able to help on some types of functionality because I have architected  similar functionality in the past.

I am looking forward to some additional information.

[Bill Graham]

I downloaded the project, but I must say it is a challenge to follow.  Most frameworks and CMSs fail to attract a sufficient following to survive, not because of missing functionality, but because of lack of clear and easy to follow documentation.

If you would like, I could take a try at creating a start to documentation, but I need to know a little more about the functionality provided and the architecture.  Anything you could provide that describes the features and structure of GAE-CMS would be a start and I could ask questions as needed and then document what I understood.  You could then review and let me know where changes are needed.

If it is along the lines of an existing CMS, let me know which one and I could build a documentation skeleton based on that CMS.

Bill

[GAE-CMS]

Hi Bill, thank you so much for your interest.  The CMS is roughly based on ExponentCMS however with a lot of modifications based on my experience of it and trying to design to eliminate its weaknesses.  You are right about documentation being key and I would love to welcome you to the project.  I am giving you immediate access to the Google Code project on account of your enthusiasm, feel free to fill out the wiki in terms of documentation.  I will briefly give you an outline of the code and feel free to ask for information where it is not clear.

Firstly have you managed to successfully deploy the system? I have received some comments that people were not able to, but nobody followed up with any information regarding their environment so I want to iron that out in case I am the only one able to actually run it even though I have it up and running on http://www.gae-cms.com (via an AppSpot account) without problem so I'm not sure.  Someone alluded to the existence of .svn directories being the problem but I have that in my local environment with no problem.

The place to start as with any App Engine project is the app.yaml file.  Simply put, it defines that compressor.py handle requests to CSS and JS files while router.py handles all other requests.

Next let's look at router.py.  The main idea is that it passes on whatever request it gets in the path on to the section.py module.  The section module will handle the request made and return a string (actually unicode) of whatever content was requested within a section which is then returned to the client.

Before we examine the section.py module, which is quite critical, let's look at the two directories at the root of the project i.e. framework and themes.  The themes directory is where any expert on ExponentCMS will recognise the influence.  Any subfolder within it is taken as a theme and can be applied to a Section.  Each theme can have multiple templates which are defined in the themes/_/templates directory.  A template is a Django template which should define all of the content within the <body></body> tags.  It has access to special variables that are defined in the section object that it has access to, and also can call certain functions defined in /framework/templatetags/filters.py.  The only actual requirement for a template is that it *must* call {{ main|safe }} which will define just the main content of the current section.  However it can define HTML tags and put Container modules within them which then can house other modules, or it can also directly define a module without a Container such as a Text module for the footer.  I use the term module here but actually the term we should use is Content as module is a Python reserved word.  It can define multiple types of Content that sit within the template.  An example of the footer content could be:

<div class="ft">
    {{ section|view:"GLOBAL, footer, Text, default"|safe }}
</div>

Here we are passing the view filter function the section object as well as saying that we want content of type GLOBAL with the identifier "footer" of type Text and view type default.  In short, it means that the footer Text Content will be global in scope meaning it is the same content across multiple sections, as opposed to LOCAL to a section.  This Text Content has default view (there can be multiple views and we can even define the same Content having different views applied to it).

Also another key is that a template can include CSS and JS files included in its theme directory.  For example:

{{ section|yuicss:"cssfonts/cssfonts.css,cssreset/cssreset.css,cssbase/cssbase.css" }}
{{ section|themecss:"base.css" }}

{% if section.mobile_ua %}
    {{ section|themecss:"mobile.css" }}
    {{ section|viewport_content:"width=240, initial-scale=1.0" }}
{% else %}
    {{ section|themecss:"layout.css" }}
    {{ section|yuicss:"cssgrids/cssgrids.css" }}
{% endif %}

Here we are stating that we want to include CSS files from the YUI framework cssfonts.css, cssreset.css and cssbase.css.  YUI is the CSS framework I have gone with although it is in early stages of this project but I just found that it was full featured enough and written in a way that complements this CMS (we could always additionally define and include jquery for example as a local css file).  YUI however is used by most Content views and is the default CSS framwork.  Going back to our snippet, we can also notice from our local theme directories CSS files, we are including base.css.  Additionally we are saying if it is a mobile client, to include the local mobile.css file, otherwise the layout.css and cssgrids.css from YUI (essentially used to have multiple columns).

I think I will end my topic of themes here however I will mention that in addition to having themes stored locally, a user can actually create and manage dynamic themes (meaning those not stored on the local file system) following the same syntax I've described via the administration.

The next topic is to discuss Sections, "modules" (Content) and everything stored within the framework directory.  I will continue that topic in my next post as this is a lot to type without any feedback so please feel free.

Regards,
Imran

[Bill Graham]

Hi Imran,

 

Thanks for the intial data on GAE-CMS. I will take a look at ExponentCMS for more info.  (This may provide a quick-start for the documentation for the various audiences.)  Perhaps you could tell me what the primary differences are between GAE-CMS and ExponentCMS (perhaps some functionality you have not yet included and what the plans are re. those functionalities.

 

Of the open source CMSs, I am most familiar with Drupal, which I tend to look at from a architectural perspective.  Their "node" concept is very powerful as it allows common functionality to be easily applied to any content type (entity type).  This works great for threaded discussion comments, for capturing the created by and dates, for managing entity status and workflow, etc.

 

I think what would help me most right now is to understand the framework part.  How are you handling the roles and permissions and how do you designate what objects/actions a permission pertains to?

 

It would also be good to understand how user-created modules are included.

 

Finally, what do you see as keying in keeping GAE-CMS beiing a high performance?  I agree with you that this is crucial.

 

Looking forward to more information.

[GAE-CMS]

Given that ExponentCMS is written in PHP and GAE-CMS is written in Python, and that I have tried to maintain a very Pythonic codebase, I would say that there are lots of differences.  As I mentioned it is very roughly based, for example you will recognise the directory structure and as mentioned, the theming is very similar, Exponent also makes use of YUI in a similar way.

I have yet to develop the permissions functionality however I have designed it so that it can be added in later with ease.  Currently permissions are either all or nothing with the AppSpot admins given all permissions and everyone else has nothing (except View permission on public sections).  However individual Content types are designed such that they must make manifest all actions that they can perform.  This will later tie into the permissions system so that each action for each Content type on each section requires a user with permissions to perform.  So for example if you had a Listing Content with actions edit_item, delete_item, create_item and view_item, you would require explicit permission to perform each action except view item which would fall back on your View permission for the section.  Again with sections you would have two permissions, View and Manage.  Manage would automatically give you permission over all actions for all Content within the section..  Also I should mention that there will be individual permissions as well as group permissions.  This is exactly how ExponentCMS deals with permissions

Currently user-created modules must be copied into the /framework/content directory and deployed onto the server.

I'm not quite sure I understand the last sentence but I think it's asking regarding how performance will be kept at a high performance.  I've used memcache at the Content level to make sure database queries are minimized.  Similarly resources such as CSS, JS, files, etc are cached and retrieving them is relatively inexpensive.  I have made every effort to minimize the cost of the system.

Okay let's look at the /framework directory.  It has in it an assets directory.  This is used throughout the system including for individual Content types to store CSS and JS files.  The only exception is framework/assets which also has a statically defined images directory containing the sprite image that is used for site management images.  I had skipped over this point in the first post but you can see it defined in the app.yaml file.

The /framework/content directory defines all of the Content types.  Currently you can see these types include Configuration, Container, Files, Navigation, Text, Themes and User.  I think it is obvious what content they each define by the name, but I will mention that they all are subclasses of the main Content class.  You can define your own Content type subclass such as Forum for example and it shouldn't be hard to implement if you look at how the others are done.  The simplest one is the Text content which is the most basic type of Content that allows you to use a RTE to enter some static text/html snippet on the page and this is just 100 lines of Python code plus the JS, CSS.

The /framework/subsystems directory defines various subsystems are used throughout.  These include cache, configuration, file, forms, permission, section, template, theme and utils.  The most important one to know is the section subsystem.  This one enables viewing a section, creating sections and subsections, reordering, and a bunch of other section related things.  It works in conjunction with the Navigation Content which is used to display various types of navigation such as dropdown, top level, children, full hierarchy, etc.  It also is used to display the current section in its basic form or replace the main container (remember {{ main | safe }} with the content of any action that is carried out on a type of content contained in a section.  If you understand the section module, you have understood a large part of this CMS.

Finally the templatetags subsystem houses filters.py which define a bunch of functions that are available at the template level.

Okay this is a brief overview.  I hope it can help you as you're browsing the code.  Please ask for help when you need and feel free to contribute directly to the project page.

Thanks,
Imran

[Bill Graham]

Hi Imran,

GAE-CMS runs fine for me locally (Windows7).

It was somewhat difficult to get started (e.g. I would not have thought that Section has anything to do with Menus).  

I think the documentation has to start with presenting the structure of a website according to GAE-CMS (what are the elements of a website and how do they relate to one another).

Next there should be a number of how tos:

1. how to structure of the site:
1. how to add or manage menu items
2. how to add pages/sections to pages
3. how to change the header (site title, slogan, logo, etc.)
   
4. how to change the footer
2. How to add/edit content
1. what types of content are supported and how to add/edit the content
3. How to change the appearance
1. How to do theming, etc.
2. How to work with CSS
4. How to add functionality
1. already available functionality/modules
2. adding jQuery functionality and plug-ins
3. develop custom modules

I think all of the above should describe what one can currently do.  If you could provide me a brief list of what you can currently do with the site, I will try to structure and expand it

In addition, I would strongly recommend:

1. providing a few sample sites as a showcase to give people an idea of the kinds of things they can do (at least some screen shots)
   
2. providing some more variation of themes (blog, club or church community, small business, etc.).  

Unfortunately many people have little imagination and unless you show them an example that is close to what they want to they cannot see how they would create what they want to have.

I also recommend quickly creating a place for designer/developer support to encourage designer/developers to create solutions for their customers around GAE-CMS.  (Having a showcase of example sites - even if just front pages, would greatly help recruiting developers and designers to the CMS).

Regarding permissions; I spent a number of years architecting an application framework and ERP system built upon it.  This including designing user, session and security management.  My number one thing was high performance.  Our solution was built on a relational database, but I have been giving thought as to how to accomplish the same in the noSQL world.  In general, I have found the permissions functionality in CMS systems to be lacking the needed flexibility.  I say this because most people are not really looking for a CMS system but rather an application framework with CMS capabilities.  This gets one into situations with complicated workflows where different users need to be able to see different views of the same content types or the same user might be able to see different content depending on the content of particular fields.  Please let me know what you were thinking of as a permission concept and I will be able to respond with some ideas you might want to consider.

Bill

[Jakob Holmelund]

I see lots of optimizations for the ndb-stuff in the project, its quite easy, and i would love to help..

[GAE-CMS]

You are right.  I didn't know about it when I started the project but it should be easy to replace the existing datastore code.  I would say there are some things that are done a little inefficiently right now.  Such as when a section is deleted, the way it determines whether content exists on a page or not and then if it is unique content without a view on another page it deletes it, otherwise not.  This could probably be done more efficiently so if you are changing the database code, it might be worth also looking at that.  Also in a lot of places, arrays of keys are stored and this could probably also be made more efficient using StructuredProperties, etc.  I am giving you access to modify, let me know privately if there is another email address you prefer.

Thanks for your help,
Imran

[GAE-CMS]

Hi Bill, have you taken a look at the theme switching functionality available on the main website http://www.gae-cms.com (the Preview Themes dropdown)?  I was hoping that I could use that area as a showcase for themes that are available.  Admittedly there are only two themes currently available.  I want people to be able to preview starter themes there and hopefully purchase one to get their own site going.  So my vision is that it will be a very extensive set of themes.  Of course the entire system is open source and GPL so there is nothing stopping anyone from creating and providing themes freely.

I think your ideas around permissions are the same as what I've envisioned, and I've kept the system flexible enough to incorporate it.  My idea is that Content developers should have access to the permissions API where they can say if a permission exists for the current user, show this or do this, etc.  They do not have to maintain permissions themselves, this will be taken care of by the system so long as they make the Content's actions manifest, the permissions subsystem will let the admin manage the role assignment based on individual/group for every Content type.  This is exactly how ExponentCMS does it.

As far as what it can currently do, you can:

- create pages (public and private)

- manage pages (edit meta data, apply specific themes to them, reorder them, delete)

- add content to pages either directly via the theme or via Containers defined in the theme.  A footer text for example would be a direct reference to a Text Content whereas a column in which you wish the admin to be able to define the content, you would use a Container.

- Content can take on one of possibly multiple views and if you give two pieces of Content of the same type the same namespace, they will give you two possibly different views of the same Content.  This can be on different pages or the same page.  Global Content will also have the same content across the entire site.

- add files to any section as well as delete them

- In the site configuration you can set the robots.txt and favicon.ico for the site as well as a bunch of other global settings

- And finally you can create or upload entire themes through the system

Imran

[ID]

I'm trying it out, seems quite interesting and clean. I have a question, how can I resize or get rid of the empty space on left side?

On Wednesday, June 13, 2012 12:06:59 AM UTC+8, GAE-CMS wrote:

[GAE-CMS]

Hi ID, thanks for trying it out.  The empty column on the left side of the default theme is for the secondary level navigation.  If your page has only one level of navigation, or else you want to use the drop down navigation instead for the top navigation, you can create your own theme which has only one column.  Have a look at this file:

https://code.google.com/p/gae-cms/source/browse/trunk/themes/Google%20Code/templates/Default.body

Replacing the following code in it:

<div class="bd yui3-g">

<div class="sd1 yui3-u-1-6">

{{ section|view:"GLOBAL, navigation, Navigation, expanding_hierarchy, 1, vertical"|safe }}

{{ section|view:"LOCAL, side1, Container, default"|safe }}

</div>

<div class="mn yui3-u-5-6">

{{ main|safe }}

</div>

</div>

with just:

<div class="bd">

{{ main|safe }}

</div>

ought to do the trick.  Also you can get a dropdown navigation by replacing the following code:

{{ section|view:"GLOBAL, navigation, Navigation, nth_level_only, 0, horizontal"|safe }}

with:

{{ section|view:"GLOBAL, navigation, Navigation, dropdown, 0, horizontal"|safe }}

Regards,

Imran

[ID]

Thanks Imran, i will try your suggestions, instead of removing, i will probably just downsize the left column to 100px. gaecms seem real nice and clean, i love the multiple content sections on a single page, makes editing large pages easy.  

will you add a function that makes comments possible at the bottom of a page?

[ID]

couple of things that i would like to report :)

1) about 10 hours after uploading gaecms to my appengine ac, the appengine free quota for "frontend instance hours" was filled and site was no longer accessable rest of that day, my site gets about 100 hits a day and was up for nearly a year with no such quota problem.

this is not a blame, i just report what happened. :)

2) when a "is private" page tried to accessed, site shows confusing errors, it would be better to redirect "access denied" errors to a user-specified page.

thanks for great GAE-CMS!

[ID]

Imran, GAE-CMS consumes the appengine's free quota for "frontend instance hours" very quickly!! perhaps need some caching/optimizing?

it turned out to be a big problem for me! google shut my site 2nd time for being over quota.

[GAE-CMS]

Dear ID, sorry for the troubles.  Can you confirm which Frontend Instance Class you are using?  I have not faced this problem with similar usage so I'm not sure why.  I do make use of lots of caching but maybe you can take a look with a fresh set of eyes?  I was just about to announce the release of the latest beta version which uses ndb instead of db so this might help you out as it has 2 additional layers of caching before the database call.  Please have a look.  Also if you can point me to your site I can try setting up a similar one and see where the problem might be?

Thanks,
Imran

[ID]

Hi Imran, this is my site http://gokturkanitlari.appspot.com/ 

I used GAE-CMS for few days only, during those days, every day my site was shut down for half day for being "over quota"  in appsot.com it shows "over quota" for "frontend instance hours" my site maybe got 200-300 page views daily then, that should not cause over quota.

i had to remove gae-cms after using just few days. i now use static files instead, have no problems since, even the page views increased to 400-500 daily.

i'm not sure what was the problem, i had not modified gae-cms, i just downloaded it and installed as it is.. i hope you can fix it.  it's great product, elegant, clean, easy, functional.. 

please let me know if you fix it, i very much like to use it.

thanks

[GAE-CMS]

Hi ID,

Thank you for your feedback and your positive comments about gae-cms.  Any CMS would fail compared to static HTML files being served in terms of frontend instance hours or any other quota for that matter.  The issue is whether the ease-of use and features that the CMS brings makes it worth it as a website creator rather than to write HTML from scratch.  If you have 10 pages it might be worth it to do in static HTML but if you have 100 it is rather cumbersome to maintain that many static files.  This is where a CMS comes in which will necessarily use up more of your quota as it has to make memcache lookups, make calls to the database and perform CPU intensive tasks that do not exist when a server is simply returning static files.

Having said that it is my goal to make gae-cms as cheap as possible and I don't think my own website gae-cms.com has gone down yet on account of using up the free account quota that it currently has even with 100s of hits.  It would probably be more useful to compare multiple CMS' to determine which is the cheapest rather than comparing it to static pages.  But I think your criticism is still somewhat valid and I will try to profile it and clear up any bottlenecks that might be unreasonably using up the free quota.

Thank you,
Imran

[ID]

hi imran, i'm not comparing gaecms to static files. i used gaewiki too with with same site, no quota problems, i found out that a wiki was not suitable for such site.

i thought gaecms was great, and i just want to make you aware of an invisible problem that i encountered, i reorted because i cared of it, else i wouldn't have bothered to sit and write a coule of lengthy messages.

have a good day.

[GAE-CMS]

Thank you ID for the feedback.

I have updated the code to improve on two things: speed and cost (by reducing frontend instance hours).  I accomplished this by introducing caching at the instance level.  This has the effect of avoiding a ton of calls to memcache which was the bottleneck that I discovered using Appstat.  The caveat is if you have more than one instance running, writes to the website will cause the output to be inconsistent across multiple instances.

I don't believe you can have multiple instances running on GAE's free quota anyway but if you are paying for more than one instance and you care about immediate global consistency, the behaviour can be switched off by setting INSTANCE_CACHING_ENABLED = False in settings.py before deployment.

My frontend instance usage per page is improved by over three times meaning that you can serve over three times as much traffic as previously before running out of quota.  I have yet to package it into a release but you can check it out from the repository for the new changes.  I hope you give it a try again ID and apologies for any misunderstandings.

Best regards,

Imran