Eventframe's future

[Kiran Jonnalagadda]

Hi all,

I've been thinking of what to do with Eventframe, which has been in stasis for a few months, and whether we should adopt something like Docpad as a foundation. Here is the conundrum:

Websites are collaborative effort. Pretty much every website we build has two or more people working on it. But how do the collaborators collaborate?

Let's look at what typically gets used in our workflow:

1. Dropbox (and equivalent tools like Box, Google Drive, Skydrive). Dropbox is a great place to share files and get everyone an updated copy instantly. Dropbox works well for code repositories too -- I use it to work on the same code across multiple machines.

But Dropbox has the granularity of a single file. Two people cannot work on the same file at the same time or they will clobber each other's changes. This is typically not a problem with source code when people are handling different files at one time, but it's a disaster for something like a spreadsheet which is one large file containing everything. This is why we use Google Docs.

2. Google Docs. Great for documents and spreadsheets that multiple people can edit at once, but it can't handle all kinds of files. We can't store HTML templates or source code in it. Etherpad is similar and open source, allowing for a custom installation, but only supports editing text.

3. Git (and GitHub). Great workflow for ensuring the integrity of your commit, but terrible for showing work-in-progress or collaborating on an edit. Two people can't edit a document together. If you want feedback from someone, you have to commit and push. Git's learning curve is steep. This is true for other version control systems too -- Mercurial, Bazaar, and even the simpler Subversion.

As you can see, every one of them works for some situations but not for the others. With Eventframe, I attempted to use two different systems, picking up the strengths of each:

1. All HTML and CSS is stored in a Git repo and committed to GitHub.

2. All content is stored in Eventframe's database, which supports multiple node types and isn't limited to a choice of two (like Google Docs with documents and spreadsheets). Content documents are versioned so multiple people can edit them.

The problem with this approach is that #1 and #2 clash. The templates in #1 need to know about the content in #2 to include it in the page, but the content in #2 needs to specify a template from #1 to render with (this is the "template" field when you edit a node).

Because #1 and #2 are inter-dependent, you have to update both at the same time for your copy to work. But #1 is updated by pulling from a Git repo, while #2 is updated by sharing JSON files, which are currently emailed between the collaborators or exported from a central installation.

A good tech architecture is layered, with each layer being aware of the layer that's below it, but having no expectations of the layer above it.

Your computer hardware shouldn't care what operating system you use, while your OS can be particular about what hardware it has drivers for. OSes in turn shouldn't make assumptions about what applications you want to use, while apps can be very OS-dependent.

Similarly in this case, we ideally have a three-layer architecture: Eventframe itself at the foundation, templates next, and content on top.

I've been trying to get templates and content unentangled, so that templates can be edited with no dependence on content, making this similar to how Wordpress themes work -- the person who made the theme does not know anything about your blog or what you will write in it, but they can make and distribute a theme anyway. However, as it turns out, our development cycle requires a close connection between templates and content. We have so far been unable to develop templates ahead of content, and our design partners have even insisted that their layer is on top, asking for content to be produced first.

So I've been considering two compromises, both of which sound wrong to me:

1. Templates move into the database and are managed alongside content in Eventframe, or

2. Content moves into the filesystem and is managed with Git or a traditional workflow (the Docpad approach).

The #2 approach here has become very popular of late thanks to static site generators and the like, but I find it discomforting. Imagine if you had a blog where, to make a new post, instead of logging into the website, you first had to install a lot of complicated software, write a file, save it to the correct place, run some generator, run a server to confirm you did it correctly, commit everything to Git, then SSH to the server and checkout?

Sounds complicated? Now imagine if this is how it worked when a user interacted with the website, to do something like posting a comment or RSVPing for an event, if behind the scenes all this had to be written to the filesystem. Filesystems are great, but databases exist for a good reason.

I don't have a good answer for how to take this forward, which is why I'd like your inputs on this.

Kiran

-- 

Kiran Jonnalagadda

+91-99452-35123

http://hasgeek.com

[Kiran Jonnalagadda]

Adding a few things we have discussed elsewhere:

1. Eventframe does not replace our existing projects like Hacknight, et al. Rather, Eventframe is the only time we've tried to establish an API pattern for content objects -- where all our previous projects have had them scattered around models, forms and views. I think this pattern formation is good because it'll lead to the development of an API that we can write tests against. I want to expand on these ideas and establish a CRUD API across all our code.

2. The main problem with Eventframe right now is that two interdependent layers, templates and content, are managed in different versioning systems: templates in Git and content in Eventframe's internal versioning system. This means there is no single authoritative source for the latest version of any website. This is the problem that needs solving: either both layers are managed in the same versioning system, or they stop being interdependent.

Kiran

-- 

Kiran Jonnalagadda

+91-99452-35123

http://hasgeek.com

[Anand Chitipothu]

For Open Library, we started with keeping templates in the database,
but that made it difficult to get latest templates on dev-instances.
Having a good export/import system will solve that issue.

Anand

> --
> You received this message because you are subscribed to the Google Groups
> "HasGeek Code" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hasgeek-code...@googlegroups.com.
> To post to this group, send email to hasgee...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Anand
http://anandology.com/

[Kiran Jonnalagadda]

On Thursday, 28 February 2013 at 12:18 PM, Anand Chitipothu wrote:

For Open Library, we started with keeping templates in the database,

but that made it difficult to get latest templates on dev-instances.

Having a good export/import system will solve that issue.

Templates can contain logic and therefore have access to user data. Eventframe uses sandboxed templates (no access to attributes starting with _ and other blacklisted attributes), but I'm still uncomfortable having templates editable via the front-end because of the level of access they have.

Proper sync requires a journal that records everything, including deletes. Without that, deleted nodes will come back on the next sync.

When we start doing things like this, we are basically creating a new version control system, which is when we have to ask: why not use an existing version control system instead of building a new half-baked version?

My only counterpoint to this is that all traditional version control systems are too heavy for the typical use case for a web app -- things like posting comments, voting and RSVPing for an event. There is no good reason for each of these actions to be version controlled.

Kiran

[Praseetha KR]

Its really a difficult problem.

If we move content to github, obviously users will find difficult it for content management; If we move theme to database, developer will have problems. So, I think, an in-between approach as it exist now is better, though the unified version controlling fails here.

A solution-but not an efficient one- can be as follows:
Developer create themes in github, once its completed it can be stored in the database of eventframe under the directory 'themes'. So that users can select the best theme for their site. It its so,

1. The themes should be with placeholders for fragments/map which users can understand and edit easily.
2. If user add a new feature like fragment/map in addition to the default theme features, there should be an option to position it in the webpage.
3. There can be an option for 'sync with git' to catch the updates.
   
4. If user want to customize the code, we can give block editing areas (inside the jinja blocks) to ensure they touch only html/css.
   
5. We can also have options for adding enhancements like carousel, collapsible panels, tabs, etc. with the help of bootstrap.
6. Some version control system (like wiki) may have to be used with content.
   

--

[Prateek Rungta]

Something else that might help:

Think of the Read API separate from the CUD API. Templates consume the Read API. There’s no reason to carry over their burden when designing the data modification workflows.

Take a look at the design behind NPR’s (much lauded) content strategy: http://blog.programmableweb.com/2009/10/13/cope-create-once-publish-everywhere/

– Prateek

[Mitesh Ashar]

Am new on the group. Not exposed to and not sure about a few things in this email. So, was wondering if I can ask questions here! :)

-- 

Mitesh Ashar

Sent with Sparrow

On Thursday, 28 February 2013 at 12:18 PM, Anand Chitipothu wrote:

[Kiran Jonnalagadda]

On Thursday, 28 February 2013 at 12:23 PM, Mitesh Ashar wrote:

Am new on the group. Not exposed to and not sure about a few things in this email. So, was wondering if I can ask questions here! :)

Sure you can. Apologies for the delay in approving the posts -- I didn't realize they were held for moderation.

Kiran

[Mitesh Ashar]

Thanks Kiran.

Have been more consumed with reading up now though. Will ask out whenever I need to! :)

-- 

Mitesh Ashar

Sent with Sparrow

--

[Kiran Jonnalagadda]

On Thursday, 28 February 2013 at 1:38 PM, Prateek Rungta wrote:

Think of the Read API separate from the CUD API. Templates consume the Read API. There’s no reason to carry over their burden when designing the data modification workflows.

Unfortunately, templates need the CUD API for things like posting comments, submitting forms, etc.

Here is what happened the first time I hit a need for CUD in a template:

https://github.com/hasgeek/fifthelephant/blob/eventframe/templates/event.html

Template in action here: http://fifthelephant.in/2012/workshop-excel

You can see the template is almost entirely logic. I was in a hurry and didn't care about doing it cleanly. Logic of this sort should move into the app and be exposed as an API to the template, and that's one of my goals for a rewrite.

Kiran

[Kiran Jonnalagadda]

On Friday, 1 March 2013 at 4:17 PM, Praseetha KR wrote:

If we move content to github, obviously users will find difficult it for content management; If we move theme to database, developer will have problems. So, I think, an in-between approach as it exist now is better, though the unified version controlling fails here.

One way to tackle this is with the concept of node attachments. I filed a ticket sometime ago describing it: https://github.com/hasgeek/eventframe/issues/15

Praseetha, you are the only person on this list who has enough working experience with Eventframe to understand the issues involved. Look at the ticket and let me know if it makes sense.

I trust the attachment approach will solve nearly all the problems you've brought up. The final one:

1. Some version control system (like wiki) may have to be used with content.
   

This is the most interesting question. How much of a CMS should Eventframe be, and why does Eventframe need to bother with versioned content at all?

As you know, Eventframe is a barebones CMS, having just enough UI to get some content in. It needs a lot of work to be a decent CMS. The main reason I'm still interested in this is because there are many usecases where a CMS is required:

Case 1: sponsor wants to update the logo they use on the website, or the text attached to it. In the current workflow, they email us text, then we put it up and email them back. They may or may not be happy with how it turns out on the site, but they don't bug us about it because it's such a hassle emailing back and forth.

New workflow: sponsor logs into Eventframe, edits the text, previews the result, and submits for approval. We look at the new version to ensure it's not running afoul of any of our rules and approve it.

Case 2: workshop facilitator receives a question about the workshop. The question is posted as a comment. They can respond directly or update the text and publish the new version directly from the website. No need to email us with corrections.

Case 3: session proposals. Same deal -- push changes to funnel and have them show up on this site, or edit directly here.

There are many reasons to allow someone to edit content directly instead of an email or git-fork driven workflow.

Kiran

[Mitesh Ashar]

The selected answer for this question could be useful: http://stackoverflow.com/questions/270658/cms-versioning-strategies-for-content

And I am sure we can learn something from the django packages here meant for versioning model content: https://www.djangopackages.com/grids/g/versioning/

I feel it would be even better if you are able to identify possibly portable segments from some of the versioning systems above, by decoupling the logic from the djago integration.

-- 

Mitesh Ashar

Sent with Sparrow

--

[Kiran Jonnalagadda]

So Drupal has one table for all kinds of nodes? Where are the type-specific properties stored? In node_revision.body?

I considered diff storage initially, but it's not practical because two or more versions may be "live" at any point -- a "published" version shown to the public, a "draft" version shown to the owner, and a "pending" version shown to moderators. We'll need to store full-text for all and purge old revisions periodically.

Kiran

-- 

Kiran Jonnalagadda

+91-99452-35123

http://hasgeek.com

[Mitesh Ashar]

-- 

Mitesh Ashar

Sent with Sparrow

On Saturday, 2 March 2013 at 4:51 PM, Kiran Jonnalagadda wrote:

So Drupal has one table for all kinds of nodes? Where are the type-specific properties stored? In node_revision.body?

I didn't really go through the drupal answer then and I was not referring to it.

I checked now, but I have no clue about drupal as I have never tried it. Hence, if I try to make sense of it, I will only be making assumptions. Rather, if there is someone here who's worked on drupal, please bring this into context for us! :)

 

I considered diff storage initially, but it's not practical because two or more versions may be "live" at any point -- a "published" version shown to the public, a "draft" version shown to the owner, and a "pending" version shown to moderators. We'll need to store full-text for all and purge old revisions periodically.

Consider, temporally the referred versions will always be in the given order:

published —> pending —> draft

which is easy to version using diffs.

But yes of course, this temporality may definitely not be present in all type of data.

-----------------

I was referring to the answer quoted from the book Managing design data. I am just presuming it may be useful. Since, you understand the current problem best, you can judge on reading, whether you can take something meaningful from it.

[Kiran Jonnalagadda]

On Thursday, 28 February 2013 at 11:47 AM, Kiran Jonnalagadda wrote:

Adding a few things we have discussed elsewhere:

1. Eventframe does not replace our existing projects like Hacknight, et al. Rather, Eventframe is the only time we've tried to establish an API pattern for content objects -- where all our previous projects have had them scattered around models, forms and views. I think this pattern formation is good because it'll lead to the development of an API that we can write tests against. I want to expand on these ideas and establish a CRUD API across all our code.

I've started a new project to work on this pattern: https://github.com/hasgeek/nodular

I'm aiming for 100% test and documentation coverage from the start and it's looking good so far. Tests are run against PostgreSQL to replicate the production environment.

This code should be reasonably complete within a week to start rewriting Eventframe using it.

Kiran