Zotonic architecture

[Jasper Timmer]

Hi all,

Is anyone so kind to explain a bit about the zotonic architecture to me, what proces is started for each user and what is always running? Arjan told a bit about why Erlang is great for webapplications at summerschool, but I forgot most of it already, but I know it included something about processes for each user and keeping connections open? I am especially curious about interaction between users, like the zchat module. It implements gen_server and is started when you activate the module, or for every user?

Thanks,

Jasper

[douglas]

Programming Erlang by Joe Armstrong

Erlang and OTP in Action by Martin Logan, Eric Merritt, Richard Carlsson

[Jasper Timmer]

Can I at least get a bit longer response where Erlang stops and Zotonic begins?

[Luke Burton]

On Jul 14, 2011, at 11:58 AM, Jasper Timmer wrote:

> Can I at least get a bit longer response where Erlang stops and Zotonic begins?

That's a bit like asking where Ruby stops and Rails begins :)

I'd recommend running Zotonic in the debug console and trying to poke at the gen_servers from there. That's what I've been doing to get my head around what is running and when.

[ll...@writersglen.com]

Hi Jasper,

This is an excellent question. I've been trying to figure it out myself.

As douglas mentioned, understanding OTP is a good start.

You can discern the architecture of Zotonic by looking at the Zotonic *.app files. When you really dig in, this is the way to go.

But, in my opinion, until you get serious about extending Zotonic, you really shouldn't have to do this. Unfortunately, however, Zotonic user docs don't really provide an adequate high-level conceptual picture of Zotonic.

It would be great if someone could come up with a block diagram. This could best be done by one of the primary Zotonic developers.

Arjan and I have been discussing just such issues.

Meanwhile, I've started a Cookbook item called justEnoughErlang to help non-Erlang programmers follow Erlang source code. I won't have time to finish and post for a few weeks yet.

As your understanding of Zotonic progresses, I would invite you to submit to the zotonic-users list Cookbook recipes to cover issues you've had to struggle with.

http://zotonic.com/documentation/875/writing-consistent-cookbook-items

If all users were to get into the habit of doing this, we would have excellent user docs in short order.

Please keep us posted.

All the best,

LRP

-----Original Message-----
From: "Jasper Timmer" <jjwt...@gmail.com>
Sent: Thursday, July 14, 2011 1:39pm
To: zotoni...@googlegroups.com
Subject: [Zotonic-Usr] Zotonic architecture

[Scott Finnie]

Hi jasper,

That's a good question, the answer to which i think would be very useful documentation for prospective users/developers. I'm certainly not the right person to answer in detail, being reasonably new to erlang and very new to zotonic.   However, as I'd like to see an answer too, here's a (very) basic start in the hope that it's easier for those more knowledgable to correct/extend than write from scratch.  (So to those people, please do...!).

First: structure. At first approximation there are six components in zotonic:
- erlang
- mochiweb
- webmachine
- postgresql
- erlydtl (more specifically a derivative thereof)
- zotonic itself

Much of the suitability for web server type applications arises from erlang's process model. Unlike more mainstream languages (c#, java, python, ...) concurrency is a fundamental,  built in part of erlang. Erlang supports this through _processes_. Note these aren't OS level processes (or threads) but provided by the erlang vm. Unlike OS processes/threads, erlang processes are very lightweight and extremely efficient. A single erlang virtual machine can support thousands of concurrent processes on very modest hardware. Furthermore, creating and tearing down a process is very fast and low cost.

In web server type applications therefore one (or more) processes can be spun up per client even in the face very high concurrent load. (I suspect this is what arjan was referring to). For more info, see joe armstrong's book as recommended elsewhere.  But in summary: erlang itself provides the underlying concurrency substrate.

Next up come mochiweb and webmachine. Together they provide HTTP handling. My knowledge is a bit hazy on the details (hopefully others will clarify) but essentially:

- mochiweb handles the bit-level processing, constructing and parsing the message stream as read from/written to the tcp port
- webmachine handles the rules of the http protocol. Basically, webmachine implements http as a big state machine, providing callbacks for handlers. There's a far better description on the webmachine website so take a look there.

In that context, zotonic provides a set of callbacks to webmachine. Those callbacks are responsible for retrieving content and rendering the response.

The content itself is stored in postgresql. Zotonic provides a data model (schema) for storing the content, as well as funtions for reading & writing it.

The rendering is handled using zotonic's derivative of erlydtl. Erlydtl is a web template language along the lines of asp/jsp etc. More specifically it's based on django templates (django is a python based framework).

In some ways you can see zotonic as the 'glue' that responds to webmachine http callbacks by selecting and rendering the appropriate content.

Zotonic provides other stuff too though. There's the admin interface, which makes setting up & administering a site much easier. It can also be extended through modules. In fact, most of zotonic itself is provided as a set of modules.

Stand out zotonic features are (1) speed, (2) scalability and (3) extensibility. (1) and (2) derive largely from erlang's concurrency characteristics. (3) is a property of zotonic's modular design, supported by webmachine's elegant http handling architecture.

And that's about where my knowledge runs out :-(

Hth,
Scott.

[Marc Worrell]

Hi Jasper,

Something short that might get you started and which we can cooperatively make into a cookbook entry... (Hi Lloyd! :))

BASIC ZOTONIC

Zotonic is a framework for building websites. It is also a CMS.
Zotonic uses its own webserver, so it doesn't need Apache, Nginx or other servers.

Inside Zotonic you can have multiple sites.
These sites are independent from each other and are managed by a process in Zotonic.
You can see a diagram of Zotonic below, every block is a process.
This is an old diagram, currently the hierarchy is different, but the idea is the same.

[ll...@writersglen.com]

Way to go, Marc!

I can see posting a mildly edited version of this in Cookbook. It would also be good if you could have an excellent graphic designer come up with a block diagram.

Beyond posting the diagram on the Zotonic site, you could have tee-shirts and posters printed up as a way to help support Zotonic development.

How's the T?

All the best,

LRP

Hi Jasper,

BASIC ZOTONIC

When a HTTP request comes in then we have a couple of processes to handle it:

1. HTTP request process, rendering templates etc.
2. Session process, also handles user data
3. Page process, handles push data to the page you are viewing in your browser

And then, when you are watching the page we have an additional process:
4. Comet or Websockets process which transports the data from the page process to the browser

A CHAT SETUP

The most simple chat server (with a single chatbox) will be something like all processes above and then:
5. A process that keeps the most recent history and all process ids of pages watching the chatbox

When (5) receives a new message, it will push the page update to all page processes (nr 3).
When then a transport arrives (number 4: comet pull or websockets) then the data is transported to the browser.

As all modules are gen_server processes, you will typically see that the module implements process (5).

WHAT IS COMMUNICATED BETWEEN SERVER AND BROWSER?

Typically the page (forms or buttons) posts so called 'postbacks' to the server.
These are transported by an AJAX call or via Websockets.
This postback is handled by a module (the delegate), in the event/2 function.
The return value of a function is always a context, which might contain new information for the page (updates, actions, etc.)
This information is rendered as javascript and send back to the browser, which evaluates the javascript.

SOME EXAMPLE CHAT MODULE DETAILS

So your chat pages should have:
- a div with all recent chat messages
- a form with an input field to add a message

The form is wired to a postback of the type 'submit'.
This is then typically send to the chat module's event/2 handler.
The event handler requests the process id of the chat server from the module manager using z_module_manager:whereis/2 and sends the incoming chat message to the module's gen_server proces.

This process could then:
- render some html/javascript using z_render:insert_bottom/3 and z_script:get_script
- send the javascript to all pages with z_session_page:add_script/2

HOW TO ADMINISTRATE ALL THOSE PIDS?

This is still a bit hard to do, we should make this simpler. Best is to check the zchat module and see how it can be done.
I think that the mod_signal from Maas might be a nice solution to this problem.

- Marc

[Jasper Timmer]

Thanks Marc and Sfinnie!

I'm trying to get the picture here. So a browser request is like this:

1) request from client to mochiweb

2) request is decoded and goes to webmachine

3) webmachine interprets request and does a callback to zotonic with url and form data etc

4) zotonic reads the dispatch file and sends a request for a template to erlydtl

4.b) zotonic does some authentication

5) erlydtl reads file from disk, and requests data from psql

6) resulting html bubbels back up to the client

This is my idea of what is happening. But this is too simple I guess. This is only for the HTTP request process? Where do the Session-, page- and websocketprocesses come in? And, what is already running when a request comes in, and what processes are started for the client specifically?

I am willing to write something for the Cookbook, or some kind of tutorial if I get the hang of Zotonic, no problem, but I am doing two things at the same time right now: reading and learing about both Erlang and Zotonic. And since I don't know what to do with my Erlang knowledge, I want to experiment with Zotonic, because that's something concrete and working on a CMS is a bit familiar for me (I know Django and CakePHP). And since Zotonic out of the box manages pages, I am looking at something different, like chat, and maybe extending? the user model with custom fields? Don't know how to do it yet, but I will find out.

Jasper