The Magnode content management system (and "Why Node.js?!")

[Austin William Wright]

Every so often (maybe twice a week in IRC) someone asks about a blog, e-commerce, or content management system for Node.js. I feel compelled to ask "Why Node.js", because it seems like one of those "I want to do it because I can" rationales. Most people have that exact reason, and not much else (that's not non-generic). But that's not to say no such reason exists.

I had this idea of mine long before Node.js, so I have what I believe are more practical reasons. A CMS is I/O heavy, not especially computationally heavy. ECMAScript is a natural choice because it's prototype-based, event-oriented, and of course, the language of the Web. A little while after having been introduced to Node.js, I realized what Node.js was perfect for, and started work somewhere around v0.4.

Node.js (or any single-process paradigm) brings some design challenges. For instance, in my application, I can't make the assumption that all legitimate requests will only use a certain amount of memory — some applications may require streaming an indefinite amount of data, for instance. However, handling multiple requests per process means the total resource usage of a single request can't be limited, at least by the OS. The only solution I know of so far is, for every attacker, to make sure that the non-generic attack of growing the process's memory to the point of crashing is more expensive than a simple, generic DOS.

I also don't want to make any assumptions how the application might be used, I can't assume that only Jade or MongoDB will be used, for instance (even if they are natively supported).

Here's my implementation: Magnode.org. It's designed as a framework, with a simple default application that tries to cover most use cases. It uses many other libraries, particularly ones I had to write, like rdf, jsonschema, and contenttype.

It works on the notion that a user requests a resource, the resource is identified, then formatted into the requested Content-Type (via the Accept header or the request URL), usually HTML or JSON, but depending on the resource, you might also format e.g. a time series as an image, or a PDF file. It's especially designed for graphs of data, like the Web (and dereferencing nodes on it, hence the name Magnode).

It's pretty capable right now: You start it up, run through a quick web-based installer to initialize the configuration, then you can log in and see a front page of blog posts (reverse chronological order), create new posts, pages, add items to menus, create new menus, create users, and such. With some coding, it should be relatively simple to create a new type of output, maybe you want to generate an image from data for instance, this should be possible with a fairly short file, maybe 50 lines (it involves simply registering a function with a domain of whatever data type you want to process, and a range of Document and image/png, or whatever media types the output would be).

While presently it only works with MongoDB, it's not designed for a specific database (only about a dozen of 88 files are MongoDB-specific), and I intend to support a number of data storage schemes. The next storage scheme will be HTTP endpoints (effectively making Magnode an HTML frontend to an HTTP API). Additionally I would like to see relational databases, an IndexedDB-style database (where you query documents using specific indexes instead of by their data), and an RDF graph database.

Here are some features that work right now:

• Make lists and tables of content. The front page, by default, is a list of blog posts.
  
• Revisioning of content - documents are only inserted into the database by default.
• Custom schemas and content types, and custom templates for formatting them.
• Resources are first-class: Most everything is an HTTP resource, including blog posts, users, the content-types themselves, and even server configuration if so desired. This means a single code base for doing everything related to changing configuration and editing content.
• Pluggable user authentication and authorization.
• Content-Type negotiation can return any number of content types with a Content-Location header, XHTML and JSON are supported natively. It also supports most of the HTTP headers like Etag, Accept, If-Match, and similar. This means the server is naively a RESTful HTTP API.

Some use cases that I've designed for, but have yet to implement, are:

• A better UI for editing structured data and markup: Right now, many operations like adjusting schema definitions requires editing raw JSON.
  
• Extensive caching of rendered resources. (Nonetheless, it still performs faster and serves more requests than a comparable installation of Drupal.)
  
• Straightforward editing of templates and themes, including compression and aggregation of CSS and JS files.
  
• Self-registration and support for third party accounts (log in with OpenID, etc).
  
• Content-Type aware formatting of error pages.
  
• User comments on resources
  
• OAuth (or similar) for delegating access of resources to third parties (bearer tokens with sandbox permissions are already supported or easily implemented).
  
• Editing and updating content in-line, and in real time (perhaps integrate the MediaWiki VisualEditor). Magnode can update resources with a standard HTML POST form, or a PUT request with a JSON document, but maybe also PUT raw HTML of the edited page, parsing the HTML back into a database record. This would mean editing pages in-line with very little client-side scripting required, and no significant requirement on the server either.
  
• Support more data sources as mentioned: relational (SQL), graph (RDF), and resource (HTTP) databases.
  

One cool use case that I've already implemented is using Magnode as a gateway that adds an HTML representation of resources to an otherwise JSON HTTP API. With this, you can use your web browser as a console to interact with the API endpoint.

I'd really appreciate any help that people can offer. If you have MongoDB running, take five minutes and go to the "Get Started" page, then let me know what you think. The source is available through the website or on GitHub. It is public domain.

Austin Wright.

[Duncan Wilkie]

Sounds interesting, I'd be interested in it supporting CouchDB and elasticsearch as a search engine. Started working on an idea that takes PDF docs extracts out the text using PDFBox (java cmd tool) and indexing it in elasticsearch.

- mrdnk

—
Sent from Mailbox for iPhone

On Wed, May 22, 2013 at 4:22 AM, jerome <jeromec...@gmail.com> wrote:

Awesome Austin. I've got some WordPress experience and I love that your theme is twentyonetwelve. I've been thinking about a NodeJS blog implementation and while I fought agains the "why Node?" question for a while I finally decided it was worth exploring. You've done a lot of the work already. I'm going to fork this maybe I can help.

Jerome

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Austin William Wright]

I'm looking closely at fulltext search options right now: One of my tech demos is a search engine for the more complex roleplaying games, at http://d20.magnode.org/ I have much of the D&D 3.5 SRD imported (because that's published with a very liberal license, such that I could even sell it if I wanted to). I am looking mostly at ElasticSearch, that looks to be very much what I need, but I haven't actually tried it out yet because the whole, well, Java thing.

Normally you'd work with formats in their greatest common denominator (things with at least application-level semantics like database records or JSON documents, this is the so-called principle of least power), and format it up to things like PDF documents (I'd like to be able to export scientific papers, for instance). But going the other way is something I'm working on too. Maybe you have a PNG image on the filesystem and you want to serve it as a JPEG, or you have a PDF and want to index it, annotate it with metadata, extract metadata, or convert it to more browser-friendly formats (use pdf.js or convert to an image). It should all be possible. For now I'm working on implementing PUT etc. for editing static files.

Austin.