Inlines

[Eric Moritz]

Mike Newman has been working on the inlines blueprint and he has the
rendering code pretty much finished and ready to merge.

We should have had a discussion about the specification for the
inlines system. I think rendering is only a part of the needs of the
inline system. I'm sorry I should of had at least a basic spec
defined before Newman started working on it.

I've defined what I think we need for the inlines system here:

http://code.google.com/p/django-pennypress/wiki/InlineRender

Let's talk about how we should modify or enhance the feature set of
the inlines system.

Thanks,
Eric.

[Michael]

I don't think we should be so strict in our processing of our XML in stories. Wellformedness is obviously important, but if we are going to force ourselves into a strict schema right off the bat, I see us having problems down the line. 

First off, I hate the idea of having 'align' being a special attribute. If align is special, shouldn't 'style,' 'class,' etc. also be privileged to grace our 'inline' tag? Each of those values should be passed to the function that is rendering out the XHTML and therefore shouldn't be treated any different then any other argument. So based on this schema, I would remove the 'align' attr and force people to use it as another 'param.'

It would be nice to see inlines to work whether the markup is XML, ReST or other markup languages by just adding a new parser module, this would be really easy to do. The inline render functions should be simple functions that accept as many or as few arguments as possible. These functions should not matter what markup is being used and could technically be used outside of the inline system. For example, we have a render_html method on a few of our models that could easily be used in these conversions. 

With this as an idea, each param node is going to be broken down into a dictionary with keys being the 'name' and the values being 'values.' Wouldn't it be simpler to to just provide those same arguments as attributes on the nodes. There is a view on the inlines that does checking for validity for the inlines, that would be even more strict than dtd schema checking. If we really want dtd checking, we can easily add a dynamic dtd based on introspection of the render functions. (This could also be explicit if we want, but that would require more code when defining a inline). 

This also allows us to avoid parsing even more nodes and node trees for each inline. I could foresee some inlines which might have like 10 params. This also seems like one heck of a lot of markup to stick to current XML schemas. We also have the ability to pass integers in XML attributes as well; with the strict param all of the values would be forced to be strings due the the CDATA nature of them. 

Also throwing out there that not only do we have to go XML --> XHTML, but we will also be going [JavaScript|Python] object --> XML. I assume that none of us are masochists and actually be using DOM methods to create and render ALL of the XML that will be placed into the story text, and the params sound like they will make life much harder. (For those of you saying that you will use DOM methods exclusively, remember Element.cloneNode and Element.createElement are some of the most intensive operations you can make in the DOM and adding a 'param' element for each argument is just as brutal.)

Overall the XML schema is something that is arbitrary, we can make it this strict if we want to, but to me something this strict will limit the extensibility of the inlines and also slow down the actual rendering without giving us anything but the appearance of sticking to already arbitrary XML guidelines. We can provide better XML schema checking if we really want to, using these methods and right now it checks for wellformedness and also allows each inline to validate the arguments that it is given. I believe this will make the most sense to editors who might be messing around in the raw XML and also I think that for WSIWYG editors the speed gains of not having to parse the extra nodes will make editing inline faster and nimble.

That said, we can simply provide another parser that will parse out the attributes with the params. In fact we could do this with the code that we have now just boiling down to the extra param level and actually check the dtd directly. I don't believe this to be necessary, we shouldn't handcuff ourselves for XML.

[Glenn Franxman]

I think Eric's schema using name/value pairs makes a decent compromise
allowing for the use of xml validators and tidy scripts while still
being very open.

[Glenn Franxman]

An alternative might be embedded yaml? I guess I'd like to see
concrete examples of alternate markup schemes before making a
descision.

[Eric Moritz]

I think you misunderstood my use of the DTD. I just used a DTD to
describe the tag and which tags and args were optional and required.
It seemed like a standard way to describe the tag without making up my
own way of describing the tag. I wasn't implying that we use strict
XML validation using a DTD.

As for validation, I think that all we need is some form validation in
the custom form field. A possible implementation is like this, the
form when doing clean will identify all the inlines in the text and
call an is_valid method on the inline class and pass the params as
kwargs or a dict to the is_valid method. The is_valid method will
raises an "InlineInvalid" error, the clean method will then raise a
ValidationError with the line number, char position and description of
the fault. This validation can be independent of the markup we're
using. For instance there could be a field called, XMLInlineField,
ReSTInlineField, etc and their clean methods will know how to find the
inline classes.

Re: align attribute, Ryan Berg agrees with you, that might be
something we will want to visit when the form field widget is
developed.

I agree that with you that the render method should just spit out
text, not a XML DOM or something.

On Feb 17, 3:58 pm, Michael <newmani...@gmail.com> wrote:

[Eric Moritz]

I think sticking with XML for version 1.0 is fine, diverting our
efforts for alternative markups maybe be an unnecessary diversion when
we have a lot to write.

On Feb 17, 4:19 pm, Glenn Franxman <gfranx...@gmail.com> wrote:
> An alternative might be embedded yaml?  I guess I'd like to see
> concrete examples of alternate markup schemes before making a
> descision.
>

> On Tue, Feb 17, 2009 at 4:17 PM, Glenn Franxman <gfranx...@gmail.com> wrote:
> > I think Eric's schema using name/value pairs makes a decent compromise
> > allowing for the use of xml validators and tidy scripts while still
> > being very open.
>

> > On Tue, Feb 17, 2009 at 3:58 PM, Michael <newmani...@gmail.com> wrote:

[Eric Moritz]

Also, using a strict schema for the story xml allows us to keep
reporters from messing up the HTML code. Trust me it happens. If we
want a site to have resemble well formed HTML, this is probably the
best way to go. the params gives us flexibility from having to
redefine the schema every time we create a new inline.

[Glenn Franxman]

I wonder if we can overload xhtml and still get the richness everyone wants?

Or rather just pattern ourselves after xhtml implimenting a fairly
strict subset. I think the less markup required, the better; and
the more it looks like standards that the end users are familiar with,
the better.

I want simple markup, simple validation with good error messages and
simple transforms to N end formats.

[Michael]

Are we really saying that:

<inline type="image">

    <param name='id' value='13' />

    <param name='align' value='center' />

</inline>

is better than:

<inline type='image' id='13' align='center' />

?

Chatting with Eric, I can see how validation of a schema is important to make sure that items are in order and that short of a dynamic schema, this would be difficult with the attributes of an inline being anything. I will spend some time and write up a renderer that can parse the params element like is laid out on the wiki. I feel like we will see a performance hit, but we can compare them side by side. 

What this has brought up is the need for validation of a schema. I will develop a formfield here that can be used by apps that want to use inlines that will also provide a schema checking. Obviously, my currently implementation would require a element to accept any attribute, which to the best of my knowledge, isn't a schema I know. So that might immediately rule out my idea. I am open for schema validation methods that people have used and are comfortable with. RelaxNG is apparently a moving target. I would avoid DTD checking like the plague. 

I will go back into my coding corner and pound out the extra render method and the form field with schema validation and come back where we can talk about this a little more concretely. 

[Glenn Franxman]

I don't think we're saying that. I think we're saying the first is
easier to validate. In my mind, the second is clearly better if
you're entering it or validating by hand.

[Glenn Franxman]

Maybe we can just let the inlines handle their own validation and
write our own validator that passes tags for validation just like for
rendering.

[Michael]

On Tue, Feb 17, 2009 at 5:06 PM, Glenn Franxman <gfra...@gmail.com> wrote:

Maybe we can just let the inlines handle their own validation and
write our own validator that passes tags for validation just like for
rendering.

That's the idea, where the text is passed to the validator and then the inlines are rendered after the fact. This part is all in the inline code. The schema validation will just add an extra level to that. Right now the renderer, validates that the XML is wellformed and that  the 'story' is the root node. 

Inlines are given the option to validate their own code at validation time so that we can check to make sure there is a object with that ID in the database for example. This should be completely separate from the schema validation. 

The formfield should be a part of the inlines app because otherwise the story text will just be raw XHTML and we won't need to worry about any of this stuff. Separation between schema validation and inline validation will be very separate however. 

[Glenn Franxman]

Sorry. I'm in favor of that.

I think that the inlines should probably validate their format and
their content in the validation phase.

Also, we should do away with the root node. It adds no value. We
can wrap the content ourselves before passing it into validation, etc.

[Michael]

On Tue, Feb 17, 2009 at 5:32 PM, Glenn Franxman <gfra...@gmail.com> wrote:

Sorry.  I'm in favor of that.

I think that the inlines should probably validate their format and
their content in the validation phase.

Also, we should do away with the root node.  It adds no value.   We
can wrap the content ourselves before passing it into validation, etc.

This sounds fine to me and easy to fix.  

[Eric Moritz]

Here's a proposal that could remove the necessity of <param> tags:

1. Validate the Inlines.
2. Render the content as XHTML and validate against the XHTML derived
schema

It makes validation a more heavier operation but it makes sure the
result that will show up on the page is valid.

Another solution for schema validation with inlines is:

1. Validate Inlines
2. Assert that the inlines are children of the root element
3. Remove the inline tags from the content
4. Validate against the XHTML schema.

I like solution 2 better than the solution 1, ideas? alternatives?

Eric.

On Feb 17, 5:39 pm, Michael <newmani...@gmail.com> wrote:

[Michael]

Since we don't have any schema validation mechanism yet, there is a third option, of just writing our own schema validator. Might be too much trouble?

[Eric Moritz]

I guess we could, but why recreate the wheel?

On Feb 17, 5:54 pm, Michael <newmani...@gmail.com> wrote:
> Since we don't have any schema validation mechanism yet, there is a third
> option, of just writing our own schema validator. Might be too much trouble?
>

[Glenn Franxman]

I think this is a good fix for sax.

The sax processor understands the core schema ( p, br, ... ) but any
time it hits an inline tag, it just starts accumulating args, cdata,
etc until it hits the /inline at which point it passes controll to the
inline's validator. Not actually too different than, say, django
template parsing.

[Michael]

On Wed, Feb 18, 2009 at 10:58 AM, Glenn Franxman <gfra...@gmail.com> wrote:

I think this is a good fix for sax.

The sax processor understands the core schema ( p, br, ... )  but any
time it hits an inline tag, it just starts accumulating args, cdata,
etc until it hits the /inline at which point it passes controll to the
inline's validator.     Not actually too different than, say, django
template parsing.

I think I agree, I started down SAX a bit last night. I just had a project dumped on me last night so I won't be able to get back to this until the weekend, but I think that using a combination might be the way to go where inline tag would be the exception. Once we have a schema validator we can decide the syntax with the code in front of us.