Visual comparison:

    L[] C[] I[] B[]  # I is worst
    L<> C<> I<> B<>

So if [] is going to be used, may I suggest using lc letters with it
then?

    l[] c[] i[] b[]
    L[] C[] I[] B[]
    L<> C<> I<> B<>

Still not great, but better IMO. Why are <> bad, by the way? Can't we
just change the meaning to be qq-like, that is: with nested content?
That means only for non-unicode >><< you need extra angle brackets.

Or maybe we introduce [] as an alternative for <>.

Also, how is C[@*INC[-1]] parsed?  # I find this very hard to parse,
                                   # visually

Likewise, %?INC{something}?

Two possible sources of inspiration for the whole documentation thing:

* Text::MetaMarkup
  * Paragraphs CAN begin with a block level html tag, "h1: heading"
  * Inline HTML tags can be used as "{b:bold}"
  * Paragraph starting with * is a list
  * Paragraph starting with # is comment
  * Verbatim paragraphs simply start with "pre:"
  * No support for tables yet

* PodTables
  * See http://pugs.kwiki.org/?PodTables

Upon reading this, it is unclear to me whether you have read about the
Kwid format or you are simply guessing that Kwid is the same syntax
used by Kwiki.

It is not the same format at all. Kwid is merely /inspired/ by Kwiki,
which in turn is inspired by the (more usenix) features of modern wiki
languages. It is more fair to say that Kwid is much more inspired by Pod
than Kwiki.

Please read what is considered to be the de facto spec:
http://svn.openfoundry.org/pugs/ext/Pugs-Documentation/perlkwid.kwid
carefully and recomment.

A few notes.

To create Kwid I carefully studied the POD information model (the
semantic tree that POD parses to). Kwid uses the exact same info model.
This means that switching between the two without loss of information
is possible.

This makes the Kwid experiment much less risky, since it will be trivial to
convert in either direction.

As to the syntax, care has been taken to ensure that all the corner
cases are covered. And also covered elegantly.

Some people have argued that Kwid is only a syntactic change to Pod. I
would argue that they are correct. :) But this does not mean it is not
an important change. Kwid has an emphasis on minimizing the markup, and
using the markup one might use to discuss programming in everyday email.
This is hard to do in POD, but very easy to do in Kwid. Thus a bug win.

It is also wrong to imply that important semantic changes cannot be made
in the future. Things like introspection and transclusion. But that is
not the current concern.

In reality, Kwid does vary ever so slightly in semantics from POD. But only in
cases where POD seemed to have a wart. For instance Kwid allows named
hyperlinks: [The Pugs Source|http://svn.openfoundry.org/pugs].

Cheers, Brian

On 15/03/05 11:46 -0500, Aaron Sherman wrote:

     C<<  $x > $y  >>

Looks pretty clear to me.

        To create Kwid I carefully studied the POD information model
        (the semantic tree that POD parses to). Kwid uses the exact same
        info model. This means that switching between the two without
        loss of information is possible.

I noted that in my original message.

        This makes the Kwid experiment much less risky

Risk was not my concern. My concern was a Wiki-like model which is
inconsistent with many of the goals of POD. POD is intended to be PLAIN
OLD documentation. Kwid breaks this model of simplicity by introducing
unique boundary characters for many types of operations, and by making
the overall presentation more complex.

While I appreciate several features of Kwid, I feel that it should not
replace POD without first adopting a POD-like simplicity.

        Some people have argued that Kwid is only a syntactic change to
        Pod. I would argue that they are correct.

They are demonstrably wrong. You cannot Parse Kwid correctly by changing
the syntax of a POD parser.

For example, the behavior of

        * foo
        bar

Is totally dependent on what context it is enclosed in (.list or
top-level). This cannot be emulated with simple syntactic changes to a
POD parser.

Interestingly, this is one of the main benefits of Kwid, IMHO.

Re-read what I wrote and think about it. I think you'll find that it
avoids some of the major pitfalls of old-POD and incorporates all of the
useful features of Kwid while maintaining a simplicity that is as
elegant as POD's.

POD 1:1 mappings:

        POD             My Kwid Proposal
        x<...>            x[...] or x{...} (where x is C, B, I, X, L, etc)
        =item *         *
        =item foo       *= foo
        =item n         *1
        =for someformat =for someformat
        =over n         =begin list

The one obvious thing to POD users is the replacement of <> with [] or
{}. Why is this? Because < and > are used in un-balanced ways in a large
number of situations, so they should not be the primary bracketing
constructs. Also because the visual cue to many users of POD is that
it's SGML-like, and that way lies danger, since POD's <>-bracketed
constructs are not intended to be balanced.

Also, there should be a "=begin block" which in HTML would be
<blockquote>, and in POD would be "=over n".

Kwid 1:1 mappings:

        Kwid            My Kwid Proposal
        *               *
        .list           =begin list ... =end list
        .someformat     =begin someformat ... =end someformat
        .comment        =begin comment ... =end comment
        [generic]       [generic]

Extensions I proposed:

        =begin x,y,z            Section used only in the given formats
        =begin !x,y,z           Section NOT used in the given formats

As you can see, what I proposed IS a simple syntactic transformation of
Kwid.

        C[$x > $y] is about as B[easy] as it gets in [Perl]

vs:

        C<< $x > $y >> is about as B<easy> as it gets in L[Perl|perl]

without going full Wikish:

        [=$x > $y] is about as *easy* as it gets in [Perl]

However, saving a couple of keystrokes and cleaning up the above text is
inconsequential compared to the massive savings in terms of taking
advantage of the legions of people who are learning Wiki syntax these
days. Making POD *more* Wiki-like without sacrificing useful features of
POD is invaluable in terms of tech writers and other
non-Perl-programmers writing useful docs in POD!

POD's syntax is certainly good enough, but doesn't match the way that
people commonly write structured prose; especially in this era of text
based formatting such as in wikis. POD's syntax /elegance/ also tends to
break down into workarounds too fast in edge cases.

NOTE: POD does have some minor warts in the data model, but they can be
      fixed later on.

Pod uses this syntax B<bold stuff> which is fine, but it is a very
common idiom to use *bold stuff* outside of Pod. Usenet posts, email,
irc, etc. People use this, they grok it, and lots of tools grok it as
well. (I'm thinking irc clients as an example).

So how do we get *there* formally and avoid making a mess where * is
just supposed to be an asterisk?

Kwid does this by formally changing

   X<...>

into

   {X...X}

Where `X` is any Pod code like `B`, `I` or `C`. Since there are only 3
codes in common use (ignore `L` for a second), Kwid thus uses {*bold*}
{/italic/} and {`code`}.

This has a subtle but significant advantage over X<...>. The difference
is that instead of the ending marker being '>' it is 'X}', which is
orders of magnitude less likely to show up in the content being escaped.
This means you avoid the X<< ... >> mess almost entirely.

But the better part of this is that the Kwid forms can be relaxed to
drop the curlies in most cases. This technique uses the principle of
"hugging". So you can say *$a = $b * $c* and get the bolded equation
since the middle * isn't hugging anything. Hugging is a nontrivial
heuristic, but let's just say it Does The Right Thing. And if you aren't
sure just say {*$a = $b * $c*}. ie, just add some curlies to what you
already have.

*Bold* and /italics/ are rather obvious, intuitive and commonly used.
Backticks for `code` was chosen because backticks are rarely used in
code. Except of course when writing about Kwid itself. But to get
C<`code`> you just go to the curlies: {``code``}. Simple. Backticks also
seem to be right visually, but that's just my opinion.

For L<...>, I decided to use the very common wiki idiom of [...] for a
link. Everything in the `...` is the same as Pod.

I is purely subjective whether Kwid's overall presentation is more or
less complex than Pod's. Kwid attempts to elegantly move towards the
modern internet era of social software, with the hope that those
participating in those arenas might feel more at home.

ie

    * foo
    bar
    * baz
    boom

matches

    * foo bar
    * baz boom

matches

    .list
    * foo bar
    * baz boom
    .list.

The explicit `.list` is only needed when the parser cannot guess from
the context.

...

I would encourage those interested in further fleshing out Kwid to join
irc://irc.freenode.net/#kwid where all of this is actively being
discussed.

        `$x > $y` is about as *easy* as it gets in [Perl]

Did you really read `perlkwid.kwid`? There is simply no mention
of `[=...]` as a markup option, which makes me wonder where you
got it from?

How, then do you differentiate:

        * Bullet list.
        1. Numbered list.
        Other
                Term/definition lists

? In POD, that would be:

        =item *

        Bullet list

        =item 1

        Numbered list.

        =item Other

        Term/definition lists

Ok, that said PLEASE DO NOT USE UNBALANCED CHARACTERS TO DELIMIT!
Please, for the love of all that is valid input to any scanner / parser
anywhere, do not re-introduce quoting hell. Really. Please. Don't. I'll
buy you a beer. I swear, just put the unbalanced operator down and step
back.

Sorry, but I use POD specifically because it makes my life simple.
Introducing unbalanced quotes into it would remove that functionality. A
few examples:

        "And then I says, `Mabel,' I says, `shut up.'"

        The ``` character is no longer used.

And of course, TONS of Gnu documentation which uses the TeX-friendly:

        This is the way you ``quote'' things.

which means cutting-and-pasting such docs is now much more
labor-intensive.

    ext/Pugs-Documentation/perlkwid.kwid

    {/foo \/} bar/}
    {{/foo /} bar/}}
    {/foo { /} } bar/}

Those are 3 possible ways to make I<foo /} bar>. Note that '{ ' and ' }'
are the "asis" or "leave me alone" indicators. But the real point is
that '/}' is rather unlikely to ever show up in italics outside this
discussion.

The idea is to DWIM and there is a lot you can do with the `[...|.../...]`
syntax. Pod's strict syntax is:

    L<text|resource/"section">

/Text/ is obviously the text that should render. /Resource/ can be a
local manpage (ie another Pod document) and then /section/ is a section
in that doc. If /resource/ is empty, the current document is assumed.

/Resource/ can also be a fully qualified url and in that case section
does not apply.

For some reason Pod does not allow L<text|url> but there seems to be no
obvious reason. (This is the the only major thing where Kwid strays from
Pod's info model).

I am not certain what use case `L[...]` could get you that isn't already
covered by `[...]`.

   *   And perhaps most importantly, keep the items consistent: either
       use "=item *" for all of them, to produce bullets; or use
       "=item 1.", "=item 2.", etc., to produce numbered lists; or use
       "=item foo", "=item bar", etc. -- namely, things that look
       nothing like bullets or numbers.

In Kwid, therefore, this:

    * Bullet list.
    + Numbered list.
    - Other
    Term/definition lists

would produce 3 single item lists. You can obviously switch types in sublists:

    * Bullet list.
    ++ Numbered list.
    * another bullet
    -- Other
    Term/definition lists

    =kwid
    ...
    =cut

is ignored by the interpreter. This turns out to be the case with both
`perl` (Perl 5) and `pugs`. I am fine with some mailing list discussion
but I would rather spend the cycles on a reference implementation that
could be easily modified later.

    http://svn.openfoundry.org/pugs/ext/Pugs-Documentation/perlkwid.kwid            
As I said in my last mail, it has moved around a bit, so pardon our dust.

    This is the way you "quote" things.

Since `` and '' don't do anything for you in either Pod or Kwid.

C<< $x[0] > $y >>  # parser's ok (so's the human)
C<< $x[0] > $  >>  # oh, obviously $y was intended

Here are some of the formatting rules for TWiki (the Wiki version I
use):

1) Elements of a bulleted list must match /^ {3}\* /
2) Elements of a numbered list must match /^ {3}1 /
3) Headings must match /^----*\++/.  Number of +s determines level
4) *bold*
5) /italic/
6) =fixed font=
7) <verbatim> put text to be rendered as-is here </verbatim>

What is the organizing priciple?  What similarities do they have?
Quick, what level heading is this: +++++ ?  And this is just the
beginning...I didn't even get into the weird cases like ==bold fixed
font== and __bold italic__, which have no perceptible relation to
their component pieces (I would have expected */bold italics/*).  Yes,
it's powerful and it can do useful things, but as soon as I stray from
the most basic stuff I find myself going back to the docs to look up
how it's done.

Contrast this to POD (I'm not trying for point-to-point equivalence):

1) All formatting starts with = in the first column.  
2) Every POD command must have a blank line above and below it.
3) A list of any type starts with =over N and finishes with =back
4) List items are denoted with =item X where X is either * (bullets),
        an int (numbered), or word/phrase.  Use only one type per list.
5) Headings are denoted by =head1, =head2, etc
6) Formatting effects are done with X<text> where X is one of:
        B (bold), C (code), I (italics).  You may also use
        X<< text >> or X<<< text >>> if you have < or > in your text.
7) Text that is indented will be rendered as-is in fixed width font.

Aside from links, that's pretty much the entire perlpodtut boiled down
into 7 bullets; a little experimentation to get the hang of it and it
all holds together nicely, easy to remember.  

I freely admit that the link syntax in POD is difficult to manage and
not as powerful as it could be.  

--Dks

PS  I'm subscribed to the list so feel free to just reply there; I
don't need a personal copy as well.

I see no reason to I[[ $foo[3] ]] at all.

That would simply be I[$foo[3]] ... we are using a real parser here, no?
I can't imagine basing this on some pile of regexps, and we all have
"matched the balanced brackets" tools at our disposal, regardless of
what parser / parser-generator we're using these days.

Here's a simple Parse::RecDescent grammar for my proposal, so that we
can talk about it in more reasonable terms. Please note that I'm
TERRIBLE with P::RD, so I'm sure someone can figure out why I keep
ending up with the string "text_chunk" in my resulting syntax tree ;-)

See attached program and sample input. Just run parseajskwid.pl on
ajskwid.kwid.

  [Kwid] in your document.

This is a hint that you expect there to be a thing named "Kwid"
somewhere and you wish that somewhere to be applied thusly:

  L[Kwid|somewhere] in your document.

Where L[Kwid] would simply fail because it is as strict as POD, and it
won't find L[Kwid|perlkwid].

Other examples of this DWIMery:

  [http://www.perl.org/] => L[http://www.perl.org/|http://www.perl.org]
  [;-)] =>
        =for html <img src="winksmily.png" alt=";-)" />
        =for !html ;-)

I'm not thrilled with it (again, too many special characters that people
might have thought they could get away with using in their
documentation), but it's not too bad at all.

Pod is one of the things Perl 5 did almost exactly right.  It's
simple, intuitive, and stays out of your way.  It gives you most of
the formatting primitives you actually *need*, and nicely balances the
need for easy-to-remember and easy-to-type formatting codes with the
need to avoid using them on accident.  It's a very clean,
low-punctuation format, which makes it visually distinctive from the
surrounding code.

Specifically, I like the use of angle brackets in Pod.  Angle brackets
are simple, distinctive shapes; they remain wide in variable-width
fonts; they're associated with formatting codes in my
(HTML-influenced) mind.  The most common use of them in Perl 5--method
call/dereference--is going away in Perl 6, which makes them even more
usable.  (I never have a problem correctly marking up C<< $foo > $bar

In my mind at least, Pod has five goals:
1. Simple.
2. Adequate.
3. Easy to write.
4. Easy to convert.
5. Readable without a formatter.

#5 may be last on the list, but it's not least.

--
Brent 'Dax' Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker

Perl 6 also adds new uses of E<gt> and E<lt> for pipelining, and further
expands the usefulness of the => operator as a pair constructor. Rules
also add new uses of these characters, but those are balanced, so
improving POD with a real grammar specification would solve for that.

Much as it may be an EMOTIONAL sticking point, it's a very minor thing.
If we can agree on everything else, and I suspect we can, then let's
come back to that.

In the latter case, you would get balanced-[] matching, and given how
hard it is for PERL to do the right thing, there, I think it's fair to
fall back on "only perl can parse perl", and just do what the eye
suggests is correct. Remember that POD (and thus Kwid) are not intended
to be Perl-specific, just Perl-friendly. You can always:

        C{$x[0] > $]}

If you need unbalanced []s

That said, I'd be ok with something like Z[name|text] and Z{name|...}
for doing external grammars, but it'd still have to live inside one of
the balanced operators correctly.

Please also note that as a side-effect of using extract_bracketed in my
grammar,

        C[\]]

works. This introduces escaping which may or may not be good. It was not
a conscious decision, just a side-effect. I PREFER:

        C{]}

Which also works.

        C[ $x[0] > $y ] # Parser's ok
        C[ $x[0] > $ ] # Parser's ok... human might care

        \n\n=head1 Foo\n\n=head2 Introduction\n\nTopics:\n\n=over 5\n
        \n=item *\n\nFoo vs. Bar\n\n=item *\n\nC<\n>\n

AJS Kwid:

        \n= Foo\n\n== Introduction\n\nTopics:\n* Foo vs Bar\n* C[\n]\n

If you prefer a requirement for a leading blank line for readability, I
could see that, and we could talk about it. I do see the value in
"=begin list/=end list", but only in cases like this:

        =begin list
        * a
        * b
        =end list

        =begin list
        * Apples
        * Grapes
        =end list

Where you wish to make it clear that you are ending the first list.

Which do you find more readable:

        =over 5

        =item C<--help>

        Help is given

        =item C<--verbose>

        Verbosity is given

        =item C<--debug>

        Debugging is done

        =back

vs:

        =begin list

        *= C[--help]
        Help is given

        *= C[--verbose]
        Verbosity is given

        *= C[--debug]
        Debugging is done

        =end list

Replace *= with + and =begin list with .list and =end list with .list.
for the original Kwid proposal.

In ASCII, ' was meant as an apostrophe, but we use it as a quote.

Yen was never meant to have anything to do with zipping.

Guillemets originally had nothing to do with parallelization.

The hacks for square brackets are exactly the same. Think of how
C[@foo[0]] would be parsed without nesting. And if you say nesting fixes
all, then consider C["]"] as a counter-example. Are you willing to parse
code in a "simple" documentation format?

I think AJS Kwid is fine if it fits your brain. It doesn't fit mine in
much the same way that Kwid doesn't fit yours, in much the same way that
Pod doesn't quite fit either of ours.

The interesting thing to me is that all 3 syntaxes map over the same
data model and thus are easily interchangable. The other interesting
thing is that all three could be supported without affecting the Perl5
or Perl6 syntax proper.

Sam "mugwump" Vilain refers to each of these syntaxes as /Pod dialects/.
He is working on more formally defining the common model or "AST" that
these dialects map to.

Given that, I am going to continue working on the Kwid dialect and
developing the Kwid tools I have started. The ideas in AJS Kwid seem too
different to incorporate without muddying the Kwid vision. I encourage
you work on the AJS Kwid dialect too. I would guess that the Kwid tools
I'm working on, when completed, could be easily adapted to AJS Kwid.

Cheers, Brian

PodTables is a more detailed and more consistent approach to a
suggestion I did long time ago.

It also states what I found important when inventing the syntax: things
to consider when inventing your own syntax for the same feature.

YMMV. I'll have the second revision available sometime tonight, I think,
hopefully with an AJS Kwid to KwidData to POD translator (lossy, but
still mostly workable).

It is, however, contrary to the spirit of POD for you or me to continue
much further down this road (see below).

My effort here was to try to PREVENT the proliferation (e.g. by Kwid and
POD butting heads and ending up in a stalemate). The only problem is
that, presented with a compromise, the Kwid folks seem to be content to
ADD it to the list of variants rather than, in fact, compromise and
collapse the list.

I'll continue only as far as is needed to propose this in full as an
example parser / converter, and then I'm going to stop. My goal is not
to proliferate the number of markups further, and I'd MUCH rather see
Perl 6 rely on POD than fragment the SINGLE MOST IMPORTANT TASK in
creating code to share with the world: documentation.

If I'm left on a desert island with POD, then the only part I'll lament
is the desert island.

The problem I have with square brackets specifically is that they get
lost really easily, especially in variable-width fonts.  Gmail, for
example, displays e-mail in a sans-serif font, and virtually all such
fonts have narrow square brackets.  The square brackets in your
examples were visually lost in the surrounding text--without spaces,
square brackets are invisible.  That's okay when you're subscripting,
because your brain doesn't really need them to understand what's going
on, but it's not when you're applying and reading formatting codes.

Further, although Perl 6 is the time to make such a change, I'm not
convinced the change is really necessary.  We might be able to avoid a
few uses of C<< >>, but is that a big enough win to change *yet
another* aspect of Perl?  Especially an aspect programmers can--and
traditionally did--ignore?

By the way, I think I've seen a few people suggest some sort of
syntax-switching mechanism for "Pod6".  The day people have to think
about what dialect of Pod they're using is the day Pod dies as a
useful documentation language.

--
Brent 'Dax' Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker

I'm pointing out that the ROUTINE case is to need < and > in unbalanced
ways. Hence we have names for them that to not involve the words "open",
"close", "left" or "right". We could name "[", "fobit", but then people
would automatically call "]" a "right-fobit" or a "close-fobit". We look
at a piece of code or documentation that contains an un-balanced fobit
and we get a little chill because it's WRONG somehow. THAT is the kind
of balanced operator you want to tie your documentation to.

It was 23k vs. 44k.

I'm stopping once it works. I'm not looking to fork POD. If people don't
like the proposal, once they can play around with it, then I'll drop it.

Note that POD dialects, and differing POD conventions already exist in
Perl 5 and are in common use.  They were designed in the original POD
with the =for tag.  At the moment, tools like `pod2html' have to be
heavily aware of the POD dialect, which I think is sub-optimal when it
comes to some of the really interesting things people have achieved
with POD.  Look at MarkOv's OODoc, or Test::Inline, for instance.

All I'm trying to do is giving these beasts a name, and defining a
mechanism by which they can be used by tools that only know how to deal
with "standard" documents - thus giving users the freedom to define a
local convention if one of them doesn't quite fit their needs.

Using a local Subversion repository, and Request Tracker, and want to
be able to put hyperlinks in POD to refer to these entities?  No
problem, just extend the dialect and add a link style.  Then select
from a dozen output tools or variants to see which one works for you.