Wiki markup

[Richardp]

Hi,

I've just had a quick look at atomicwiki and although the system looks
quite interesting I was surprised that the wiki markup is stored
verbatim within the XML files.

If it were possible to parse the wiki markup into XML, then this would
allow much more powerful use of embedded XQuery scripts against the
wiki database.

I realise this is probably a design decision that had some important
thinking behind it and I wondered if those reasons could be outlined a
little?

Best regards,
Richard

[Wolfgang]

Hi,

> I've just had a quick look at atomicwiki and although the system looks
> quite interesting I was surprised that the wiki markup is stored
> verbatim within the XML files.

Yes, I already thought about changing that. The only problem is that we
would need to transform the XML back into wiki markup for editing.
AtomicWiki allows arbitrary extensions to be written as XQuery
functions, which generate XHTML. We would need a way to reverse this,
i.e. pass in the XHTML and have get the wiki markup back again. This is
probably not a big issue though. Maybe we could use a naming convention
for the XQuery functions that implement an extension...

Wolfgang

[Richardp]

Hi Wolfgang,

I was thinking about this because I believe using the type="text"
construction in the atom feed is not appropriate when wiki markup is
included in the text. The relevant part of RFC 4287 says the text must
be human readable, but what we have is really markup using the wiki
conventions.

<http://tools.ietf.org/html/rfc4287#section-3.1.1.1>

So, to be properly compliant with the RFC I believe that wiki markup
in a type="text" element is not allowed. I'm imagining what happens
when a feed reader looks at the feed ... it wont be able to handle
wiki markup (clearly) but it would have half a chance with XHTML.

I think converting from a subset of XHTML to wikimodel markup is
pretty easily accomplished using an XSL stylesheet. For the extensions
I guess the best way to go is use your own elements in a separate
namespace.

A quick google search finds me a stylesheet that is based on the
mediawiki syntax. This could be a useful starting point:

<http://mozile.mozdev.org/0.8/demos/html2wiki.xsl>

I'm a little surprised that such a conversion is not part of the
wikimodel framework ... it would provide part of a good round-trip
test suite?

Best regards,
Richard

On Oct 8, 4:32 pm, Wolfgang <wolfg...@exist-db.org> wrote:
> Hi,

[Wolfgang]

Hi Richard,

> So, to be properly compliant with the RFC I believe that wiki markup
> in a type="text" element is not allowed.

Probably yes.

> I think converting from a subset of XHTML to wikimodel markup is
> pretty easily accomplished using an XSL stylesheet. For the extensions
> I guess the best way to go is use your own elements in a separate
> namespace.

Yes, that's a good solution. We could easily store the extensions as
elements in e.g. the "wiki" namespace and expand them later during the
final rendering of the content. Writing a stylesheet to convert the
XHTML back into wiki text will be easy then.

I guess I'll find some time for AtomicWiki next week and will implement
the ideas we discussed. It shouldn't be too much work and I desperately
need a search feature anyway.

Wolfgang

[Fabini]

Sounds great. And will you include a conversion tool for existing
AtomicWiki content? I know it's cheeky to ask, but I've already got
quite a bit stored on your execllent wiki. In effect it would be a one-
off run of XHTMLListener wouldn't it?

Fabini.

[Wolfgang]

> Sounds great. And will you include a conversion tool for existing
> AtomicWiki content?

Sure, I will need to convert my existing wiki contents as well.

Wolfgang

[Wolfgang]

>> I guess I'll find some time for AtomicWiki next week and will implement
>> the ideas we discussed. It shouldn't be too much work and I desperately
>> need a search feature anyway.

I committed a first bunch of changes today: all wiki markup is now
transformed into XHTML before being stored. For editing, the entries are
changed back into wiki markup (using XHTML2CommonWiki.xsl).

Macros and extensions are saved as elements in the wiki namespace. For
example, the {code} macro is saved as

<wiki:macro xmlns:wiki="http://exist-db.org/xquery/wiki"
name="code">...</wiki:macro>

For display, the XHTML is sent through the XQuery function
atom:filter-entry() to dynamically expand those macros.

To migrate old wiki contents to the new format, you just need to call

build.sh migrate

once after you restored your old stuff (see INSTALL file). So far, I
only tested the standalone version, not the .war distribution.

Next step for me is to implement a general search facility, which should
now be rather easy since everything is stored as proper XHTML.

Wolfgang

[Fabini]

Thanks, Wolfgang.

It built up fine and is working succesfully both in Jetty and within
Tomcat. All my existing blog and wiki entries seem to have migrated
successfully in both systems (although I executed your migrate.xql
XQuery directly from oXygen rather than using Ant).

The only problem I had to fix was as a result of my own mistake. A
couple of macros I had written returned XHTML which worked OK under
the previous version, but in the new version all the elements were
returned with an atom prefix. My own fault for not specifying the
XHTML namespace on some of the elements I was sending back. Easy to
fix and works fine now.

Thank you again for your time in creating this improvement.

Fabini

[Wolfgang]

Since all wiki entries are now stored as XHTML, adding a general search
facility for AtomicWiki has become easy:

http://atomic.exist-db.org/util/Search

The search functionality is implemented by the XQuery script
data/search.xql and relies on Lucene for the full text queries.

To use this with your existing setup and data, a few steps need to be
applied. I guess the easiest approach is to

* backup your data using eXist's backup facility
* shut down and clean your db by removing everything from
webapp/WEB-INF/data
* run "build.sh setup" to restore the default wiki setup
* start the server and selectively restore data from your backup, again
using eXist's backup tools. Do not change /db/system/config or the
/db/wiki/util feed (which contains the search forms)!
* run webapp/migrate.xql or "build.sh migrate" once to transform your
existing data into XHTML

Wolfgang

[Fabini]

Thanks Wolfgang - the new typeface does look better. And I was just
about to start this morning puzzling over what had happened to my
summaries when your posting here announced it was fixed.

AtomicWiki built and loaded fine, following your instructions.

Just one tiny bug, easily overcome: in the migrate.xql XQuery, the
closing tag on the summary update is set to </content> rather than </
summary>

Thanks again for your effort.

Fabini

[Fabini]

I've now installed the Tomcat version of AtomicWiki, and all works
well apart from the "Quick Search" on the right-hand panel which links
to

http://<host>:<port>/util/search

rather than

http://<host>:<port>/<context-name-for-AtomicWiki>/util/search

which results in "page not found" within Tomcat.

Accessing the two search entries in /db/wiki/util/ works fine.


Fabini

[Richardp]

Hi Wolfgang,

I've re-started with a completely clean installation, and I'm getting
an error message.

Firstly the ft namespace is undeclared in search.xql; that was easily
fixed. However I then get a different error message:

"Call to undeclared function: ft:score [at line 299, column 36]"
whenever I do a search.

I've updated eXist to the latest svn code (as well as AtomicWiki) and
recompiled but this problem persists. Any suggestions?

Cheers,
Richard

[Wolfgang Meier]

Hi Robert,

> "Call to undeclared function: ft:score [at line 299, column 36]"
> whenever I do a search.

I forgot to say that you need to enable the lucene module in eXist
before you build it. Just edit extensions/indexes/build.properties:

# Lucene integration
include.index.lucene = true

and call build.sh/build.bat. After the build, you should find
exist-lucene-module.jar in $EXIST_HOME/lib/extensions from where the
AtomicWiki build script should pick it up.

Wolfgang

[Richardp]

Hi Wolfgang,

OK, that works fine now. It also solves the undeclared namespace
problem ... although the namespace you've used for ft is different to
the W3C one.

I didn't know that there was a lucene module ... is there any
documentation for it?

Cheers,
Richard

[Wolfgang Meier]

> I didn't know that there was a lucene module ... is there any
> documentation for it?

No, it is not properly documented yet, but I'm currently writing a
short introduction. I will paste the link here once it has been
published.

Wolfgang

[Wolfgang]

> No, it is not properly documented yet, but I'm currently writing a
> short introduction. I will paste the link here once it has been
> published.

http://atomic.exist-db.org/blogs/eXist/LuceneIndexModule

Wolfgang