Parsing (well, lexing, really) wikipedia markup

[Sam Denton]

I'm wanting to parse some Wikipedia pages.

Wikipedia template data looks like this:  {{my template|arg one|arg two|keyword=value}}

In a template definition, you can use variable expansion, like this:  {{{1|default for arg one}}}

I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run.

My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in:  {{{{{keword}}}|arg one}}

Those five braces in a row are problematic.  My first thought is that I'd like to return two tokens when I see them, a two brace token followed by a three brace one, but I'm having problems figuring out how to do that.  My second thought is to define parser rules that start with a five-brace token, but that's not so easy, either.  Any suggestions on how to fix things?  Thanks.

[A.T.Hofkamp]

On 09/06/11 00:20, Sam Denton wrote:
> I'm wanting to parse some Wikipedia pages.
> Wikipedia template data looks like this: {{my template|arg one|arg two|keyword=value}}
> In a template definition, you can use variable expansion, like this: {{{1|default for arg one}}}
> I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run.
> My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in: {{{{{keword}}}|arg one}}

If this is the only way they can be nested, you can use scanner states, that is, define a scanner
state 'outside template', which matches {{ only. when encountering {{, switch to a 'inside template'
scanner state which matches {{{ only. When encountering }}, switch back to the 'outside template'
scanner state.

An alternative solution would be to use a scannerless parser. I am however not sure whether these
exist for Python.

Sincerely,
Albert

[Brian Clapper]

Also, have you investigated the tools listed on this page?

http://www.mediawiki.org/wiki/Alternative_parsers

There are several Python solutions listed.
--
-Brian

Brian Clapper, http://www.clapper.org/bmc/
Weiler's Law:
Nothing is impossible for the man who doesn't have to do it himself.

[Sam Denton]

On Thursday, June 9, 2011 6:38:44 AM UTC-5, Brian Clapper wrote:

[...] have you investigated the tools listed on this page?

http://www.mediawiki.org/wiki/Alternative_parsers

There are several Python solutions listed.

I had looked at those before starting this project, and at the time they seemed a bit overkill for what I needed.  Now, of course, they seem less so.  :)

Of the tools listed, mwlib is the only one that looks suitable for my needs.  All the other Python solutions are geared towards creating human readable output, whereas I'm wanting to populate a database.  I'm also looked for a pure Python solution, as I plan to deploy onto Google's app engine (where the database will be available under a Creative Commons license compatible with Wikipedia).

Thanks!

[Sam Denton]

One other idea that's occurred to me (I think I saw it somewhere in the PLY pages, but I can't find it now) is to nest lexical scanners.  I keep my current scanner and feed it into a filter that mutates the tokens as needed and emits the tokens I need.  In practice, the messy parts of Wikipedia's mark-up can be resolved by looking a few tokens ahead. So, when I find '{{{{{', I can pull tokens until I find either a '}}}' (which seems to consistently be the fourth token after that one) or a '}}' (which I haven't seen occurring, but better safe than sorry), and then emit either '{{{' and '{{' or '{{' and '{{{'.

Here's a simple proof-of-concept:

    class wrapper(object):

        def __init__(self, klass):

            self.klass = klass

            self.stack = []

        def lex(self, *argv, **kwds):

            self.lex = self.klass.lex(*argv, **kwds)

            return self

        def input(self, *argv, **kwds):

            return self.lex.input(*argv, **kwds)

        def token(self):

            if self.stack:

                return self.stack.pop()

            token = self.lex.token()

            if token is not None:

                if token.type == 'LBRACES5':

                    new_token = lex.LexToken()

                    new_token.type = 'LBRACES3'

                    new_token.value = '{{{'

                    new_token.lineno = token.lineno

                    new_token.lexpos = token.lexpos

                    self.stack.append(new_token)

                    token.type = 'LBRACES2'

                    token.value = '{{'

                    new_token.lexpos += 2

            return token

        def __iter__(self):

            return self

        def next(self):

            t = self.token()

            if t is None:

                raise StopIteration

            return t

        __next__ = next

lexer = wrapper(lex).lex()

[...]