Parsing (well, lexing, really) wikipedia markup

85 views
Skip to first unread message

Sam Denton

unread,
Jun 8, 2011, 6:20:16 PM6/8/11
to ply-...@googlegroups.com
I'm wanting to parse some Wikipedia pages.
Wikipedia template data looks like this:  {{my template|arg one|arg two|keyword=value}}
In a template definition, you can use variable expansion, like this:  {{{1|default for arg one}}}
I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run.
My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in:  {{{{{keword}}}|arg one}}
Those five braces in a row are problematic.  My first thought is that I'd like to return two tokens when I see them, a two brace token followed by a three brace one, but I'm having problems figuring out how to do that.  My second thought is to define parser rules that start with a five-brace token, but that's not so easy, either.  Any suggestions on how to fix things?  Thanks.

A.T.Hofkamp

unread,
Jun 9, 2011, 3:58:22 AM6/9/11
to ply-...@googlegroups.com
On 09/06/11 00:20, Sam Denton wrote:
> I'm wanting to parse some Wikipedia pages.
> Wikipedia template data looks like this: {{my template|arg one|arg two|keyword=value}}
> In a template definition, you can use variable expansion, like this: {{{1|default for arg one}}}
> I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run.
> My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in: {{{{{keword}}}|arg one}}

If this is the only way they can be nested, you can use scanner states, that is, define a scanner
state 'outside template', which matches {{ only. when encountering {{, switch to a 'inside template'
scanner state which matches {{{ only. When encountering }}, switch back to the 'outside template'
scanner state.

An alternative solution would be to use a scannerless parser. I am however not sure whether these
exist for Python.

Sincerely,
Albert

Brian Clapper

unread,
Jun 9, 2011, 7:38:44 AM6/9/11
to ply-...@googlegroups.com

Also, have you investigated the tools listed on this page?

http://www.mediawiki.org/wiki/Alternative_parsers

There are several Python solutions listed.
--
-Brian

Brian Clapper, http://www.clapper.org/bmc/
Weiler's Law:
Nothing is impossible for the man who doesn't have to do it himself.

Sam Denton

unread,
Jun 10, 2011, 1:12:04 PM6/10/11
to ply-...@googlegroups.com
On Thursday, June 9, 2011 6:38:44 AM UTC-5, Brian Clapper wrote:

[...] have you investigated the tools listed on this page?

http://www.mediawiki.org/wiki/Alternative_parsers

There are several Python solutions listed.

I had looked at those before starting this project, and at the time they seemed a bit overkill for what I needed.  Now, of course, they seem less so.  :)

Of the tools listed, mwlib is the only one that looks suitable for my needs.  All the other Python solutions are geared towards creating human readable output, whereas I'm wanting to populate a database.  I'm also looked for a pure Python solution, as I plan to deploy onto Google's app engine (where the database will be available under a Creative Commons license compatible with Wikipedia).

Thanks!

Sam Denton

unread,
Jun 10, 2011, 2:15:56 PM6/10/11
to ply-...@googlegroups.com
One other idea that's occurred to me (I think I saw it somewhere in the PLY pages, but I can't find it now) is to nest lexical scanners.  I keep my current scanner and feed it into a filter that mutates the tokens as needed and emits the tokens I need.  In practice, the messy parts of Wikipedia's mark-up can be resolved by looking a few tokens ahead. So, when I find '{{{{{', I can pull tokens until I find either a '}}}' (which seems to consistently be the fourth token after that one) or a '}}' (which I haven't seen occurring, but better safe than sorry), and then emit either '{{{' and '{{' or '{{' and '{{{'.

Here's a simple proof-of-concept:

    class wrapper(object):
        def __init__(self, klass):
            self.klass = klass
            self.stack = []
        def lex(self, *argv, **kwds):
            self.lex = self.klass.lex(*argv, **kwds)
            return self
        def input(self, *argv, **kwds):
            return self.lex.input(*argv, **kwds)
        def token(self):
            if self.stack:
                return self.stack.pop()
            token = self.lex.token()
            if token is not None:
                if token.type == 'LBRACES5':
                    new_token = lex.LexToken()
                    new_token.type = 'LBRACES3'
                    new_token.value = '{{{'
                    new_token.lineno = token.lineno
                    new_token.lexpos = token.lexpos
                    self.stack.append(new_token)
                    token.type = 'LBRACES2'
                    token.value = '{{'
                    new_token.lexpos += 2
            return token
        def __iter__(self):
            return self
        def next(self):
            t = self.token()
            if t is None:
                raise StopIteration
            return t
        __next__ = next

lexer = wrapper(lex).lex()
[...]
Reply all
Reply to author
Forward
0 new messages