I'm wanting to parse some Wikipedia pages. Wikipedia template data looks like this: {{my template|arg one|arg two|keyword=value}} In a template definition, you can use variable expansion, like this: {{{1|default for arg one}}} I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run. My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in: {{{{{keword}}}|arg one}} Those five braces in a row are problematic. My first thought is that I'd like to return two tokens when I see them, a two brace token followed by a three brace one, but I'm having problems figuring out how to do that. My second thought is to define parser rules that start with a five-brace token, but that's not so easy, either. Any suggestions on how to fix things? Thanks.
> I'm wanting to parse some Wikipedia pages. > Wikipedia template data looks like this: {{my template|arg one|arg two|keyword=value}} > In a template definition, you can use variable expansion, like this: {{{1|default for arg one}}} > I defined my lexer to grab runs of '{' and '}' and return different tokens depending on the length of the run. > My problem is, I'm hitting cases where a template's name is a variable expansion, resulting in: {{{{{keword}}}|arg one}}
If this is the only way they can be nested, you can use scanner states, that is, define a scanner state 'outside template', which matches {{ only. when encountering {{, switch to a 'inside template' scanner state which matches {{{ only. When encountering }}, switch back to the 'outside template' scanner state.
An alternative solution would be to use a scannerless parser. I am however not sure whether these exist for Python.
> On 09/06/11 00:20, Sam Denton wrote: >> I'm wanting to parse some Wikipedia pages. >> Wikipedia template data looks like this: {{my template|arg one|arg >> two|keyword=value}} >> In a template definition, you can use variable expansion, like this: >> {{{1|default for arg one}}} >> I defined my lexer to grab runs of '{' and '}' and return different tokens >> depending on the length of the run. >> My problem is, I'm hitting cases where a template's name is a variable >> expansion, resulting in: {{{{{keword}}}|arg one}}
> If this is the only way they can be nested, you can use scanner states, that > is, define a scanner state 'outside template', which matches {{ only. when > encountering {{, switch to a 'inside template' scanner state which matches {{{ > only. When encountering }}, switch back to the 'outside template' scanner state.
> An alternative solution would be to use a scannerless parser. I am however not > sure whether these exist for Python.
Also, have you investigated the tools listed on this page?
I had looked at those before starting this project, and at the time they seemed a bit overkill for what I needed. Now, of course, they seem less so. :)
Of the tools listed, mwlib is the only one that looks suitable for my needs. All the other Python solutions are geared towards creating human readable output, whereas I'm wanting to populate a database. I'm also looked for a pure Python solution, as I plan to deploy onto Google's app engine (where the database will be available under a Creative Commons license compatible with Wikipedia).
One other idea that's occurred to me (I think I saw it somewhere in the PLY pages, but I can't find it now) is to nest lexical scanners. I keep my current scanner and feed it into a filter that mutates the tokens as needed and emits the tokens I need. In practice, the messy parts of Wikipedia's mark-up can be resolved by looking a few tokens ahead. So, when I find '{{{{{', I can pull tokens until I find either a '}}}' (which seems to consistently be the fourth token after that one) or a '}}' (which I haven't seen occurring, but better safe than sorry), and then emit either '{{{' and '{{' or '{{' and '{{{'.
Here's a simple proof-of-concept:
class wrapper(object): def __init__(self, klass): self.klass = klass self.stack = [] def lex(self, *argv, **kwds): self.lex = self.klass.lex(*argv, **kwds) return self def input(self, *argv, **kwds): return self.lex.input(*argv, **kwds) def token(self): if self.stack: return self.stack.pop() token = self.lex.token() if token is not None: if token.type == 'LBRACES5': new_token = lex.LexToken() new_token.type = 'LBRACES3' new_token.value = '{{{' new_token.lineno = token.lineno new_token.lexpos = token.lexpos self.stack.append(new_token) token.type = 'LBRACES2' token.value = '{{' new_token.lexpos += 2 return token def __iter__(self): return self def next(self): t = self.token() if t is None: raise StopIteration return t __next__ = next
