BOOLEAN grammar using modgrammar

[Ravi Shankar]

Hello, 

I want to implement a simple boolean grammar using modgrammar for a python based search module. All i want to support is AND and OR with different predicates. For eg: i want the users to enter a search query like the following: 

att1=val1 AND att2=val2 AND att3>val3 OR (att4<val4 AND att5=val5)

I can write the comparators for attributes external to modgrammar that will determine the value of att1=val1 , att2=val2 , att4<val4 etc. 

How can i use modgrammar to define this grammar and then obtain the resulting boolean value of the final expression ? 

[Alex Stewart]

Hey Ravi,

That task should actually be fairly similar to the task of parsing and evaluating infix math expressions, which is actually included as an example case in the Modgrammar documentation (http://packages.python.org/modgrammar/examples/infix_precedence.html), so you might want to start from there.

You will, obviously, want to change some of the operators from "/", "*", "+", etc to be "AND", "OR", etc, and change what the individual value() methods do to evaluate each bit of the expression, but other than that the process should be almost exactly the same..

One thing you will need to be careful of, since you are using text words as operators ("AND" and "OR"), instead of special symbols, is how you handle whitespace.  You will probably want to set the P*Expr classes for "AND" and "OR" to have grammar_whitespace_mode = 'required'.  This will cause Modgrammar to make sure that the input has to have whitespace around the "AND" or "OR" in order to match (so it won't match something like "SAND=MORE" in the input and think it's a boolean expression)..

Please let me know if you have more specific questions, or run into problems, and I can try to help you out..

--Alex

--
You received this message because you are subscribed to the Google Groups "modgrammar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modgrammar+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ravi Shankar B]

Thanks Alex for your response ! I could put together something and get it working. However, since i am using python 2.6 in our environment that i cannot change because of some restrictions in our proudct's python libraries , i have the following two questions: 

1. Do you have any backport of modgrammar 0.10 that will work with python 2.6 ? 

2. I found an older backport of modgrammar @ http://rembish.github.io/modgrammar-py2/index.html which i am using to get this working which is based out of an older version of modgrammar. Following is the code i have for this one: 

{code}

from modgrammar import *

class PredicateKeywordAttribute (Grammar):

    """

    Defines the grammar for keyword search attributes. (method, module, package, class)

    """

    grammar = (LITERAL("method") | LITERAL("module") | LITERAL("package") | LITERAL("class"))

class PredicateOperator (Grammar):

    """

    Operators that we support for predicates. (=, !=)

    """

    grammar = (LITERAL("=") | LITERAL("!="))

class Predicate (Grammar):

    """

    Defines the grammar for a predicate.

    Eg: module = test*, package=a.b.*, method=test_smoke, class=Test*Smoke*Tests

    """

    grammar = (PredicateKeywordAttribute, PredicateOperator, OPTIONAL(LITERAL("\"") | LITERAL("'")),

               WORD("*A-Z0-9a-z._"), OPTIONAL(LITERAL("\"") | LITERAL("'")))

class Operator (Grammar):

    """

    Operator supported between predicates. (and, or)

    """

    grammar_whitespace = False

    grammar = (LITERAL("and") | LITERAL("or") | LITERAL("AND") | LITERAL("OR"))

class Query (Grammar):

    """

    Grammar for the final query expression which will just be a list of predicates separated by the operator.

    """

    grammar = (LIST_OF(Predicate, sep=Operator),";")

{code}

However this one seems to pass the query text "class = test*AND module = test*;* even when i set grammar_whitespace = False or grammar_whitespace = True for the grammar 'Operator'. 

myparser = Query.parser()

result = myparser.parse_string("class = test*AND module = test*;") 

The above test gives me back the following with both grammar_whitespace set to True or False. 

class = test*AND module = test*;: Parsed elements - (<LIST><'class = test*', 'AND', 'module = test*'>, L(';')<';'>)

How can i force the grammar to require whitespace around 'AND' and 'OR'. 

And thanks for the wonderful easy library ! 

Ravi 

--
You received this message because you are subscribed to a topic in the Google Groups "modgrammar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modgrammar/QPIV5DKwxsk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modgrammar+...@googlegroups.com.

[Alex Stewart]

1. Do you have any backport of modgrammar 0.10 that will work with python 2.6 ? 

Unfortunately, no.. as you discovered Alex Rembish did apparently port a version of Modgrammar to Python 2.x, but unfortunately since he didn't let me know about it at the time, I didn't find out until I stumbled across it some time after I'd already made quite a few changes to the codebase, so it wasn't something I could easily merge back in..

 

I'm not particularly opposed to making Modgrammar work in older versions of Python, and would be open to any submissions people might have in that regard, I just haven't ever had a need for it in my own projects, and with the other stuff on my plate at the moment I don't have a lot of time to spend on that sort of thing..

2. I found an older backport of modgrammar @ http://rembish.github.io/modgrammar-py2/index.html which i am using to get this working which is based out of an older version of modgrammar. Following is the code i have for this one: 

<snip>  

However this one seems to pass the query text "class = test*AND module = test*;* even when i set grammar_whitespace = False or grammar_whitespace = True for the grammar 'Operator'. 

Yeah, one of the biggest changes I've made since the version that rembish took was to substantially revamp the whitespace handling (and whitespace options) largely because it wasn't that easy to do that sort of thing (e.g. requiring whitespace in specific locations) with the old model..

One thing to note is that in the older version, "grammar_whitespace = True" does NOT require that whitespace be present, it just makes it accept it if it is (i.e. it is the equivalent of grammar_whitespace_mode='optional' in the current Modgrammar).  What you probably want instead is to set grammar_whitespace = False everywhere (that is, at the module level), and then explicitly put WHITESPACE tokens in those places where you want to require there be whitespace.

So for example, you might want to make Operator be more like:

class Operator (Grammar):

    """

    Operator supported between predicates. (and, or)

    """

    grammar_whitespace = False

    grammar = (WHITESPACE, LITERAL("and") | LITERAL("or") | LITERAL("AND") | LITERAL("OR"), WHITESPACE)

Note also that in order for that to work, the initial whitespace must not be consumed by any other grammar before it gets to looking for an Operator, which means that grammar_whitespace must be False for any enclosing grammar as well (in this case, Query).  If you set grammar_whitespace = False at the module level, then you shouldn't have to worry as much about that.  (This is another aspect which is substantially improved in the newer versions of Modgrammar..)

So that might work for you..  I'm not sure off hand whether that will solve all your problems or not, though.. it depends a bit on how complex your grammar ultimately ends up being..

--Alex