Use of eval()
Kent
version that just makes a specialized Rule class to handle the
urldata=key:value rules. Do you have other use cases that require the
eval() or just that one?
Kent
Brian Jones
I was slowly coming to the conclusion that the Rule class could either
be replaced by a simple list, or could be further enhanced to support
more advanced rules :-)
The use of eval() was specifically to handle the urldata=key:value
scenario. You can read *exactly* the reasoning behind the use of
eval() in this instance on my blog -->
http://www.protocolostomy.com/2008/03/11/what-i-learned-about-python-today-eval/
You can also see that I took some heat for it in the comments :)
brian.
Kent
that gets rid of eval(). Each rule has a getter attribute that gets
the value for comparison. Most rules use a plain operator.attrgetter
for the rule. A separate UrlDataRule class handles the special case of
url parameter matching.
I get about a 3x speedup with this change. I hope you like it!
Kent
On Mar 19, 8:04 am, Brian Jones <bkjo...@gmail.com> wrote:
> Hi Kent,
>
> I was slowly coming to the conclusion that the Rule class could either
> be replaced by a simple list, or could be further enhanced to support
> more advanced rules :-)
>
> The use of eval() was specifically to handle the urldata=key:value
> scenario. You can read *exactly* the reasoning behind the use of
Brian Jones
I haven't been able to look at the code, but coincidentally I needed
loghetti to run really fast just as this message came in, so I svn
up'd, ran my query, and it worked great after I added the explicit
"(object)" to the UrlData class definition (apparently required for
python < 2.5).
Also coincidentally, I was using loghetti to parse --urldata, so it
was a good test ;-)
In addition, I also ran it for the first time with CommandLineApp 2.6,
which worked fine as well.
Kent, whenever you have a moment, if you could further explain the
performance issues with using eval() in this context, or point me to
something that explains that, I'd be really grateful, because I'm
unsure how to unwrap eval() to see what operations it's performing
that make it inefficient.
Thanks a bunch.
brian.
--
Brian K. Jones
Python Magazine http://www.pythonmagazine.com
My Blog http://www.protocolostomy.com
Kent Johnson
> it worked great after I added the explicit
> "(object)" to the UrlData class definition (apparently required for
> python < 2.5).
?? It is good practice but I can't think of a reason why it would be
required. Were you having some problem without it?
> Kent, whenever you have a moment, if you could further explain the
> performance issues with using eval() in this context, or point me to
> something that explains that, I'd be really grateful, because I'm
> unsure how to unwrap eval() to see what operations it's performing
> that make it inefficient.
Python is often thought of as an interpreted language but that is not
completely true. Python code compiles to byte-code which is then
interpreted. Running a python program, the whole program (actually each
separate module) is compiled to byte-code when it is loaded, then the
interpreter runs to execute it.
When you use eval(), you are passing it a string. The string has to be
byte-compiled, then interpreted.
So using eval() to evaluate the rules, every (used) rule is compiled for
every line of the log file. This is expensive.
One way to optimize this would be to explicitly compile the string and
eval() the compiled code. You can do this using compile().
http://docs.python.org/lib/built-in-funcs.html#l2h-18
But for loghetti, the eval() didn't really seem to be adding much value
- and as you found out, eval() is generally frowned on when there is an
alternative - so I just got rid of it.
Kent
Mohanaraj
Somewhat tangential to the your email, I have a query regarding the
Rule changes you have made.
I like the fact that you have taken out the eval - I think thats
headed in the right direction as well - however , how do you see the
application of the 'cmp' attribute of the Rule class ? I have made
some suggestions on how the Rule class and its interaction with the
strainer method of Filter class can be changed as well in an earlier
post. What do you think of my suggestions ?
Either way, it does seem like we are going to need some stabilization
of these issues so that we can start building more cool Filters
classes for loghetti. Kevin ?
Just my 2 cents.
Mohan
Kent Johnson
> I like the fact that you have taken out the eval - I think thats
> headed in the right direction as well - however , how do you see the
> application of the 'cmp' attribute of the Rule class ? I have made
> some suggestions on how the Rule class and its interaction with the
> strainer method of Filter class can be changed as well in an earlier
> post. What do you think of my suggestions ?
I saw your earlier suggestion. I didn't follow it because
- you didn't actually add any functionality
- introducting another function call into the filter loop slows it down
slightly but noticably
I didn't want to impact performance without a gain in functionality.
I do think that having a Rule method that fully evaluates the Rule,
returning True or False, is a good design. I'm not sure if I would use
the __call__() method or perhaps make an eval() method but that is a
fine point. Hmm, the rules passed to the filter could actually be simple
functions created by a factory, or bound methods of a Rule instance;
this might be faster than storing the Rules themselves.
Using a switch to evaluate the comparison is probably not a good idea. I
would put the switch in the __init__() method and either
- pick and store a comparison function from the operator module
- create a custom eval() method that does the right thing
Kent
Brian Jones
On Mon, Mar 31, 2008 at 9:26 PM, Kent Johnson <ken...@tds.net> wrote:
> ?? It is good practice but I can't think of a reason why it would be
> required. Were you having some problem without it?
Without the explicit "object", I (and others) get a "syntax error",
but only for python versions prior to 2.5. My dev box is 2.5.1, but my
production machines are 2.4, and a buddy runs 2.3 and saw the issue as
well.
> When you use eval(), you are passing it a string. The string has to be
> byte-compiled, then interpreted.
Ok, so I haven't had time to really study the code you put in place at
all, but just to generalize this concept -- wouldn't this be true for
passing any type (string, int, mytype, whatever) to any function or
method? Some of the tips I've picked up about repeated method calls
and moving things outside of loops (or even classes) are old lessons,
but have a much greater impact when you're testing with 250MB (or
larger) log files. Thanks for that.
>
> So using eval() to evaluate the rules, every (used) rule is compiled for
> every line of the log file. This is expensive.
Right on.
> One way to optimize this would be to explicitly compile the string and
> eval() the compiled code. You can do this using compile().
> http://docs.python.org/lib/built-in-funcs.html#l2h-18
I haven't ever used compile() -- thanks.
>
> But for loghetti, the eval() didn't really seem to be adding much value
> - and as you found out, eval() is generally frowned on when there is an
> alternative - so I just got rid of it.
Well, somewhere I admitted that I doubted eval() was *The* solution.
It was one way to get over a particular hump at the time, but it
sounds like what you've committed is quite a bit better in terms of
elegance, not to mention performance.
I plan to study and understand this code before the end of this week,
so that I can "0wn* the concepts and not be such a bonehead going
forward. :-)
brian.
Mohanaraj Gopala Krishnan
>
> Mohanaraj wrote:
>
> > I like the fact that you have taken out the eval - I think thats
> > headed in the right direction as well - however , how do you see the
> > application of the 'cmp' attribute of the Rule class ? I have made
> > some suggestions on how the Rule class and its interaction with the
> > strainer method of Filter class can be changed as well in an earlier
> > post. What do you think of my suggestions ?
>
> I saw your earlier suggestion. I didn't follow it because
> - you didn't actually add any functionality
> - introducting another function call into the filter loop slows it down
> slightly but noticably
I appreciate the performance concern. What I had hoped to do was to
add a bit more
flexibility to the way the Rule class could be used by letting the cmp
attribute actually define
the comparison operation. So this will allow for the comparison to be
determined at the point
of instantiation. I agree that it did not directly add end user
functionality, but I was hoping it could
make writing rules easier.
> I didn't want to impact performance without a gain in functionality.
>
<snip/>
> Hmm, the rules passed to the filter could actually be simple
> functions created by a factory, or bound methods of a Rule instance;
> this might be faster than storing the Rules themselves.
Interesting observation.
> Using a switch to evaluate the comparison is probably not a good idea. I
> would put the switch in the __init__() method and either
> - pick and store a comparison function from the operator module
> - create a custom eval() method that does the right thing
Point taken.
Mohan
Kent Johnson
> Without the explicit "object", I (and others) get a "syntax error",
> but only for python versions prior to 2.5. My dev box is 2.5.1, but my
> production machines are 2.4, and a buddy runs 2.3 and saw the issue as
> well.
Ah. The parentheses are the problem, in Python < 2.5 you would say either
class Foo(object):
or just
class Foo:
with no parens.
Anyway thanks for the fix and the explanation.
>> When you use eval(), you are passing it a string. The string has to be
>> byte-compiled, then interpreted.
>
> Ok, so I haven't had time to really study the code you put in place at
> all, but just to generalize this concept -- wouldn't this be true for
> passing any type (string, int, mytype, whatever) to any function or
> method? Some of the tips I've picked up about repeated method calls
> and moving things outside of loops (or even classes) are old lessons,
> but have a much greater impact when you're testing with 250MB (or
> larger) log files. Thanks for that.
Well, it's true that any strings or other constants in the program must
be compiled, but in general this happens just once, when the module is
loaded, as part of the module compilation. Strings passed to eval() are
compiled every time the eval() executes. This is more comparable to the
module compilation than to the compilation of a string constant.
Kent