[Python-ideas] Interrogate alternate namespace keyword and concept

[Prozacgod]

A while back i had learned about the 'with' statement and at first I naively thought that it worked similarly to a with statement I was familiar with, from delphi/pascal - obviously it doesn't and I was instantly hit with the idea of how useful it would be to have a keyword that could use a namespace from an object, hitting the familiar __getattr__ functions and related.

the keyword I proposed would be 'interrogate' (which is rather long, could use another one like 'using' or something) but.. thats not really important to me, the idea is, so

class test():
   def __init__(self):
      self.x = 42

foo = test()
bar = test()
x = 33
bar.x = 22
interrogate foo, bar:
  print x

-----

output is 42 since x is found inside of foo before bar, and locals would be interrogated last, but a more usefull example would be ..

class test():
  def __getattr__(self, name):
    if len(name) == 1 and (ord(name) in range(ord('a'), ord('z'))) :
      return ord(name)


test_ns = test()

interrogate test_ns:
  print a

----
output is 97

using the above example lexical closures may be awkward to implement in the language itself, this is a bit of a contrived example.. but.. eh..

# if I understand python enough, then I believe that when we enter this block, and create variables, they go out of scope,
# and there are no exceptions to this rule, ever so .. in order to implement mine obviously scope would be the same
# mechanism so I have oh_bother declared in the local namespace first

oh_bother = None
interrogate test_ns:
  def geronimo():
    return z
  oh_bother = geronimo

print oh_bother()
---
output is 122

--
-Prozacgod

"Prozac may heal the mind, but friends can mend the soul"

[Steven D'Aprano]

On Fri, 14 Aug 2009 04:22:07 pm Prozacgod wrote:
> A while back i had learned about the 'with' statement and at first I
> naively thought that it worked similarly to a with statement I was
> familiar with, from delphi/pascal - obviously it doesn't and I was
> instantly hit with the idea of how useful it would be to have a
> keyword that could use a namespace from an object, hitting the
> familiar __getattr__ functions and related.

What is the motivation? Is it just to reduce typing?

Being an ex-Pascal coder myself, I originally missed having a
Pascal-like 'with' statement too. But now I'm not so sure it is needed.

I know this proposal (under the name 'with') has been discussed before,
but I can't find a PEP for it... ah, here you go, it was in the FAQs:

http://www.python.org/doc/faq/general/#why-doesn-t-python-have-a-with-statement-for-attribute-assignments

C-# doesn't have one either:
http://msdn.microsoft.com/en-us/vcsharp/aa336816.aspx

I don't think there are any problems with a Pascal-style 'with'
statement that couldn't be overcome, but I don't think the benefit is
great enough to create a new keyword for it. Can you explain in more
detail why this proposed feature is useful?

--
Steven D'Aprano
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Nick Coghlan]

Steven D'Aprano wrote:
> I don't think there are any problems with a Pascal-style 'with'
> statement that couldn't be overcome, but I don't think the benefit is
> great enough to create a new keyword for it. Can you explain in more
> detail why this proposed feature is useful?

Also, if you just want to be able to chain multiple namespaces together,
you can do that by implementing an appropriate class with a custom
__getattr__ method.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------

[ilya]

Even if for some reason you needed a statement similar to Pascal's
"with" (and no, I don't know any example where this could be useful
since you'll lose access to anything other then the class), you could
implement it with existing syntax:

class Test:
'''Class to be interrogated.'''
def __init__(self, value):
self.value = value

test = Test(10)

class getattrs(dict):
'''An auxiliary class.'''
def __init__(self, instance):
self.instance = instance
def __getitem__(self, name):
return getattr(self.instance, name)
def __setitem__(self, name, value):
return setattr(self.instance, name, value)

# interrogate test: value += 5
exec('value += 5', getattrs(test))

print(test.value)
# prints 15.

[ilya]

Sorry, it's actually even easier; interrogate() is a one-liner:

class Test:
'''Class to be interrogated.'''
def __init__(self, value):
self.value = value

test = Test(10)

def interrogate(what, how):
exec(how, what.__dict__)

# interrogate test: value += 5

interrogate(test, 'value += 5')

# interrogate test: print(value)
interrogate(test, 'print(value)')

[Steven D'Aprano]

On Fri, 14 Aug 2009 07:35:55 pm ilya wrote:
> Sorry, it's actually even easier; interrogate() is a one-liner:
>
> class Test:
> '''Class to be interrogated.'''
> def __init__(self, value):
> self.value = value
>
> test = Test(10)
>
> def interrogate(what, how):
> exec(how, what.__dict__)

Apart from the security implications of exec(), it also takes a fairly
hefty performance hit. In Python 2.6:

>>> from timeit import Timer
>>> setup = 'from __main__ import test, interrogate'
>>> Timer("interrogate(test, 'value += 5')", setup).repeat()
[18.503479957580566, 18.218451023101807, 18.218581914901733]
>>> Timer("test.value += 5", setup).repeat()
[0.33056807518005371, 0.33118104934692383, 0.33114814758300781]

That's a factor of 55 times slower -- not precisely an optimization.

--
Steven D'Aprano

[Prozacgod]

The motivation came up in this thread on python-forum.org

http://www.python-forum.org/pythonforum/viewtopic.php?f=2&t=11606

My main motivation most likely is just that it would open up some very interesting meta-programing or dynamic programming opportunities, variables that exists entirely programmatically without the prefix of object_instance.foo, surely just syntactical sugar to some (or even most) - I have found it to be valuable syntax.  just like the current with statement bypasses some syntactical nusances of try: except: .. this would bypass the nusance of  inst.x inst.y inst.z ...

After posting last night I realized A pitfall of implementing this - would be

interrogate(foo, bar):
  xylophone = 1024

using this reference - ..
http://docs.python.org/reference/datamodel.html#customizing-attribute-access

__setattr__ has no current way to fall through, so foo would get the setattr event, and not be able to allow interrogate to hand it to bar.

I posit that raising the AttributeError exception would be sufficient, for the hand off to occur, and if foo, then bar both did it, then one of two ways to handle it would be the current scope variables would get it, (eg it would be __del__ after the end of the block) or putting the new variable in the parent namespace, since it is somewhat assumed that locals would follow after bar

[Prozacgod]

I just read this inside of the C# reasoning, and this pretty much kinda says, look its to difficult...

This is a good reason "Another approach is to push a scope and make the property hide the local variable, but then there's no way to refer to the local without adding some escape syntax."

bascially, they added the dot syntax to acces variables that are intended to be directed towards object namespace.  Which I do agree helps with disambiguation,

foo = 32
interrogate(test_ns):
  print .foo
  print foo

But don't find all that paletable.  My thoughts on this would be to cover the parent local namespace, and even globals with your object, the object gets it first everytime, and if it raised an exception, then the normal method would be implemented.  If it were important to have a variable that was named the same, then access it outide the interrogate block or otherwise..  But if you were a developer and working on writing a program, knowing you wanted to use interrogate, then wouldn't you be wise to choose different variable names?  I think I would.

Also doing this is kinda kludgy feeling to me..

function(args).dict[index][index].a = 21
function(args).dict[index][index].b = 42
function(args).dict[index][index].c = 63

write this:

ref = function(args).dict[index][index]
ref.a = 21
ref.b = 42
ref.c = 63

even though it is optimised, and works well, this feels more like the programmer adapting to the environment instead of the environment working for the programmer.

[ilya]

Well,

> foo = 32
> interrogate(test_ns):
> print .foo
> print foo
>

can be changed to

foo = 32
_ = test_ns
print _.foo
print foo

which actually has less symbols and doesn't have any ambiguity as for
the meaning of the new statement!

> But don't find all that paletable. My thoughts on this would be to cover
> the parent local namespace, and even globals with your object, the object

If a suggestion is to resolve names into several dictionaries, I don't
think this is practical in any way! As the source you cited correctly
explains, there is an irreparable problem. Suppose I write

foo = 32
interrogate(test_ns):
print bar # test_ns.bar
print foo # global foo

Then some time later I add global variable bar somewhere in the module
and the program is mysteriously broken! The same thing happens if you
first consult test_ns.__dict__ -- somebody can always add foo to
test_ns, its parent or *even object class*.

So I don't think there's a good way to define semantics of
'interrogate' without some kind of dot-syntax, which is debated for
Python for quite a long time without success.

[ilya]

That's a useful statistics, but the bottleneck is **only** because of
parsing 'value +=5'.

Here's how I time it:

# insert my old program here...

from timeit import Timer
from codeop import compile_command

def timing(something):
setup = 'from __main__ import test, interrogate, command, inc5'
best = sorted(Timer(something, setup).repeat(3, 1000))[0]
print('{0!r} -> {1:.3} ms'.format(something, best))
print('# test.value =', test.value)

command = '''
for i in range(10):
value += 1
'''

inc5 = compile_command(command)

timing("interrogate(test, command)")
timing(command.replace('value', 'test.value'))
timing("interrogate(test, inc5)")

Result:

15
'interrogate(test, command)' -> 0.0908 ms
# test.value = 30015
'\nfor i in range(10):\n test.value += 1\n' -> 0.00408 ms
# test.value = 60015
'interrogate(test, inc5)' -> 0.00469 ms
# test.value = 90015

so interrogate() with additional precompiling introduces very little
overhead. Though I agree it's inconvenient to write functions as
strings; I think someone smarter than me can find a way to do it like
a regular function call.

[Prozacgod]

This makes sense, obviously you should compile it first, which when executed should execute exactly the same as the original context python.

[Prozacgod]

Oh .. good call, you right, this would be difficult to observe, find, and correct

>foo = 32
>interrogate(test_ns):

>  print bar  # test_ns.bar
>  print foo  # global foo

>Then some time later I add global variable bar somewhere in the module
>and the program is mysteriously broken! The same thing happens if you
>first consult test_ns.__dict__ -- somebody can always add foo to
>test_ns, its parent or *even object class*.

In delphi/pascal they left this up to the developer, so you're supposed to divine that a new property was made on a class, to me that is a reasonable trade-off, I could choose to only use this syntax on code that I'm writing and supporting, or realize something could break - BUT I also understand the mentality of - it shouldn't randomly break ever, which is a reasonable supposition of code you write.

After this discussion I'm wondering why it ever ended up in any language to begin with?  Seems like it opens up to many issues to the developers using the language and the language implementors.

[Steven D'Aprano]

On Sat, 15 Aug 2009 06:29:11 am ilya wrote:
> That's a useful statistics, but the bottleneck is **only** because of
> parsing 'value +=5'.
>
> Here's how I time it:
>
> # insert my old program here...
>
>
> from timeit import Timer
> from codeop import compile_command
>
> def timing(something):
> setup = 'from __main__ import test, interrogate, command, inc5'
> best = sorted(Timer(something, setup).repeat(3, 1000))[0]

min(alist) is a more-direct, simpler, faster, easier to read way of
calculating sorted(alist)[0].

> print('{0!r} -> {1:.3} ms'.format(something, best))
> print('# test.value =', test.value)
>
> command = '''
> for i in range(10):
> value += 1
> '''

You're muddying the water by including a for-loop and call to range()
inside the code snippet being tested. We're trying to compare your
function interrogate(test, 'value += 1') with the standard call to
test.value += 1. Why include the time required to generate a range()
object, and iterate over it ten times, as part of the code snippet? All
that does is mix up the time required to execute common code and the
time required to execute the code we care about.

If it's not obvious why your approach is flawed, consider this extreme
example:

Timer("time.sleep(1000); interrogate(test, 'value += 1')", ...)
Timer("time.sleep(1000); test.value += 1", ...)

The differences in speed between the interrogate call (using exec) and
the direct access to test.value will be swamped by the time used by the
common code.

A more accurate measurement is to remove the "for i in..." part from
command, and increase the number=1000 argument to Timer.repeat() to
10000.

> inc5 = compile_command(command)

This is an unfair test. We're comparing directly accessing test.value
versus indirectly accessing test.value using exec. Regardless of
whether the caller compiles the statement manually before passing it to
exec, or just passes it to exec to compile it automatically, the cost
of that compilation has to be payed. Pulling that outside of the timing
code just hides the true cost.

Pre-compiling before passing to exec is a good optimization for the
cases where you need to exec the same code snippet over and over again.
If you want to execute "for i in range(1000): exec(s)" then it makes
sense to pull out the compilation of s outside of the loop. But
generally when timing code snippets, the only purpose of the loop is to
minimize errors and give more accurate results, so pulling out the
compilation just hides some of the real cost.

> timing("interrogate(test, command)")
> timing(command.replace('value', 'test.value'))
> timing("interrogate(test, inc5)")
>
> Result:
>
> 15

Where does the 15 come from?

> 'interrogate(test, command)' -> 0.0908 ms
> # test.value = 30015
> '\nfor i in range(10):\n test.value += 1\n' -> 0.00408 ms
> # test.value = 60015
> 'interrogate(test, inc5)' -> 0.00469 ms
> # test.value = 90015
>
> so interrogate() with additional precompiling introduces very little
> overhead.

Only because you're ignoring the overhead of pre-compiling. A more
accurate test would be:

timing("inc5 = compile_command(command); interrogate(test, inc5)")

> Though I agree it's inconvenient to write functions as
> strings;

If you think that's inconvenient, just try writing functions as code
objects without calling compile :)

[Steven D'Aprano]

On Sat, 15 Aug 2009 09:53:41 am Prozacgod wrote:
> This makes sense, obviously you should compile it first, which when
> executed should execute exactly the same as the original context
> python.

We want to compare the speed of the following statement:

test.value += 1

with the equivalent:

code = compile("value += 1", '', 'exec')
exec code in test.__dict__

Why do you think it is appropriate to ignore the time taken by the
compile() when comparing the two? Put it this way: if you want to know
how long it takes to make fresh stir-fried chicken, you have to include
the ten minutes it takes to chop up the vegetables and chicken, and not
just the two minutes it takes to stir-fry them.

[Antoine Pitrou]

Prozacgod <prozacgod@...> writes:
>
> My main motivation most likely is just that it would open up some very
> interesting meta-programing or dynamic programming opportunities, variables
> that exists entirely programmatically without the prefix of
> object_instance.foo, surely just syntactical sugar to some (or even most) -

This sounds very PHP-like (doing indirections like $$foo because there isn't any
powerful introspection or OO system, and many things (e.g. functions, classes)
aren't first-class objects and can't be passed as parameters). I don't think
Python should grow something like that; instead people should learn more
structured ways of accessing data. getattr() and friends are powerful enough for
playing access tricks.

(in other words: -1 from me)

[Prozacgod]

I'm leaning towards a -0.5 right now myself, simply because of the ambiguities that have been mentioned, python is anything but ambiguous, and that's a distinction I like about it, in every case I can think of python does what you'd expect without random oh, but there is this qualifier in this context .. (*cough* perl *cough* ruby *cough*).. even if I think I like the syntax, it doesn't really make sense to implement it at this time, or likely ever.

I got the inspiration for the idea from delphi/pascal, but in retrospect it turns out that the primary large (~1million lines) commercial app I worked on where we used that syntax (heavily I might add), they started to phase out the use of the with statement all over the code because of these same abiguities creating obscure difficult to track down bugs.

[Greg Ewing]

Prozacgod wrote:

> After this discussion I'm wondering why it ever ended up in any language
> to begin with?

Pascal has it because it doesn't have any general way of
binding a local name to some nested part of a data structure.

You can always do this in Python, and also in C using the
& operator, so those languages don't need a Pascal-style
with-statement.

--
Greg

[Greg Ewing]

Steven D'Aprano wrote:
> if you want to know
> how long it takes to make fresh stir-fried chicken, you have to include
> the ten minutes it takes to chop up the vegetables and chicken,

Don't forget to include the time required to catch and kill
the chicken as well!

--
Greg

[Steven D'Aprano]

On Sun, 16 Aug 2009 11:09:59 am Greg Ewing wrote:
> Steven D'Aprano wrote:
> > if you want to know
> > how long it takes to make fresh stir-fried chicken, you have to
> > include the ten minutes it takes to chop up the vegetables and
> > chicken,
>
> Don't forget to include the time required to catch and kill
> the chicken as well!

"In order to make an apple pie from scratch, you must first create the
universe." -- Carl Sagan

Of course all timing tests make certain assumptions, and the art of
optimization is partly to recognise when those assumptions are
violated. If you do have a situation where you are executing the exact
same code repeatedly, then pre-compiling is a good optimization over
calling exec on the string directly. But you can't exclude the time for
pre-compiling in the general case.

--
Steven D'Aprano

[ilya]

Thanks for code tips! I appreciate them.

Ok, I kind of not sure I correctly remember why this topic was raised,
so let me remember.
You said that using a separate function has a performance hit. The
purpose of my program was to establish that timing of

function interrogate() == statement interrogate + compile time

It doesn't appear to me that you dispute this claim, so let's accept
it. Since a typical program shouldn't compile lots of code snippets
(not in number of millions certainly) this means the performance of
function and statement is the same for out purposes. I believe this
closes performance question.

> min(alist) is a more-direct, simpler, faster, easier to read way of
> calculating sorted(alist)[0].

> You're muddying the water by including a for-loop and call to range()