Python immitator [was RE: embedding - parsing]
Zack Roadhouse
Here yah go. For all of you who need more code than what Just posted
this morning. Here is a python script that emulates the interpreter
behavior. This code is usefull for those situations in which you want
more controll over the interpreter or when you want to talk to a
python process using a pipe or socket instead of a tty. Here is an
example use (using the ShellWindow.py in the Demos/tkinter directory
of the Python 1.4 distribution):
---- pyshell.py -----
import Tkinter,ShellWindow
root = Tkinter.Tk()
root.minsize(1, 1)
sw = ShellWindow.ShellWindow(root,"python1.4 -u main.py")
sw.pack(expand=1, fill="both")
sw.focus_set()
root.mainloop()
----------------------
---- main.py -----
#!/usr/bin/env python1.4
import sys, os, traceback, string
from types import *
def print_exc(limit=None, file=None):
"""slight modification of traceback.print_exc() which
skips the outermost traceback object."""
if not file:
file = sys.stderr
tb = sys.exc_traceback
# we're going to skip the outermost traceback object, we don't
# want people to see the line which excecuted their code.
if tb:
tb = tb.tb_next
traceback.print_exception(sys.exc_type, sys.exc_value, tb,
limit, file)
file.flush()
class ExitPython:
"""A class which when called or examined on
the command line, will exit the interpreter."""
def __call__(self):
raise SystemExit
def __repr__(self):
print "Come again!"
raise SystemExit
# If the shell prompts aren't set then
# use the normal defaults.
if not hasattr(sys, 'ps1'):
sys.ps1 = '>>> '
if not hasattr(sys, 'ps2'):
sys.ps2 = '... '
# Maximum size of a single python code chunk
read_chunk = 2**16
# Python buffer -- place to store multiline code strings
pybuf = ''
# Stuff to be executed
stuff = ''
# Dictionary in which to execute the code.
# Pretends to be main.
execution_space = {
"__doc__" : "Zachary Roadhouse's Python Interpreter Emulator",
"__name__" : "__main__",
"exit" : ExitPython()
}
#import __main__
#execution_space = __main__.__dict__
#exit = ExitPython()
# Write the welcoming sign
print "Python",sys.version
print sys.copyright
sys.stdout.write(sys.ps1)
sys.stdout.flush()
while 1:
stuff = os.read(sys.stdin.fileno(),read_chunk)
# Remove whitespace
temp = string.strip(stuff)
# If the buffer is not empty and the input is empty,
# it is the end of a multiline expression.
# Transfer the contents of the buffer to be executed
# and clear the buffer.
if pybuf and not temp:
stuff = pybuf + '\n'
pybuf = ''
# Check the last character of the input to see if
# it signals the start of a multiline expression.
elif temp and ((temp[-1] == ':') or (temp[-1] == '\\')):
pybuf = pybuf + stuff + '\n'
# There is something in the buffer, add the new
# stuff to it.
elif pybuf:
pybuf = pybuf + stuff + '\n'
# If the buffer is empty and there is something to execute
if not pybuf and string.strip(stuff):
try:
code = compile(stuff + '\n', '<stdin>', 'single')
except SyntaxError, detail:
if type(detail) == TupleType and detail[0] == \
'unexpected EOF while parsing':
pybuf = stuff + '\n'
else:
print_exc()
pybuf = ''
else:
try:
exec code in execution_space
except SystemExit, msg:
raise SystemExit, msg
except:
print_exc()
pybuf = ''
# If the buffer is not empty, write out the second
# prompt indicting a multi-line expression
if pybuf:
sys.stdout.write(sys.ps2)
else:
sys.stdout.write(sys.ps1)
sys.stdout.flush()
--
--Zack
/---------------------------------------------------------------------
Zachary F. Roadhouse Home: (401)863-xxxx
Box 220, Brown University, Providence, RI 02912 Work: (401)863-2177
E-MAIL: Zachary_...@brown.edu
WWW: http://www.techhouse.brown.edu/~zack
Manus Hand
The Python imitator is cool, but (though this comes as no surprise to anyone)
not quite complete. The most important difference is that I can't seem to
get the Python imitator to accept incomplete tuples, lists, and dictionaries:
>>> (1,2,3
File "<string>", line 2
^
SyntaxError: invalid syntax
The part of the code which seems to want to remedy this is:
try:
code = compile(stuff + '\n', '<stdin>', 'single')
except SyntaxError, detail:
if type(detail) == TupleType \
and detail[0] == 'unexpected EOF while parsing':
pybuf = stuff + '\n'
but at least in Python 1.5a2 (my personal plug to get you to join the PSA),
the [0] element of the exception detail is nothing but "invalid syntax" and
I can get this same detail[0] to be generated with blatantly wrong (though
complete) lines like, oh, say, just a single dollar-sign.
Another thing which shows up as "invalid syntax" in the true Python
interpreter but not in the imitator is a line like:
4:
You'd think this would be fairly trivial to fix, except that you have this
behavior of the real interpreter to worry about:
"test"; if 1: <--- syntax error
if 1: # comment <--- just fine
if 1 test: <--- syntax error
if "1: <--- syntax error
if """1: <--- just fine, with an open """
Anyway, although I know we're beyond the field of real usefulness, it would
be interesting to make an imitator which behaves EXACTLY like the interpreter.
I have made some improvements to the input mechanism (as posted, it did
not recognize EOF as an exit criterion, nor did it handle KeyboardInterrupts
correctly).
Anyone have any ideas that would help the continuation line problem (short
of reimplementing in Python pieces of the parser, which I'm afraid is the
only way to address some of the later examples)? Especially the apparent
difference between 1.4 and 1.5a2's exception detail for literal sequences
left unclosed?
Manus
Manus Hand
The imitator is cool (but incomplete, of course). My most important
problem is that, using 1.5a2 (there's my blatant plug for youse to
join the PSA), unclosed tuples, lists, and dictionaries are not
detected:
>>> (1,2,3
File "<string>", line 2
^
SyntaxError: invalid syntax
The part of the code that seems to want to solve this problem is:
try:
code = compile(stuff + '\n', '<stdin>', 'single')
except SyntaxError, detail:
if type(detail) == TupleType \
and detail[0] == 'unexpected EOF while parsing':
pybuf = stuff + '\n'
but in 1.5a2, the [0] element of the exception detail is nothing but
"invalid syntax". This same detail is given by such blatantly wrong
(but otherwise complete) lines like, oh, say a single dollar-sign.
The whole continuation line detection mechanism is but an incomplete
imitation. For instance, all the following cases would need to be
taken into account (with the true Python's verdict on them as specified
below):
"test"; if 1: <--- syntax error
if 1: # comment <--- just fine
if 1 tests: <--- syntax error
if """1 <--- just fine
Unfortunately, this looks to me like to write a complete interpreter
imitator in Python, it would take reimplementing some parts of the
parser :-(
I know we're far beyond the realm of usefulness (or are we?), but it
is an interesting exercise (to me, at least) to plug these holes and
be able to have a "more perfect" imitator.
I've remedied some shortcomings of the input mechamism (it didn't
detect EOF as an exit criterion, and it didn't handle KeyboardInterrupts
correctly), but the apparent difference between 1.4 and 1.5a2's handling
of the syntax error for unclosed literals necessitated this message, and
my dawning realization (as touched on above) that even fixing that one
case that was supposedly already covered by Zack's code won't complete
the project.
Anyone have any ideas how to solve these problems?
Manus
Manus Hand
The imitator is cool but (of course) incomplete. The most important
problem I've found is that in 1.5a2 (my blatant plug for youse to
join the PSA), unclosed tuples, lists, and dictionaries are not
detected:
>>> (1,2,
File "<string>", line 2
^
SyntaxError: invalid syntax
The part of the code that seems to want to remedy this is:
try:
code = compile(stuff + '\n', '<stdin>', 'single')
except SyntaxError, detail:
if type(detail) == TupleType \
and detail[0] == 'unexpected EOF while parsing':
pybuf = stuff + '\n'
In 1.5a2, the [0] element of the exception detail is simply "invalid syntax."
I can get this same detail to show up for just about any old syntax error,
such as a line containing only a single dollar-sign character.
Even if this problem were solved, the imitatation would not be 100%. Here
are some cases that need to be coded for (with the true Python's verdict
on their legality):
"test"; if 1: <--- syntax error
if 1: # comment <--- just fine
if 1 tests: <--- syntax error
if """1 <--- just fine
So it seems that to make the thing truly accurate would require recoding
some parts of the parser in Python. :-(
Although I know we're beyond the realm of usefulness (or are we?), it is
an interesting exercise (to me, at least) to write a "more perfect"
interpreter imitator.
I have gussied up the input mechanism some (it didn't recognize EOF as an
exit criterion and it didn't handle KeyboardInterrupts like the true
interpreter does). Anyone have any bright ideas for how to plug some of
the other holes mentioned in this message (most especially the apparent
difference in the way 1.4 and 1.5a2 handle unclosed literals)?
Manus
Manus Hand
Forgive the repeat postings. My news poster had me fooled into believing it
was throwing my messages away without posting them. I'm not fooled easily;
it was doing a pretty darn good imitation of a broken newsreader. Anyway,
my apologies for all the wasted bandwidth, but I can make it all worth it
if only one reader of his humble message of regret will read and obey the
following extremely well-hidden subliminal message:
JjOiIiNn TtHhEe PpSsAa....
Manus
Guido van Rossum
> The imitator is cool but (of course) incomplete. The most important
> problem I've found is that in 1.5a2 (my blatant plug for youse to
> join the PSA), unclosed tuples, lists, and dictionaries are not
> detected:
>
> >>> (1,2,
> File "<string>", line 2
>
> ^
> SyntaxError: invalid syntax
I experimented a bit with this, and found out that you have to attempt
to compile the source *three* times: as is, with a newline appended,
and with *two* newlines appended. The program below ("mainloop.py")
will do a decent job of imitating Python's behavior. It differs in
two ways that I can't be bothered to fix right now:
1) If you type a comment by itself, it will continue to emit a
continuation prompt until you type a statement. "Real" Python will
give a continuation prompt too (alas), but it will give up when you
enter a blank line. The only solution I can think of is to explicitly
check for a # sign as the first nonblank character.
2) If you enter the start of a statement that requires a block
(e.g. "if 1:") and then enter a blank line, again, the code below
will ask for more; the real Python gives a syntax error. (Blank lines
inside brackets are treated the same by both though -- both ask for
more.)
As far as I can tell, this works identical with Python 1.4 and 1.5.
--Guido van Rossum (home page: http://www.python.org/~guido/)
"""mainloop.py -- a nearly exact imitation of the Python main loop."""
import traceback, sys
def run(code, env):
try:
exec code in env
except:
sys.last_type = sys.exc_type
sys.last_value = sys.exc_value
sys.last_traceback = sys.exc_traceback
traceback.print_exception(sys.last_type,
sys.last_value,
sys.last_traceback)
def mainloop(env):
print "Fake Python", sys.version
print sys.copyright
# Set sys.ps1 and sys.ps2 if they are undefined;
# they are only set in interactive mode
try:
sys.ps1
except AttributeError:
sys.ps1 = ">>> "
try:
sys.ps2
except AttributeError:
sys.ps2 = "... "
# Source collected so far; empty if at start of statement
source = ""
while 1:
if source:
prompt = sys.ps2
else:
prompt = sys.ps1
try:
line = raw_input(prompt)
except EOFError:
break
except KeyboardInterrupt:
source = ""
print "\nKeyboardInterrupt"
continue
if source:
source = source + "\n" + line
else:
source = line
# Compile three times: as is, with \n, and with \n\n appended.
# If it compiles as is, it's complete. If it compiles with
# one \n appended, we expect more. If it doesn't compile
# either way, we compare the error we get when compiling with
# \n or \n\n appended. If the errors are the same, the code
# is broken. But if the errors are different, we expect more.
# Not intuitive; not even guaranteed to hold in future
# releases; but this matches the compiler's behavior in Python
# 1.4 and 1.5.
err = err1 = err2 = None
code = code1 = code2 = None
try:
code = compile(source, "<input>", "single")
except SyntaxError, err:
pass
try:
code1 = compile(source + "\n", "<input>", "single")
except SyntaxError, err1:
pass
try:
code2 = compile(source + "\n\n", "<input>", "single")
except SyntaxError, err2:
pass
if code:
run(code, env)
source = ""
elif code1:
pass
elif err1 == err2:
traceback.print_exception(SyntaxError, err1, None)
source = ""
if __name__ == '__main__':
mainloop(globals())
Tim Peters
> [manus hand, wrestling with new news software]
> ...
> Anyway, my apologies for all the wasted bandwidth, but I can make
> it all worth it if only one reader of his humble message of regret will
> read and obey the following extremely well-hidden subliminal
> message:
>
> JjOiIiNn TtHhEe PpSsAa....
Wow -- that's about the nicest stereogram I've ever seen! How did you make it
shimmer? Let alone rotate?! Those new PSA Decoder Glasses make everything
more beautiful.
psa-cured-my-kidney-stones-too-ly y'rs - tim
Tim Peters tim...@msn.com, t...@dragonsys.com
not speaking for Dragon Systems Inc.