Metasyntax/Macros
Paddy McCarthy
but ended up with just too many hits and got lost.
The reason I was looking was that I can get envisage an application
but seemed to remember that the addition of macros to Python got shot
down before but could not remember the reasons.
How I think it would be used? - In Electronics people write simulation
models 'programs' in several domain specific languages - VHDL, Verilog
etc. When they need to Verify these models they find that they gain
from using a higher level programming language, and often augment
their test suite with scripting languages like Perl as well as domain
specific languages such as Specman and Vera. If Python was that higher
level language then i thought that a macro facility would allow some
of the domain specific languages syntax to be re-used , People could
write
module my_mod (x,y,z) ...
and create a Python class. By choosing a careful set of macros the
resultant {ython-with-macros source could be understandable to people
with only the domain specific language knowledge,
If their is one-to-one 'mapping' between the macro and the non-macro
Python it represents, then maybe the interpreter could, when given the
macro definitions and some code, spit out a version where the macros
are substituted wherever possible; or another version where all macros
are expanded to their core-python statements & expressions.
I guess I'm interested in two types of opinions
1) What do you think - disregarding implimentation issues
2) What do you think as a whole
yet another thought experiment from me -
Cheers, Paddy.
Terry Reedy
"Paddy McCarthy" <padd...@netscape.net> wrote in message
news:2ae25c6b.0305...@posting.google.com...
> I went googling for messages on Python and adding a macro facility,
> but ended up with just too many hits and got lost.
>
> The reason I was looking was that I can get envisage an application
> but seemed to remember that the addition of macros to Python got
shot
> down before but could not remember the reasons.
They are all in the hits you gave up on ;-) (Try just those in 2002.)
One quick summary is that Guido wants Python code to look like Python
code and be more or less universally readable by other Python
programmers. He is not going to be the one to help fork Python into
separate sublanguage communities.
> specific languages such as Specman and Vera. If Python was that
higher
> level language then i thought that a macro facility would allow some
> of the domain specific languages syntax to be re-used , People could
> write
>
> module my_mod (x,y,z) ...
>
> and create a Python class.
This shows the problem already. 'Module' is already in use to refer
to modules, which are different from classes. If a person has such a
misunderstanding, what will they make of sys.modules? -- or of the
word 'module' in all current Python
writings.
You can, of course, use the C preprocessor or equivalent to make such
word-to-work translations before feeding the result to a Python
interpreter.
Terry J. Reedy
A. Lloyd Flanagan
<excerpt>
> and create a Python class. By choosing a careful set of macros the
> resultant {ython-with-macros source could be understandable to people
> with only the domain specific language knowledge,
>
And only those people. The trouble with macros in general, and C
macros in particular, is that they can be used to mangle the language
into almost any form the macro-writer wants. The result is something
that does not make sense to any other person on the planet.
IMHO, we already have plenty of languages that are ideal for writing
unreadable code. Let's not turn Python into one of them.
If you want a language with a powerful designed-in macro facility,
check out Lisp.
Paddy McCarthy
> "Paddy McCarthy" <padd...@netscape.net> wrote in message
> news:2ae25c6b.0305...@posting.google.com...
> > I went googling for messages on Python and adding a macro facility,
> > but ended up with just too many hits and got lost.
> >
> > The reason I was looking was that I can get envisage an application
> > but seemed to remember that the addition of macros to Python got
> shot
> > down before but could not remember the reasons.
>
> They are all in the hits you gave up on ;-) (Try just those in 2002.)
> One quick summary is that Guido wants Python code to look like Python
> code and be more or less universally readable by other Python
> programmers. He is not going to be the one to help fork Python into
> separate sublanguage communities.
>
facility, there is a lot of support for syntax generated by sets of
macros?
I would of thought that language designers/developers would support
the correct implimentation of the macro facility, and that support for
a particular macro set would be like support for Python programs now.
> > specific languages such as Specman and Vera. If Python was that
> higher
> > level language then i thought that a macro facility would allow some
> > of the domain specific languages syntax to be re-used , People could
> > write
> >
> > module my_mod (x,y,z) ...
> >
> > and create a Python class.
>
> This shows the problem already. 'Module' is already in use to refer
> to modules, which are different from classes. If a person has such a
> misunderstanding, what will they make of sys.modules? -- or of the
> word 'module' in all current Python
> writings.
I guess my example was too simple.
If I wrote a library to support testing Verilog code, then the Verilog
programmers have a way representing concepts that may have ( an
extended ), representation in the Python library. I thought that a
Python macro facility might allow the library developer to also supply
macros that helped the Verilog 'bridge the gap' between Verilog and
the Python library.
Such code may be less readable to the 'out=of-domain' Pythoneer, but
*should* be more readable to the Verilog programmer, (or it's of no
use).
Please also remember my idea that the macro feature set should be able
to both expand macros to 'pristine' Python, and also to take Python
source and macro definitions and automatically generate 'coloured'
Python were macros have been applied whenever possible.
>
> You can, of course, use the C preprocessor or equivalent to make such
> word-to-work translations before feeding the result to a Python
> interpreter.
I wouldn't want a file to file preprocessor as I have a gut feeling
that macros as part of the interpreter would be much more powerful - I
just don't know in what ways. (Meta-macros anyone :-)
>
> Terry J. Reedy
Thanks for helping me explore the idea.
Donald 'Paddy' McCarthy.
Terry Reedy
> "Terry Reedy" <tjr...@udel.edu> wrote in message
news:<iUudnYfQA-6...@comcast.com>...
> > "Paddy McCarthy" <padd...@netscape.net> wrote in message
> > news:2ae25c6b.0305...@posting.google.com...
> > > I went googling for messages on Python and adding a macro
facility,
> > > but ended up with just too many hits and got lost.
> > >
> > > The reason I was looking was that I can get envisage an
application
> > > but seemed to remember that the addition of macros to Python got
> > > shot down before but could not remember the reasons.
Yes.
> > They are all in the hits you gave up on ;-) (Try just those in
2002.)
> > One quick summary is that Guido wants Python code to look like
Python
> > code and be more or less universally readable by other Python
> > programmers. He is not going to be the one to help fork Python
into
> > separate sublanguage communities.
My impression is that GvR has opposed non-ASCII/Unicode identifiers,
and certainly non-English 'native language' keywords, both of which
*have* been suggested as additions to standard Python, for much the
same reason.
Please remember that I am acting here as a reporter as much as a
protagonist. You more are less asked for reasons.
> Ahh, are you saying then that with other languages that have such a
> facility, there is a lot of support for syntax generated by sets of
> macros?
Not sure what you are asking but let me try this. The C preprocessor
is used for various reasons. 1) Macros enable generic functions like
max(x,y) (x <= y? x: y). Python functions are generic already. 2)
They enable inline 'functions'. This improves execution speed but
with some readability problems (and possible bugs) when there are
side-effects. 3) They define constants, which again is about speed,
and thereby less a concern for Python. 4) They can be used to write
'C' code that look a lot like, for instance, Pascal. I believe there
were once programming shops that had private Pascal/C coding
standards. But this ghettoized programmers who worked therein.
From what people have said, Lisp macros are even more powerful, for
both use and abuse, but they are ourside my experience.
> > > module my_mod (x,y,z) ...
> > >
> > > and create a Python class.
> >
> > This shows the problem already. 'Module' is already in use to
refer
> > to modules, which are different from classes. If a person has
such a
> > misunderstanding, what will they make of sys.modules? -- or of
the
> > word 'module' in all current Python
> > writings.
> I guess my example was too simple.
I think it was perfect. This falls under category 4 above.
...
> Such code may be less readable to the 'out=of-domain' Pythoneer,
Python is about readability.
> but *should* be more readable to the Verilog programmer,
At the price of being ghettoized. You are free to use domain-specific
names for values, functions, and classes (user-defined types) and call
the result Python. You can also use (and abuse) the metaclass
facility to do some of the things people do with sophisticated macro
facilities (and call the result Python, though with enough 'abuse',
one might be stretching it). You can even take the source and rename
the keywords and do whatever, but then maybe you should call it
PMCPython or VeriPython and take responsibility for supporting the
users. (For all I know, this sort of thing has already been done.)
Once a second set of keywords were allowed, there would soon be a
hundred. Whereas I like Python for having a set small enough that I
can usually remember them.
Terry J. Reedy
Michael Glassford
dialects (http://rebol.com/docs/core23/rebolcore-15.html#sect9), which need
not be implemented as macros at all--certainly not in a way that has all the
drawbacks of C macros.
Mike
Paddy McCarthy
> I think possibly what the OP was looking for was something like REBOL
> dialects (http://rebol.com/docs/core23/rebolcore-15.html#sect9), which need
> not be implemented as macros at all--certainly not in a way that has all the
> drawbacks of C macros.
>
> Mike
>
Firstm If C style macro processors have put you off adding macros to
Python then I can synpathise with you. But does it have to be that
way?
If you have written a Python Package or modules or class hierarchys to
add functionality to Python for other *programmers* to then go and
use, then the access of that extra functionality within Python might
be enhanced if you could modify the syntax used. i.e. providing macros
with the Package, but interpreted please, after-all I would want it
still to be Python :-)
Thanks for the REBOL link, it seems to be an interesting
implimentation - but I have just scaned it once, and I don't wantto
jump too soon into limiting my thoughts with implimentation details,
when at the moment I don't seem to have tickled enough debate beyomd
thoughts of cpp and m4.
Isn't a consideration of macros just like considering, say, functional
programming, or aspect -orientated programming; namely just another
area of language design that should be considered (maybe it is - and
has been soundly rejected - I realy should read all of those google
hits... :-)
Cheerio, Paddy.
David Abrahams
> "Michael Glassford" <glass...@hotmail.com> wrote in message news:<mailman.1053093806...@python.org>...
> Isn't a consideration of macros just like considering, say, functional
> programming, or aspect -orientated programming; namely just another
> area of language design that should be considered (maybe it is - and
> has been soundly rejected - I realy should read all of those google
> hits... :-)
FWIW, at the ACCU conference in Oxford Guido actually said (to a
packed roomful of people) that he was interested in thinking about
ways to support the definition of domain-specific languages in
Python. So the basic goal stands a chance, though I'm unsure what
the answer looks like.
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com
Paddy McCarthy
Now that is great!
Its nice to know the subject is not closed - and at such an important level too.
Cheers, Pad.
Paddy McCarthy
But that *is* the point. I don't want a Lisp or a C or a Rebol macro
facility.
I just have not closed my mind to the thought of macros within Python.
I just haven't thought of a pythonic way of doing it.
Any implimentation I guess would have to include modification and
introspection of the parser so something like help(__statements__) and
help(__operators__) would print the docstrings of statements together
with details on how the statement is parsed, or the associativity etc
of operators (including the built-in ones).
As for producing code that no one can read: Yes! Documentation and
introspection must be a part of a good Python macro system but I think
it is analogous to using a package. Unless you read-up about it, you
don't know what a package does. Python helps by having things like
help(). For any proposed macro solution Python would need appropriate
help and debugging extensions!
Maybe the default behavior when in interactive mode could be for each
macro instance to have its expansion printed when it is encountered.
This default could be then be optionally turned off.
A. Lloyd Flanagan
> introspection must be a part of a good Python macro system but I think
> it is analogous to using a package. Unless you read-up about it, you
> don't know what a package does. Python helps by having things like
> help(). For any proposed macro solution Python would need appropriate
> help and debugging extensions!
>
> Maybe the default behavior when in interactive mode could be for each
> macro instance to have its expansion printed when it is encountered.
> This default could be then be optionally turned off.
Hmmm... interesting. OK, I'll reserve judgement until I see something
more concrete. I think you're definitely approaching it with the
right ideas. Good luck!!
Bengt Richter
It just occurred to me that the tokenizer separates out string tokens
before anything is compiled, so if there were a special kind of string
syntax like raw string syntax but with, say, '$' as a prefix instead of
'r', then that string could be treated specially at tokenization time.
What I have in mind is having the tokenizer recognize a $'xxx' string token,
but not pass it on as a string token in its output stream of tokens.
Instead, the tokenizer would effectively substitute a generated string in place
of the """$'xxx'""" source and proceed to tokenize _that_ instead.
The substitution string would be generated by exec-ing the 'xxx' part in a special
dict, but with a tweaked exec mode that would return the string value of the
last expression evaluated (this is so we don't have to assign to a dummy variable,
or limit ourselves to expressions).
If the last expression was not a string, its repr value would be returned.
So you could write, e.g.,
a = $'b'
and the tokenizer would do the specially tweaked exec on the string 'b' in the
aformentioned special environment (sys.tokenenvs.<filename> ?) [Note 1], such
that the string value of the last expression (in this case just whatever is bound to b)
would be processed as source in the place of the $'b'. If b was not bound to a string,
repr(b) would be substituted. Thus if you wrote
a = $'[2*3]'
the expression [2*3] would be evaluated to get [6] and the repr would be '[6]'
and that string would be substituted in place of the """$'[2*3]'""" source so
that the net input to the tokenizer that had just produced 'a' and '=' tokens
would now see '[6]' and produce the three tokens '[', '6', and ']' (with associated
tupled info)
Because the $-string is exec-ed, it is possible to write stuff with side effects
that could be picked up later. Or, e.g., $"""file("include.py").read()""" would
stream the bytes of include.py right into the tokenizer in place of the $-string source.
Or if you wanted a read-time time stamp in your source, you could write
read_time = $"""import time; `time.ctime()`"""
which would have a result as if the source had been something like
read_time = 'Tue May 27 15:35:19 2003'
[Note 1] Having a potentially pre-loaded dict to exec in would permit persistent side
effects that could be referred to in a subsequent $'some_string'. Having the first $'xxx'
string in a file source.py create a persistent dict as sys.tokenenvs.source by default
and preloading it at least with
{'__filename__': <the path to the source.py currently being tokenized>}
would allow some interesting things.
Extending the syntax so that $name'xxx'' would exec 'xxx' in sys.tokenenvs.name
rather than using the current source file name would allow you to write e.g.,
$project_xxx"""build_id()""" or such.
Of course this would be something to limit for security purposes, but it seems interesting.
The tokenizer would have to be recursively reentrant though, I guess ...
Just HOTTOMH, so fire away ;-)
Regards,
Bengt Richter
Donald 'Paddy' McCarthy
I went back to the Python Language Reference Manual, it may or may not
be used in the current interpreter but my thoughts are along the lines of:
* Introduce a keyword 'macro' that binds a macro definition to a name.
* A bit like a class or function definition, after the macro
definition is made, the interpreter is modified to expand any instances
of the macro found in the manner specified in the macro body.
* The macro object should have a disable method that when called will
stop the interpreter from expanding the macro.
* The scope of the macro definition should NOT be like that of other
names. It should be in force until disabled, or the same name is bound
to a different macro (whereapon that macro is enabled), or from when a
previousely disabled macro is re-enabled.
^ Their should be a way to (hierarchically), group macros together so
that they can be imported/exported/enabled/disabled as a group, or part
of the hierarchical group.
Defining a macro:
-----------------
from __future__ import macros
macro IN_MACRO:
''' - Verilog like input definition
in a,b,c ==expands to==> a = In("a"); b = In("b"); c = In("c")
'''
# New syntax extension rules (productions) use equals sign.
# whitespace on RHS of = below can be any or no white
# -space separators found in source,
input_stament = 'in' __name__ ( ',' __name__ )*
# extend the predefined statements with the new input statement
# Can also be interpreted as stating where to look for the new macro
__statement__ = __statement__ | input_statement
# Save some typing :-)
i = input_statement
# Compute the expansion (I have used := for this )
expand := i[0] ' = In("'i[0]'")' \
( __newline__ i[1][j] ' = In("'i[1][j]'")' )*[ for j in len(i[1])]
# macro definitions return their expansion
return expand
in x, y, zed
IN_MACRO.disable()
in s
Traceback ( File "<interactive input>", line 1
in s
^
SyntaxError: invalid syntax
########################
In the scheme above, all the standard language productions would be
available within a macro definition with the usual double underscore
pre- and suffix, allowing them to be redifiened such as the redefinition
of what constitutes a statement above.
The new productions capture the syntax of the macro, and the expansion
rules show how to expand the captured syntax to a replacement that is
re-scanned by the interpreter (for maybe other macros as well).
I added a doc string :-)
The expansion rule might be too much in the realms of pseudo-code in the
notation given for expanding arbitrarily repeated productions. The [for
j ...] bit being after the (... i[1][j]...
As well as __newline__, __indent__, and __dedent__ could be used to
produce arbitrary indenting in expansions, based, of course, on the
indentation of the original macro
I guess a macro could be setup to be recognised in an expression and
expand to complete an expression and also have statements.
I guess macros could be grouped by allowing macros to either define a
single macro or contain macro definitions (a statement that is just the
name of a pre-defined macro could also add the macro to the group)
macro a:
pass
macro b:
pass
macro c:
a
b
c.disable()
# both a and b are now inactive
c.b.enable()
# equivalent to b.enable() in this case
######################
I have read Bengt's post. (thanks).
His ideas were different from mine - but it did prompt me to write
something more.
Paddy.
Donald 'Paddy' McCarthy
Donald 'Paddy' McCarthy wrote:
> Defining a macro:
> -----------------
Anyone for meta-macro programming ;-)
Paddy McCarthy
> Just HOTTOMH, so fire away ;-)
>
> Regards,
> Bengt Richter
Hi Bengt,
Read your email, and can see that you've carefully thought about some
implimentation issues too.
With you syntax of recognising macros by a $'...' construct I wonder
if you could get a lot of your functionality by adapting one of the
many HTML templating systems so that instead of outputting HTML that
is sent to a browser, the output is python sent to the python
interpreter.
I think Alex has a cookbook entry for templating that has variable
macro recognisers that could probably be adapted.
The only thing is, I don't think this solution provides enough of
Python like documentation and modularisation of macros.
But please, this is just my opinion, and I don't mean to offend, just
air my views.
Thanks for your Idea, Paddy.
Michele Simionato
<snip>
> Anyone for meta-macro programming ;-)
The subject of macros come often in this newgroup. I've got
the impression that it is not difficult to implement them, the
problem is that the developers DON'T WANT them. I think they are
most probably right. Nevertheless, I have just finished an
experimental
project with the purpose of adding a macro facility to Python (it was
a proof of concept for me). This is done via a pure Python module,
called (according to the Principle of Minimal Surprise ;) 'macro.py'.
At the moment, it is still little tested, but it works along the
following
lines:
1. In order to define a macro, the user has to supply a template
function,
i.e. is a function that takes as arguments Python expressions in
string
format and returns as output a string Python expression (or
statement).
In the present implementation I have forced the time-honored
convention
that macros/template functions are all capitals.
An example is the following:
#<example1.py>
def PRINT(*args):
"The arguments are strings; returns a print statement"
return "print "+','.join(['"%s =",%s' % (a,a) for a in args])
#</example1.py>
PRINT works as follows:
>>> from example1 import PRINT
>>> PRINT()
'print '
>>> PRINT('x')
'print "x =",x'
>>> PRINT('x','y') # etc.
'print "x =",x,"y =",y'
2. In order to use the macro (the macro evaluate the output of the
template
function) the user has to write it inside a function or inside a
class;
the function or the class are then explicitly interpreted by using the
command 'expand' in the 'macro' module, which expands the template
function.
This is an example of usage:
#<example1.py>
import macro
def print_x(x):
PRINT(x) # will expand --> print "x=",x
expanded_print_x=macro.expand(print_x)
#</example1.py>
3. At this point, in ``expanded_print_x`` the macro PRINT(x) is
effectively
expanded to ``print "x=",x`` and calling the function gives the result
>>> from example1 import expanded_print_x
>>> expanded_print_x(1)
x = 1
4. Since using macro.expand repetitively can be annoying, I have
provided a metaclass facility. When a class has a ``__metaclass__ =
macro.expand()`` attribute, macros are automagically expanded in all
its methods and attributes, and in its derived classes, too.
#<example2.py>
import macro,example1
class B(object):
__metaclass__=macro.expand()
class C(B): # macros are automagically expanded
def __init__(self,x=None):
example1.PRINT(x) # --> print "x=",x
#</example2.py>
>>> from example2 import C
>>> c=C()
x = None
It works also for macros containing other macros (but
not for recursive macros).
Consider for instance the following ternary operator
#<example3.py>
def IF(cond,e1,e2):
return '(%s and [%s] or [%s])[0]' % (cond,e1,e2)
#</example3.py>
It can be composed with PRINT as in the following script:
#<example3.py>
from example1 import PRINT,macro
def print_ternary():
PRINT(IF(True,1,0)) # --> IF(True,1,0) = 1
print_ternary=macro.expand(print_ternary)
#</example3.py>
Here is how it works:
>>> from example3 import print_ternary
>>> print_ternary()
IF ( True , 1 , 0 ) = 1
I wonder if this approach could satisfy your requirements.
My module is still experimental and not for distribution; as I said,
the principal interest to me was to show that implementing macros IN
SOME WAY in pure Python is indeed possible with a reasonable amount of
effort.
However, implementing them WELL is a completely different story and I
expect
it to be extremely difficult ;)
Cheers,
--
Michele Simionato - Dept. of Physics and Astronomy
210 Allen Hall Pittsburgh PA 15260 U.S.A.
Phone: 001-412-624-9041 Fax: 001-412-624-9163
Home-page: http://www.phyast.pitt.edu/~micheles/