Extending your shell with "def" and a decorator

[Colin Walters]

I was talking with a coworker recently, and he mentioned that he
thought Hotwire had a number of places where it was kind of "over
design patterened". I think he's right; the Builtin API is one of the
places where this is particularly true. So this evening I spent a few
hours at a coffee shop redoing that API.

Here's how you extended Hotwire before:

class SetenvBuiltin(Builtin):
__doc__ = _("""Set environment variables.""")
def __init__(self):
super(SetenvBuiltin,
self).__init__('setenv',
threaded=True)

def execute(self, context, args, options=[]):
for arg in args:
k,v = arg.split('=', 1)
os.environ[k] = v
return []
BuiltinRegistry.getInstance().register_hotwire(SetenvBuiltin())

Not very pretty. One principle this violates is "Don't repeat
yourself" - you can see we have the term "setenv" repeated 4 times.
The design pattern bits like getInstance() leak through. The choice
of "execute" is rather arbitrary in retrospect, and the necessity of
the "return []" isn't obvious. One other confusion point is that
execute() takes a self parameter, but you should never use it (since
it's not threadsafe)!

With the new API (r1142), here's how you do it:

@builtin_hotwire()
def setenv(context, *keyvalues):
_("""Set environment variables.""")
for arg in keyvalues:
k,v = arg.split('=', 1)
os.environ[k] = v

Pretty cool, eh? You can now define builtins that are really
decorated just Python functions. A fair bit of magic happens under
the hood of @bulitin_hotwire, but I think it's worth it.

I really want Hotwire to be very easy to customize; if you can think
of any arbitrary Python code that will make your life better, Hotwire
should help you run it.

Here's the workflow now for when you have a function, and want to be
able to easily run it from Hotwire:

from hotwire.builtin import builtin_user

@builtin_user()
def myfunc(context):
return 42

Save the above as say ~/myfunc.py.

In Hotwire:
py-eval -f ~/myfunc.py

Now type 'help'. You should see 'myfunc' listed in the "User Builtin
Commands" section. Try running it:

myfunc

Now you can manipulate the results of your function further in any way
you like:

py it+1
(Ctrl-Shift-K)
|grep 4

Going a little bit more complex, add this to the bottom of ~/
myfunc.py:

@builtin_user()
def myls(context):
"""List files in current directory."""
import os
return os.listdir(context.cwd)

Re-evaluate the file:

py-eval -f ~/myfunc.py

Type "help" again and verify that you see myls. Now running it:

myls
|iter|newline|grep foo

For a real-world example of this stuff, here's some custom Hotwire
builtins (just updated to the new API) which I use to administer
Amazon SQS, using the Python boto library:

import os,sys,re

from hotwire.builtin import builtin_user

@builtin_user(options=[['-d', '--delete']])
def sqs_get(context, queuename):
from boto.sqs.connection import SQSConnection
# FIXME hack
execfile(os.path.expanduser('~/.aws'))
conn = SQSConnection()
qconn =
conn.get_query_connection(api_version=SQSConnection.Version20080101)
q = qconn.create_queue(queuename)
msg = q.read()
if '-d' in context.options:
q.delete_message(msg)
return msg

@builtin_user()
def sqs_push(context, queuename, msgtext):
from boto.sqs.connection import SQSConnection
# FIXME hack
execfile(os.path.expanduser('~/.aws'))
conn = SQSConnection()
qconn =
conn.get_query_connection(api_version=SQSConnection.Version20080101)
q = qconn.create_queue(queuename)
q.write(q.new_message(msgtext))

So from Hotwire I can type things like:

sqs-push testqueue4-verbum-org "hello world"
sqs-get testqueue4-verbum-org
py it.get_body()

[quick]

I'm trying to write a builtin_user python function using this
methodology, but I ran into a little trouble. Here's some helpful
information (for folks other than Colin who already knows this), with
a couple of questions at the end.

This function will operate on a list of files. I would like to have
it either operate on file objects provided via a hotwire pipe or else
on filenames explicitly provided to the function. In addition, I'd
like to be able to pass an option to the function.

Here's what I have:

@builtin_user(input=InputStreamSchema(File, optional=True),
output=str,
undoable=True,
idempotent=True,
argspec=MultiArgSpec('path'),
options=[['-a', '--argument']])
def mybuiltin(context, files, options=[]):
if len(files) == 0 and not context.input:
raise ValueError(_("Must specify at least one file"))
fs = Filesystem.getInstance()
fpath = lambda arg: FilePath(arg, context.cwd)
flist = map(fpath, files)
if context.input:
flist.extend(imap(lambda f: f.path, context.input))
have_arg = '-a' in options
for afile in flist:
pass

I modelled this after builtins/rm.py, which seems to do roughly the
same thing although it still uses the old-style declarations.
Trying to use it ran into problems with argument parsing. It turns
out that it should be:

@builtin_user(input=InputStreamSchema(File, optional=True),
output=str,
undoable=True,
idempotent=True,
argspec=MultiArgSpec('path'),
options=[['-a', '--argument']])
def mybuiltin(context, *files):
if len(files) == 0 and not context.input:
raise ValueError(_("Must specify at least one file"))
fs = Filesystem.getInstance()
fpath = lambda arg: FilePath(arg, context.cwd)
flist = map(fpath, files)
if context.input:
flist.extend(imap(lambda f: f.path, context.input))
have_arg = '-a' in options
for afile in flist:
pass


Specifically, options are passed as a member of context and the only
parameter should be *files which (for MultiArgSpec) is a tuple of 0 or
more arguments.

The processing of pipe elements and/or explicit arguments seems like
it would be a common idiom for builtins. Could some more decorator
magic or
context manipulation (or even just a helper function) be done to
simplify this more?

Also, does the decorator builtin work with classes as well as
functions? Explicitly, if I wanted to make my builtin a subclass of
the FileOpBuiltin, how would I go about it using the decorator syntax?