Functions Of Functions Returning Functions

[Lawrence D’Oliveiro]

The less code you have to write, the better. Less code means less
maintenance, and fewer opportunities for bugs. Here is an example of
how I was able to knock a few hundred lines off the size of a Python
module.

When I was writing my Python wrapper for HarfBuzz
<https://github.com/ldo/harfpy>, there were a lot of places where you
could define routines for HarfBuzz to make calls to, to customize the
type-shaping process in various ways.

Of course, HarfBuzz <http://behdad.github.io/harfbuzz/> is a library
written in C++, and it doesn’t know that the callbacks you pass are
actually written in Python. But ctypes
<https://docs.python.org/3/library/ctypes.html> provides an answer to
this: its CFUNCTYPE function lets you wrap Python functions so that
they become callable from C/C++ code.

When I said there were a lot of places for callbacks, I meant a lot of
places. For example, there is a HarfBuzz object called “hb_font_funcs”
<http://behdad.github.io/harfbuzz/harfbuzz-hb-font.html>, which
consists of nothing more than a container for 14 different action
callback routines. Not only that, but each one lets you pass a
separate “user data” pointer to the action callback, along with an
optional “destroy” callback which can do any necessary cleanup of this
data, which will be called when the hb_font_funcs object is disposed.

Imagine having to set all of these up by hand. Each API call to
install a callback would look something like this:

def set_xxx_callback(self, callback_func, user_data, destroy) :
"sets the xxx callback, along with an optional destroy callback" \
" for the user_data. The callback_func should be declared as follows:\n" \
"\n" \
" def callback_func(self, ... xxx-specific args ..., user_data)\n" \
"\n" \
"where self is the FontFuncs instance ... description of xxx-specific args."

def def_wrap_xxx_callback(self, callback_func, user_data)
# generates ctypes wrapper for caller-specified Python function.

@HB.font_get_xxx_func_t
def wrap_xxx_callback(... xxx-specific args ..., c_user_data) :
... convert xxx-specific args from ctypes representation to ...
... higher-level Python representation, pass to callback_func, ...
... along with user_data, then convert any result to ctypes ...
... representation and return as my result ...
#end wrap_xxx_callback

#begin def_wrap_xxx_callback
return \
wrap_xxx_callback
#end def_wrap_xxx_callback

#begin set_xxx_callback
wrap_callback_func = def_wrap_xxx_callback(self, callback_func, user_data)
if destroy != None :
@HB.destroy_func_t
def wrap_destroy(c_user_data) :
destroy(user_data)
#end wrap_destroy
else :
wrap_destroy = None
#end if
# save references to wrapper objects to prevent them prematurely
# disappearing (common ctypes gotcha)
self._wrap_xxx_func = wrap_callback_func
self._wrap_xxx_destroy_func = wrap_destroy
hb.hb_font_funcs_set_xxx_func(self._hbobj, wrap_callback_func, None, wrap_destroy)
#end set_callback

Just think if you had to do this 14 times. And then there are a couple
of other HarfBuzz objects with their own similar collections of
callbacks as well...

Luckily, I figured out a way to cut the amount of code needed for this
by about half. It’s the recognition that the only part that is
different between all these callback-setting calls is the
“def_wrap_xxx_callback” function, together with a few different
attribute names elsewhere. So I encapsulated the common part of all
this setup into the following routine:

def def_callback_wrapper(celf, method_name, docstring, callback_field_name, destroy_field_name, def_wrap_callback_func, hb_proc) :
# Common routine for defining a set-callback method. These all have the same form,
# where the caller specifies
# * the callback function
# * an additional user_data pointer (meaning is up the caller)
# * an optional destroy callback which is passed the user_data pointer
# when the containing object is destroyed.
# The only variation is in the arguments and result type of the callback.

def set_callback(self, callback_func, user_data, destroy) :
# This becomes the actual set-callback method.
wrap_callback_func = def_wrap_callback_func(self, callback_func, user_data)
if destroy != None :
@HB.destroy_func_t
def wrap_destroy(c_user_data) :
destroy(user_data)
#end wrap_destroy
else :
wrap_destroy = None
#end if
setattr(self, callback_field_name, wrap_callback_func)
setattr(self, destroy_field_name, wrap_destroy)
getattr(hb, hb_proc)(self._hbobj, wrap_callback_func, None, wrap_destroy)
#end set_callback

#begin def_callback_wrapper
set_callback.__name__ = method_name
set_callback.__doc__ = docstring
setattr(celf, method_name, set_callback)
#end def_callback_wrapper

Something else that saved even more code was noticing that some
callbacks came in pairs, sharing the same routine types. For example,
the font_h_extents and font_v_extents callbacks had matching types,
they were simply operating along different axes. For both of these, I
could define a common callback-wrapper definer, as follows:

def def_wrap_get_font_extents_func(self, get_font_extents, user_data) :

@HB.font_get_font_extents_func_t
def wrap_get_font_extents(c_font, c_font_data, c_metrics, c_user_data) :
metrics = get_font_extents(self, get_font_data(c_font_data), user_data)
if metrics != None :
c_metrics.ascender = metrics.ascender
c_metrics.descender = metrics.descender
c_metrics.line_gap = metrics.line_gap
#end if
return \
metrics != None
#end wrap_get_font_extents

#begin def_wrap_get_font_extents_func
return \
wrap_get_font_extents
#end def_wrap_get_font_extents_func

and the code for defining all 14 callbacks becomes as simple as

for basename, def_func, protostr, resultstr in \
(
("font_h_extents", def_wrap_get_font_extents_func, "get_font_h_extents(self, font_data, user_data)", " FontExtents or None"),
("font_v_extents", def_wrap_get_font_extents_func, "get_font_v_extents(self, font_data, user_data)", " FontExtents or None"),
... entries for remaining callbacks ...
) \
:
def_callback_wrapper \
(
celf = FontFuncs,
method_name = "set_%s_func" % basename,
docstring =
"sets the %(name)s_func callback, along with an optional destroy"
" callback for the user_data. The callback_func should be declared"
" as follows:\n"
"\n"
" def %(proto)s\n"
"\n"
" where self is the FontFuncs instance and font_data was what was"
" passed to set_font_funcs for the Font, and return a%(result)s."
%
{"name" : basename, "proto" : protostr, "result" : resultstr},
callback_field_name = "_wrap_%s_func" % basename,
destroy_field_name = "_wrap_%s_destroy" % basename,
def_wrap_callback_func = def_func,
hb_proc = "hb_font_funcs_set_%s_func" % basename,
)
#end for

The above loop is executed at the end of creating the basic FontFuncs
class, to fill in the methods for setting the callbacks.

This shows the power of functions as first-class objects. The concept
is older than object orientation, and is often left out of
object-oriented languages. I think Python benefits from the fact that
it had functions before it had classes.

[Steve D'Aprano]

On Sun, 18 Sep 2016 08:28 pm, Lawrence D’Oliveiro wrote:

> This shows the power of functions as first-class objects. The concept
> is older than object orientation, and is often left out of
> object-oriented languages. I think Python benefits from the fact that
> it had functions before it had classes.

You're right about Python having functions first:

steve@runes:~$ python0.9.1
>>> def f():
... pass
...
>>> class A:
Parsing error: file <stdin>, line 1:
class A:
^
Unhandled exception: run-time error: syntax error



However it only gained closures and nested scopes in Python 2.2, or with a
__future__ directive in 2.1.


https://www.python.org/dev/peps/pep-0227/




--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[dieter]

Lawrence D’Oliveiro <lawren...@gmail.com> writes:
> The less code you have to write, the better. Less code means less
> maintenance, and fewer opportunities for bugs.

While I agree with you in general, sometimes less code can be harder
to maintain (when it is more difficult to understand).

Some time ago, we had a (quite heated) discussion here about
a one line signature transform involving a triply nested lambda construction.
It was not difficult for me to understand what the construction did;
however, the original poster found it very difficult to grasp.

Often, functions returning functions are more difficult to understand
than "first order" functions (those returning simple values only) --
at least for people not familiar with higher abstraction levels.

[Lawrence D’Oliveiro]

On Monday, September 19, 2016 at 6:54:31 PM UTC+12, dieter wrote:
> Some time ago, we had a (quite heated) discussion here ...

I have noticed that people get very defensive about things they don’t understand.

> Often, functions returning functions are more difficult to understand
> than "first order" functions (those returning simple values only) --
> at least for people not familiar with higher abstraction levels.

Maybe if the alternative meant writing 200 extra lines of code (as in this case), they might feel differently...