Gauging interest adding function annotations to Lua's locking API

[Brent Pappas]

Hi Lua community,

Recently I stumbled upon Lua's lua_lock() and lua_unlock() macros and noticed that while they are used fairly frequently in Lua's source code (just grepping for "lua_lock" yields 80 uses, minus the definition site), there is no way to tell which of Lua's internal functions acquire a lock (e.g, lua_getfield()) or release a lock (e.g., finishrawget()) without looking at the definitions of the functions themselves.

This seems like it would impose a cognitive burden on new and existing Lua devs, since they would essentially need to read the definition of every one of Lua's internal functions before calling it to know whether the function acquires or releases a lock, or expects the caller to hold a lock when calling it.

Would there be any interest in adding function annotations to Lua for specifying which internal functions acquire and release a lock, or expect the caller to hold a lock? The annotations could be defined like this:

// Annotation for functions that only call lua_lock()

#define lua_acquires(L)

// Annotation for functions that only call lua_unlock()

#define lua_releases(L)

// Annotation for functions that expect their caller to have locked L beforehand

#define lua_holds(L)

I'm leaving the macro definitions empty here because the purpose of the macro would be just to signal to developers which functions acquire/release/hold locks, not to add any new functionality to the functions themselves. I suppose one could define them to some sort C attribute though instead.

The macros would be invoked like this to annotate, for example, the lua_getfield() and finishrawget() functions from earlier:

LUA_API int (lua_getfield) (lua_State *L, int idx, const char *k) lua_acquires(L);

static int finishrawget (lua_State *L, int tag) lua_releases(L) { ...

This tells a developer that lua_getfield() acquires a lock and finishrawget() releases a lock just by looking at their declarations. Devs using modern tools would be able to see this information even more easily, since many IDEs and LSPs these days can show a function's declaration if one just hovers their cursor/mouse over the function's name.

Another benefit of these annotations would be that they would make it possible to develop a custom-tailored static analysis for Lua's locking API to ensure that locks are being acquired and released appropriately. This would be akin to the RCU-checking features that the sparse tool provides for the Linux kernel's RCU API.

I'm currently working on a static analysis tool with features for checking RCU API usage and would be willing to extend it to checking usage of the Lua locking API if there is any interest.

I'm curious what the Lua team thinks of this idea and would appreciate their feedback (maybe the locking API is not significant enough to warrant this attention?).

[Roberto Ierusalimschy]

> Recently I stumbled upon Lua's lua_lock() and lua_unlock() macros

> <https://github.com/lua/lua/blob/c1dc08e8e8e22af9902a6341b4a9a9a7811954cc/lapi.h#L29-L36>

> and noticed that while they are used fairly frequently in Lua's source code
> (just grepping for "lua_lock" yields 80 uses, minus the definition site),
> there is no way to tell which of Lua's internal functions acquire a lock
> (e.g, lua_getfield()

> <https://github.com/lua/lua/blob/c1dc08e8e8e22af9902a6341b4a9a9a7811954cc/lapi.c#L716C1-L716C66>)

> or release a lock (e.g., finishrawget()

> <https://github.com/lua/lua/blob/c1dc08e8e8e22af9902a6341b4a9a9a7811954cc/lapi.c#L739>)

> without looking at the definitions of the functions themselves.
> This seems like it would impose a cognitive burden on new and existing Lua
> devs, since they would essentially need to read the definition of every one
> of Lua's internal functions before calling it to know whether the function
> acquires or releases a lock, or expects the caller to hold a lock when
> calling it.

No internal function ever aquires the lock, but all API functions aquire
the lock before calling any other function. Therefore, all Lua internal
functions can assume the interpreter is "locked". In your specific
examples, lua_getfield is an API function. (See the annotation LUA_API.)
finishrawget is a kind of exception: It is static function used only to
finish some API functions, so it may be considered part of the API. (As
it is static, it is easy to check that all its uses are in tail calls
from API functions.)


-- Roberto

[Brent Pappas]

> No internal function ever aquires the lock, but all API functions aquire

> the lock before calling any other function.

Ah I see, so if one sees the LUA_API macro on a function signature they can assume that the function calls lua_lock() if necessary.

I see a few functions annotated with LUA_API that don't call lua_lock() (e.g., lua_absindex(), lua_gettop(), lua_rawset()); I'm assuming these functions don't need the lock for some reason then?

> finishrawget is a kind of exception

Ok, so I suppose auxgetstr() and auxsetstr() are examples of the same sort of exception since they only call rcu_unlock() as well?

I guess another exception would be lua_error(), which doesn't need to unlock (according to a comment near its return statement).

Thank you for your insight.