[Sbcl-help] callback guidance

[Hemanth Kapila]

Hi,

Thanks for the awesome compiler. Kudos to you all.

Could someone provide some guidance on the right way to do foreign callbacks?

I have some c/c++ code that needs to call back into lisp. I tried
using cffi, and things seem to work fine so far (I am on windows. I
have not tested on linux. Nor did I try multi threading yet).

Infact, I see that callback support in sbcl is better than ccl in some
sense (for instance, cffi:defcallback need not be top level).
Based on old messages on this forum[3], it seems callbacks have been
in use for a while.

However, the sbcl manual[0] seems to discourage foreign callback,
saying, "calling c from lisp is sometimes possible but is extremely
hackish and poorly supported as of sbcl 0.7.5".
The cffi manual[1] seems to indicate that not all platforms of sbcl
support callbacks.
Also, there is this bug 310198[2] says "Support for them already
exists, and is utilized by CFFI. but the interface should be polished
and exported from SB-ALIEN" but also refers to some 1% deadlock.

Can someone kindly help me understand the situation with callbacks?
Is it a significant performance penalty?
And are there any examples to look at on the good practices to follow
while writing a callback? Do I need to have defcallback only at the
top level? Any help here would be greatly appreciated.
The manual suggests us to refer to funcall0..funcall3 but I could not
really make out from the relevant file. (iiuc, it is
src/runtime/funcall.c )

Many thanks,
Hemanth
PS: I would be very happy to pitch in with fixing or testing or fixing
the documentation. But, am afraid, my knowledge of the compiler/cl is
iffy and might need help with where to look for info.


[0]http://www.sbcl.org/manual/#Foreign-Function-Interface, section
9.7.4 , Calling Lisp From C:
[1] https://common-lisp.net/project/cffi/manual/html_node/Implementation-Support.html
[2] https://bugs.launchpad.net/sbcl/+bug/310198.
[3] https://sourceforge.net/p/sbcl/mailman/message/28438971/


_______________________________________________
Sbcl-help mailing list
Sbcl...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

[ded7...@yandex.ru]

>However, the sbcl manual[0] seems to discourage foreign callback,
>saying, "calling c from lisp is sometimes possible but is extremely
>hackish and poorly supported as of sbcl 0.7.5".

0.7.5 sounds long ago…

>The cffi manual[1] seems to indicate that not all platforms of sbcl
>support callbacks.

I would indeed suppose that the callback support has some differences
across platforms.

>Can someone kindly help me understand the situation with callbacks?
>Is it a significant performance penalty?
>And are there any examples to look at on the good practices to follow
>while writing a callback? Do I need to have defcallback only at the

In case of SBCL/Linux I am using callbacks from FUSE. This seems to run
fine for days on the end. This is a purely user-side perspective, of
course.

One thing I would recommend to remember about is that with callbacks you
have much more diverse set of «interesting» effects. For normal FFI you
mainly care about memory ownership/management; for callbacks the code
you interact with can also have interesting ideas about thread
management, global error/signal handling, etc.

With FUSE, for example, I can use FUSE message parsing code and let it
call back to my handlers, but I cannot let FUSE event loop manage the
message fetching, because then it has ideas about thread management and
then SBCL GC crashes because its expectations are broken…

[Douglas Katzman via Sbcl-help]

Infact, I see that callback support in sbcl is better than ccl in some
sense (for instance, cffi:defcallback need not be top level).

Stylistically it's probably not the best to put a defcallback at a nop-toplevel position.

Its expansion includes (at least) a DEFUN plus a DEFPARAMETER, and everything about them is global, so it would sort of surprise me if it worked.

I suppose you could capture a lexical variable if that's the intent, but to have the forms not be executed because they're inside an IF or something even hairier - I'm not sure why you would want that.

 

Also, there is this bug 310198[2] says "Support for them already
exists, and is utilized by CFFI. but the interface should be polished
and exported from SB-ALIEN" but also refers to some 1% deadlock.

I can't remember for certain what the deadlock was, and I wrote that comment. 

But I kind of feel as if it would have been fixed by https://sourceforge.net/p/sbcl/sbcl/ci/ad9fa52a6228e69194cd252130ec4381f0d9bc75

Which is to say, I definitely saw a deadlock occur, and it was definitely because of the issue cited in that change.

As to whether foreign threads make it better, worse, or neither; that's the part I don't know. Sorry about that.

 

Is it a significant performance penalty?

It would probably be a horrible idea to put a call from C to Lisp in a hot loop.

To give a picture of just how bad a C-> lisp call is, it entails:

- potentially an mmap() of about 4 MiB (depending on whether there was already some memory in a pool reserved for foreign threads)

- for non-x86, a memset() of 2 MiB of memory "whether you need it or not"

- probably about 3 or 4 calls to pthread_sigmask to block and unblock signals

- probably at least one pthread mutex acquire/release, plus a few Lisp mutex acquire/release

- establishing about three CATCHes, some unwind protects, and doing something like 10 special variable bindings

And return from Lisp to C entails doing the opposite of everything just mentioned.

To think that a call across a function boundary is equivalent to all that is sort of mind-boggling when put in perspective with a normal call.

You would most definitely not want to write a lisp callback to just compute the sum of two small integers.

It's best to have the lisp code do a ton of work, to make it worthwhile to have used a callback at all.

 

The manual suggests us to refer to funcall0..funcall3 but I could not
really make out from the relevant file. (iiuc, it is
src/runtime/funcall.c )

You shouldn't have to use funcallN. The lisp callback appears in C like a plain old function pointer.

Doug

[Hemanth Kapila]

Wow.
Thanks a lot for the info.

That helps a lot.
I will try and redesign my library to avoid calling back into lisp in
a hot path.

Thanks @ded7...@yandex.ru for the user side perspective.

much appreciated,
Hemanth

[Stas Boukarev]

Callbacks from a lisp thread shouldn't involve that much work, but
they still spill the registers onto the stack, at least because of
having to call pthread_getspecific for the current thread (which could
be avoided).

[Douglas Katzman via Sbcl-help]

Ah, right, I was thinking in my mind that the callback was from a non-lisp thread as I wrote my response.

Lisp -> C -> Lisp is not bad. Can we eliminate the pthread_getspecific on win32 ? It's removed everywhere except macOS and win32, afaict.

[Stas Boukarev]

Seems to be using FS/GS, so should be doable. Darwin has something similar.
But beyond that, there's still a lot of work needed. A special entry
point would need to be invented, so that it can be called directly
with the C registers, and not going through an array.

[Stas Boukarev]

Yes, that's more reasonable, but there's still noticeable overhead
(for the foreseeable future).

On Mon, Jul 27, 2020 at 9:45 PM Hemanth Kapila <saihe...@gmail.com> wrote:
>
> Oh thanks,
>
> So my scenario is Lisp->C->Lisp with no threads involved on the c side.
> So, that is a reasonable use?
>
> I should try to get some perf numbers perhaps.
>
> thanks,
> Hemanth

> --
> I drink I am thunk.