extempore and opencv

[Simon Stapleton]

It's been - a while - since I last used extempore. Life got in the way. For about 12 years.

And I sudenly find myself with a new performance project, something that's been in the back of my mind for about 25 years, a follow on from a series of juggling pieces I did 1997-1999, all done with super-8, and went down pretty well.

The basic conceit was to juggle as a team with myself, and explored ideas of the hyper-reality of super-saturated film color versus a somewhat more shabby reality, the boundaries between reality and image and the physical boundary of the screen. I was very happy with it as it stood. However, the performance itself was, not to put too fine a point on it, bloody hard, tied to a very rigidly scheduled 24 frames per second and an image to music sync that was somewhat approximate. Even a more "conventional" juggling performance locked to a piece of music is rigid, but has the possibility of, in the case of an error or a drop, simply skipping part of the routine and eventually sliding more or less seamlessly back into the piece's timeline. When the performance also has to sync to a projected image that continues no matter what, options become very limited, and you have to be really concetrating - after all, most of the time, the action you're supposed to be working in sync with, is happening behind you, and blinded by the projector, you can't see it.

But I somewhat digress. I find myself with a "need" to upgrade that perfromance to the brave new digital world. To go from flicker to glitch. We have technology available which was nothing short of science fiction those 25 years ago, and these open some very interesting possibilities. Without going into too many boring details, I am thinking multiple modular projection screen objects on stage, tracked via opencv, and a video projector used to project parts of potentially multiple video clips onto those screens simultaneous with live performance happening around them; the creation of virtual spaces around those screens, and the like. And generative music.

Being able to track objects on stage means being able to sequence parts of the performance. I'm not sure, but I might even be able to detect failures on my part ("drops", as they are known in juggling parlance) and integrate them into the performance. At the least, I should be able to remove most, although probably not all, of the overall timing stress, making the whole damn thing more fun to do.

So, overall, this seems like a win-win-win situation, with more possibilities to explore, and a less stressful result. But it requires being able to script opencv, to create a sequencing engine, for any of this to work. Now, I could do that in python, but hell, I loathe python, even if I could probably leverage zarf's "boodler" to do a generative soundscape type thing as well. But extempore is nice to code for, and handles timing properly, so it's back to extempore for Simon.

Which is where my question comes in. I run OSX, and have an even more severe allergic reaction to windows than I do to python. And looking at the git repo, it seems the opencv stuff only currently works for windows, and with a statically linked opencv to boot. I'm guessing the former is down to nobody bothering to make an OSX compatible CMakelists.txt - I can fix that - but I was wondering if there is any particular reason why opencv needs to be statically linked other than "it was easiest to do that way"?

Anyway, cheers all in advance for any ideas you might have.

Simon

[Ben Swift]

Hey Simon

You shouldn't need to link against OpenCV at build time, all the opencv examples use the usual Extempore "bind-lib" dynamic linking approach.

However, here's an update I posted recently to this list:

Nice to hear from you.  In the interests of not leaving you hanging, here's an update from me. I was on parental leave from March 2023 until June 2024, so things have been quiet (on the work front at least) in that time.

Extempore is still my weapon of choice for my own livecoded musicmaking (I did an extempore set at ICLC this year) but on the development side of things Extempore's a bit... perhaps "dormant" is the right word. The music/audio stuff still works fine, but some of the graphics stuff might be trickier, depending on your platform. On macOS there's the added complication that Extempore still requires Rosetta to run (again, it works fine under Rosetta).

None of those things are unfixable (far from it!) but they're also not five-minute fixes, and neither Andrew or I are able to devote much time to making said changes right now. So there's just no timeline on it. But if Extempore is working well for you (or others on the list) then great - it still is for me too :)

In particular, Extempore doesn't build cleanly on aarch64 (which I presume you're on). I've just been using a binary from the releases page which runs under Rosetta.

You're more than welcome to use it, and if you do manage to fix things in ways which will be helpful to others, then PRs would be gratefully received. But I just wanted to clue you in before you invested a bunch of time on stuff - fiddling will _definitely_ be involved.

Ben

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/extemporelang/2080AAA1-F0AD-425D-A505-30FD8A82F8A1%40gmail.com.

[Simon]

Thanks for that Ben.

I did the stupidest / simplest thing possible, i.e just mangled a CMakefile together and hoped for the best.  And whaddya know, it kinda works.

I'm running on Intel rather than aarch, which makes things a bit easier, but now I'm butting up against permissions issues - if I want to use the camera I need to have "NSCameraUsageDescription" in my info.plist, and as extempore is a command line binary I'm not sure how to do that - apparently it's do-able to embed that in the binary via XCode but I have no idea how to do it with CMake. I'll look into it and get back with my results.

Frankly, I'm not enjoying the newer OSX releases very much, they're so screwed down security wise they're pretty much a pain in the hole to use, let alone actually program for.  Dammit. 

[Simon]

And some more feedback.  I really don't want to get into platform bashing here, but it feel like I might, after 37 years of running exclusively Apple operating systems, have to jump ship to something like linux or one of the BSDs.  It's doing my damn head in.  Managed to get an info.plist embedded in the binary, with the NSCameraUsageDescription stuff set.  A bit of a pain to do, but not too hard.  Try running, everything works, until I try to access the camera.  blam.  you need entry com.apple.security.device.camera in Info.plist.  OK, add that, relink.  Same error. Try again.  Same error.  Try only that entry.  Same error.  Do some more googling.  Find obscure discussion on apple's support site.  Turns out, the error message is utterly misleading.  What it means is that I need that entry in "Entitlements.plist" , as well as the original one in Info.plist, and then I need to sign the binary using a specially crafted and paid for developer key..

I understand the need for  - well, actually, scratch that, I don't.  I hate that I have to sign a piece of software, then sod about putting arcane crap in obscure xml files, than when I try and run it, interact with some entirely different window to "allow" it to access parts of my machine or my files on my hard disk. If I wanted to bugger about with xml configuration files I'd be programming in Java.  The people who designed this were smoking crack.

Anyway, if I run extempore from a terminal (and inherit terminal's entitlements, thus making all the above moot, because that makes absolute sense) and access it using extempore-mode rather than doing extempore-repl, I can get access to the camera, as evidenced by the little light going on, but I get no sound, and the video capture window doesn't open OH MY GOD I'M GOING MAD.

None of this is a problem with extempore, of course.

I think I'll go have a little lie down now.

[cas tix]

O.o
Sorry I'm speechless.