Dynamically loaded extensions

[Garrett Serack]

Howdy folks,

So I did something odd yesterday.

I was thinking about extensions and SciTE, and I wanted to build an
extension as a DLL that could be loaded at run-time.

I wandered back thru the mailing list before I did anything, and it
appeared that the director interface was the only real option for
loading extensibility at runtime.... but, I really didn't want it out
of process, and I wanted to be able to use OnKey (and others) from the
Extension interface that Director didn't expose (that would have been
pretty slow :D)

At first I just hacked in a couple of methods to the DirectorExtension
to dynamically load a DLL, and rather than using WM_COPYDATA, I just
had it look for a couple of functions in the DLL to hook it up.

That gave me a DLL-based 'director' ... now, that's not precisely what
I wanted, but pretty cool and easy nonetheless.

Last night, in a fit of sleeplessness I went and refactored what I
did, removed my changes to DirectorExtension, and created a new class
(I called "DynamicExtension"), which could do the same thing, but
didn't have the unnecssary stuff that DirectorExtension did.

I made a couple of small, non obtrusive changes to Extension and
MultiplexExtension (just to allow me to determine if a particular
dynamic extension is already loaded) and hacked in a property to load
it--(more q's about that in a bit tho')

The DLL need only expose four functions:
SetMessageFunction -- needed to have SciTE give a callback to the
extension.

ExtensionExecS -- a function that Scite Calls with a two strings
(message/data)

ExtensionExecI32 -- a function that Scite Calls with a string and a
32bit int (message/data)

ExtensionExecI64 -- a function that Scite Calls with a string and
a 64bit int.(message/data)

I originally just had the one that took two strings, but I felt that
some of the messages would be wastefully slow if the data had to be
encoded as a string. (and actually, it started out as a single string,
encoded exactly the same as Director :D)

As it is, it's still nearly exactly the same as the way Director
works, except it doesn't have to do additional encoding of the
message. (same messages)

Now, I can write my extension as a DLL with a very simple binary
interface, and it's pretty damn version proof--an extension doesn't
have to be compiled against any particular version of SciTE (well, it
does have to support my DynamicExtension!)--and it supports the entire
ExtensionAPI.

I didn't look at what it would take in the Linux side yet, but I
haven't done anything resembling rocket-science... the delta to sync
that won't be very large I imagine.

I wrote alot of this when I should have been sleeping, so I want to go
thru my code before I post it... probably by the weekend.

Questions
========

1. I guess first, is : Does anyone care?

2. In order to load an extension, right now I hacked in code for a
property "inprocextension=whatever.dll" ... my code supports loading
multiple extensions (using the MultiplexExtension), but ... I didn't
think of a way to express that in the configuration. What would be the
best way to implement that?

3. I only glanced at the Lua extension code, and I think I'd like to
steal the ability to grant my extensions the ability to use arbitrary
functions/procs like you can from Lua... anything I should know before
I do that?

Followup
========
So the reason I did this, is because I have written some code in C#
that can host an Active Scripting host in a few lines of code, and I
would like to be able to use VBScript/JScript to automate/extend
ScITE.

At first glance, you might be inclined to tell me that I can't create
a DLL with C bindings from C#--but I've got a secret weapon. I've got
a utility that will take a .NET assembly and fix it so that I can
expose arbitrary methods as C exports, with whatever calling
convention I use.

With this trivial extension interface, I wrote a C# class that
implements it, which can then do anything I want in .NET, without
having to do anything to SciTE at all to load it. As far as SciTE is
concerned, it's just a DLL that has the right exports.

And now I stand at a crossroads. Already, I've got my proof-of-concept
working where I'm loading my C# extension, and it's working
beautifully. I'm 2 or 3 hours away from having my VBScript/JScript
extension functional too.

Ideally, I'd like to see this type of thing included in the default
SciTE build, simply so that I can continue to use stock SciTE builds
with my own custom extensions. I'm more than happy (once I've
published it) to make the appropriate modifications (and implement up
the Linux side too) to get it committed.

Garrett

[Luis Rascão]

Integrating C# assemblies in scite is definitely an advantage, mainly
due to how easy it is to implement anything (Win32 API, argh). One
question though, how secret is this secret weapon of yours?
LR

[Garrett Serack]

LOL!

Well, I'll publish the details on that too.

Really, it's not "secret" as much as not-widely-known.

I've been meaning to blog about it for quite some time.

I'll make sure I write out the details about that--it's pretty darn
useful in a number of situations.

G

> > Garrett- Hide quoted text -
>
> - Show quoted text -

[Neil Hodgson]

Garrett Serack:

> The DLL need only expose four functions:

I'm not a big believer in custom DLL extension interfaces since
that was what COM (and similar) was supposed to enable in a better
way. OTOH if there are people that really want this, I'll include it.

> 2. In order to load an extension, right now I hacked in code for a
> property "inprocextension=whatever.dll" ... my code supports loading
> multiple extensions (using the MultiplexExtension), but ... I didn't
> think of a way to express that in the configuration. What would be the
> best way to implement that?

Normal .properties convention is space separated lists (although I
just noticed that openpath uses ; / : ).

Neil

[Luis Rascão]

Cool, i'll keep an eye out for that blog entry then
LR

[Garrett Serack]

Heh-heh...

> I'm not a big believer in custom DLL extension interfaces

I'm not a big fan either--I always say 90+% of the time, if you need
to make a new interface... you're probably missing something.

Which, is why before I even started I went back thru the mailing list
to see how DLL extension for SciTE are made, just so I didn't have to
modify SciTE itself.

Once I found that there wasn't... I looked at the DirectorExtension,
but it had some drawbacks:
- Out of proc.... not a big fan of that, and because it's out of
proc,
- didn't support synchronous calls, which I wanted
- it also was likely to be underperforming for some low-latency,
high volume events (OnKey)
- I wasn't thrilled about all the string encoding/decoding.
- I think one of the things I'm doing might want access to the
Scintilla buffer.

So, I started with simply adding as little as possible to the Director
Extension, figuring I would try to not invent anything not absoluetly
neccesary, but it was clear that it was still going to sub-optimal.

> .. COM (and similar) was supposed to enable in a better way.

LOL... I guess I'm glad that you never decided to add a COM interface
to ScITE.. [shudder]...

That all being said, maybe I need to think this through once more...

*thinks hard*

Hang on here... maybe I *can* make it a bit simpler, and less
"original"

Perhaps I'll drop the string message parameters alltogether, and
instead, use a single function which has a more familiar
declaration...

ExtensionMsgProc( UINT iMessage, WPARAM wParam, LPARAM lParam )

If I do that, I'll just simply use the (prexisting) appopriate Message
identifiers for all the events. well... at that point i might as well
also add a lower-level hook into SciTEWin::WndProc, so that the
extension can handle/consume what ever it likes.

That would drop all the rest of the unnecessary string handling, and
the "unhandled-message-happy-path" would be really really tight--the
extension can do a switch on the message and return nearly instantly.

Hmm... I can drop the DynamicExtension's handling/replaying of messges
that I'd get from hooking into WndProc, and keep only the functions
that I won't get at the appropriate time by handling SC_NOTIFICATION:

Initialise,Finalise, OnMacro(?), OnExecute, OnSwitchFile,
ActivateBuffer,
InitBuffer, Load, Close, RemoveBuffer, OnOpen, OnBeforeSave,
OnSave, Clear

...

This would make it no harder--hell, probably even *easier* to
implement an extension DLL, and pretty much removes all the
limitations.

I think I have some more codin' to do!

thanks,

G