Embedding Erjang in a CEP Engine

[Darach Ennis]

Hi guys,

 

Here's some slides from a presentation to the Erlang User Group in London yesterday

demonstrating how to embed erjang. Erjang + erlang behaviors is a nice integration ploy

for leveraging erlang within Java VM based solutions. In this context it was integrated into

a CEP Engine (StreamBase), but hopefully it's generally useful:

 

http://www.slideshare.net/darach/streambase-embedded-erjang-erlang-user-group-london-20th-april-2011

 

Cheers,

 

Darach.

[Jetztgradnet]

Hi Darach,

how did you handle shutting down Erjang in your application? It calls
System.exit() somewhere in the shutdown code and I need it to
"behave".

Regards,

Wolfgang

[Darach Ennis]

Hi Wolfgang,

I was lucky in the IO case and the shutdown case. I followed Kresten's recipe for embedding

Erjang here: https://github.com/trifork/erjang/commit/8221f25ce2f082f405b4c06f13260893cd16e489

which doesn't include a best practice for a controlled shutdown sequence. I elected to let shutdown

hang wet. The shutdown issue was a non-issue in my case as only have a single instance of the erjang

runtime.

I avoided IO issues as I don't use System IO at all. My interface or contract with erjang and erlang modules

are a set of behaviours where no IO is expected or implied, in other words the behaviours are assumed side-effect

free. As I control the integration context and define the behaviours that erlang (erjang) modules are bound to I have

all the control I need. In my case the (implied) protocol across the erjang/erlang and integration boundary and the

behaviours that enshrine them were sufficient.

I think you've put a lot more thought into lifecycle management. What's common about our use cases

given your previous email is that we're both using or attempting to use erjang in a containerized environment

where erjang is being embedded. In both our application contexts we would like to maintain a session with

erjang. In my case a lighter weight application controlled scheduler and an alternative to erjang.RPC would be

optimal, but mine is a niche use case and my thoughts aren't yet fully formed and are at least half baked at

this time!

If you're at the Erlang Factory in London next week let's hook up. Lifecycle management is something that

most folk using erjang in an embedded way will want. I'd imagine most of us deploy into a containerized environment

of some description so control of and hooks into lifecycle events are a shared concern and challenge for

embedded erjang.

Cheers,

Darach.

[Jetztgradnet]

Hi Darach,

thanks for your detailed answer!

> I followed Kresten's recipe for embedding Erjang here:https://github.com/trifork/erjang/commit/8221f25ce2f082f405b4c06f1326...

That's what I did as well.

> I elected to let shutdown hang wet. The shutdown issue was a non-issue in my case as only have a
> single instance of the erjang runtime.

OK, so I understand you simply keep your Erjang istance around
"forever" without shutting it down during runtime of your app.

> I avoided IO issues as I don't use System IO at all.

I would like to have a universal erjang wrapper, which allows IO as
well, so I need to channel any IO to/through a configurable channel,
which can, e.g. forwarded into a HTTP response.

> My interface or contract with erjang and erlang modules
> are a set of behaviours where no IO is expected or implied, in other words
> the behaviours are assumed side-effect free.
> As I control the integration context and define the behaviours that
> erlang (erjang) modules are bound to I have all the control I need.

OK, that's probably the best approach for a self-contained app.

> In my case a lighter weight application controlled scheduler and an
> alternative to erjang.RPC would be optimal, but mine is a niche use
> case and my thoughts aren't yet fully formed and are at least half baked at
> this time!

(Why) do you consider erjang.RPC heavy weight? I plan to use this as
well to allow control from the Java side and maybe bridge web
requests. Also I would like to create a bridge into the OSGi world,
which would also require RPC for Java-Erlang interaction. What might
make it heavy weight is the (de)marshalling of Java objects in/from
Erlang terms.

> If you're at the Erlang Factory in London next week let's hook up.

Would be interesing, but this is unfortunately no option for me. I'm
currently looking into Erlang/Erjang as a hobby project, so there is
neither time nor budget to go to a Erlang conference. But I would like
to continue this discussion in this forum.

> Lifecycle management is something that most folk using erjang in an embedded way will want. I'd imagine most of us deploy into a containerized environment
> of some description so control of and hooks into lifecycle events are a shared concern and challenge for embedded erjang.

Yes, that's why I'm looking into to biggest limits, which is currently
the call to System.exit() at shutdown and other issues for containers,
which support reloading, i.e. Servlet containers and OSGi
environments.

Again, thanks for your thoughts!

Wolfgang

[Darach Ennis]

Hi Wolfgang,

Cost is relative I guess. If you're thinking on the order of 15K (interpreted) or 25K (compiled) calls per second

then erjang.RPC is absolutely fine. I suspect for 99% of use cases that is more than a pleasent sufficiency. 

I'm in the other 1% camp. On my laptop I'm measuring ~42 microseconds for an RPC call (including term

marshalling overhead) for some simple erlang function calls. But, I work in low and ultra low latency environments.

In the general case 42us is fabulous. If the latency budget on your applications critical path is 100us though then it

represents a considerable overhead! So it's all about cost/value tradeoff.

Although, as erjang is already benchmarking faster than plain old erlang... I'm hopeful!!! :)

Cheers,

Darach.

[Kresten Krab Thorup]

Hi folks,

If one of you want to write up the "host interface"

interface ErjangHost {
InputStream getInput();
OutputStream getOutput();
OutputStream getErr();
void system_exit();
...
}

Then I'd be glad to help you find the places in the code where it needs to be used. Perhaps there is more than the above (system properties, initial arguments, etc.)

Out of the box, Erjang can only run one Erlang VM per JVM. But if you wrap it up in a separate class loader then you could run more (I did that in my Ruby VM).

Kresten

[Jetztgradnet]

Hi Kresten,

> If one of you want to write up the "host interface"
>
> interface ErjangHost {
> InputStream getInput();
> OutputStream getOutput();
> OutputStream getErr();
> void system_exit();
> ...
> }

yes, thanks, that's exactly what I'm looking for. Although I would
reverse the order of invocation, i.e. call set{Input|Output|Error}
Stream(), etc. to be used by Erjang.

Why is a system_exit() necessary? I would expect that after calling
ERT#shutdown(), that OTP shuts down and then the Kilim Scheduler along
with all created thread pools, no?

> Then I'd be glad to help you find the places in the code where it needs to be used. Perhaps there is more than the above (system properties, initial arguments, etc.)

Anyway, I would volunteer to make the required changes, if you could
give me some pointers.

> Out of the box, Erjang can only run one Erlang VM per JVM. But if you wrap it up in a separate class loader then you could run more (I did that in my Ruby VM).

Running multiple Erjang runtimes is not (yet) on my horizon, but at
least in OSGi, it might be possible, e.g. to run multiple versions, as
each bundle has its own class loader. We'll see about that later,
first I would like to get Erjang to properly shut down itself without
taking the servlet container with it :-)

Regards,

Wolfgang

[Kresten Krab Thorup]

The file

erjang.driver.FDDriverInstance

contains the mapping from file descriptor (1,2,3) to corresponding io stream. That's the logic that is ultimately reached by erlang:port_open( ... [{fd, 1}] ...) which is how erlang reads from stdin.

The logic in there is not very good because System.in is not async, and so there needs to be a thread that reads and passes things on to erjang as a message. If you have a better environment (Java7/NIO2) then we could make those async too; or if you have a way to feed the I/O to erjang piecemeal rather than on a blocking InputStream.

I can see I already did some of the generic work, look for ERT.set_stdio ... perhaps you should try to make that work.

As for shutting everything down, ... you should be able to just issue an init:stop() and that should do it (which is what ERT#shutdown does). erlang:halt() is supposed to be abrupt ... but perhaps you can make erjang.m.erlang.ErlProc#halt which is the erlang:halt BIF just do ERT.shutdown when running in embedded mode? Or perhaps there is an option in Kilim to shut it down, ... I'm not sure. Don't have time to investigate that right now. We might need some extra logic to clean up async threads...

Kresten

[Jetztgradnet]

> Cost is relative I guess. If you're thinking on the order of 15K
> (interpreted) or 25K (compiled) calls per second
> then erjang.RPC is absolutely fine. I suspect for 99% of use cases that is
> more than a pleasent sufficiency.
>
> I'm in the other 1% camp. On my laptop I'm measuring ~42 microseconds for an
> RPC call (including term
> marshalling overhead) for some simple erlang function calls. But, I work in
> low and ultra low latency environments.
> In the general case 42us is fabulous. If the latency budget on your
> applications critical path is 100us though then it
> represents a considerable overhead! So it's all about cost/value tradeoff.

OK, that's quite some requirement. So good luck at reaching your
target latency!

Wolfgang