Something like Java OSGi in Go language?
philip
I want something like Java OSGi in the Go Language.
Alternatively, as a workaround, I guess I could make a Go program
which can compile and launch a new Go program for each "module"?
They could communicate together? What do you think? is that as close
as I could get to OSGi?
Philip
Rob Heittman
philip
For my desires, I only want the ability to load in new code at
runtime. OSGi in Java gives me that.
So my idea was that the Go programs would have to be different
processes running and communicating, so that I could stop and start
different processes and this would allow the code to be compiled and
updated. This is like service oriented architecture where the
processes are services and can be taken down or re-started, but the
disadvantage is parallelism, multiple processes could be bad against
the CPU due to context switching. Go provides good parallelism, but I
guess only within the one process which is running.
Taking down and entire monolithic application written in Go at runtime
to recompile and re-deploy seems ... bad, hence I would rather spilt
it up into processes. The problem here is that when codes is updated,
how do we re-deploy without upsetting the environment? To keep a high-
uptime? I'm sure Google is interested in that?
As for your other thoughts about modularity, isolation, and
organization, I don't need them, its nice to have them of course.
However I do need to have high-uptime and be able to re-deploy re-
compiled code without disturbing customers.
Thanks, Philip
Eleanor McHugh
> This is a really interesting question, but one that's quite orthogonal to Go as I see it. Nevertheless, as a heavy consumer of OSGi, I share it.
>
> I think some of it depends on what OSGi means to you. I'm reminded of those HSBC ads in airports where they have one picture repeated 3 times with different connotations. For me, I use OSGi in Java to get better modularity, isolation, and organization. In theory I should be using it to provide lifecycle support as well, but few existing bodies of Java code are designed with dynamic hot start/stop behavior in mind, so, not so much. But then I want to distribute my services across the cloud, and OSGi doesn't address multiple collaborating processes on different nodes, so something like Paremus is needed to spread across multiple JVMs ... and the complexity of the simple OSGi design starts to ratchet up rapidly.
>
> Since OSGi is a specific refinement atop the Java classloader. Since Go eschews the class hierarchy style of Java, and "Object" itself is much more loosely defined in Go, the whole architecture of OSGi doesn't really port apples-to-apples. I think it's important to re-examine the question of what you're modularizing, isolating, or organizing.
>
> I've taken on board the many admonitions against reinventing Java or C in Go, and so I have lately been trying to think about modular application design in a Go way. One approach that works quite well for me so far -- that you mentioned as a "workaround" -- is that I can neatly achieve {modularity,isolation,organization} with multiple federated Go processes interacting via RPC. The downside, of course, is relying on RPC vs. IPC, but in my application domains the necessary communication between modules is discrete enough for this to not be a real drawback.
>
> On the front of the OSGi benefits I listed above, though, have you looked at Akka, which uses an actor-based model to raise the level of service abstraction away from something closely coupled to the JVM classloader? The general architecture of Akka (if not the specific implementation) *does* port somewhat directly to a different-thinking model like Go. I suspect that Go code, bridged via REST or JSON-RPC APIs, could federate nicely among Scala and Java code that uses an architecture like that, something which I hope to try out soon.
>
> As Go is being targeted as a systems language, there isn't much happening at a super high level of abstraction. But -- and seriously, I've only been working in Go for like, a month -- there is great potential to do so. The plumbing is all there and I'm already having good success with small experiments replacing C, C++, C# and Java misfits with Go code. I think it's only a matter of time before someone tries to do something at an application-framework level with Go, despite that being far from the language's stated objectives.
>
> I'm thinking of Eleanor McHugh and Michael Hoisie here ... their work has been pushing Go into realms usually associated with high-level languages, and I've already gotten some mileage out of each.
I'm glad GoLightly's proving useful to someone :)
One of the ideas I'm toying with for a future stable version of the library is compilation of Go to a virtual machine implemented in GoLightly, allowing hot-loading of multiple Go applications within a single process space and replacing conventional IPC with channels. Something similar to OSGi could then be built atop this, with the added benefit that code could be moved fairly easily between processes and/or hosts by supporting operations from the pi-calculus extensions to CSP - an area I definitely want to explore as part of making GoLightly cloud-friendly.
In the meantime it should be relatively simple for Philip to implement a poor-man's OSGi equivalent using the traditional unix process model and monitoring/control via pipes. I recommend Beej's Guide to IPC as a great introduction to the subject.
Ellie
Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
raise ArgumentError unless @reality.responds_to? :reason
Eleanor McHugh
> Taking down and entire monolithic application written in Go at runtime
> to recompile and re-deploy seems ... bad, hence I would rather spilt
> it up into processes. The problem here is that when codes is updated,
> how do we re-deploy without upsetting the environment? To keep a high-
> uptime?
I used to build multiprocess realtime control software and the customary approach for that is to use multiple hosts with a shared watchdog to bring up a new instance immediately when the current active instance terminates. For software upgrades this means upgrading the secondary systems first and then switching one of them in so that the primary can be upgraded, in the process hopefully avoiding any loss of traffic.
On a single host I'd suggest using two or three control processes linked by a named pipe to act as the watchdog. With a hardware watchdog the swap would effectively be instantaneous, but even with a software watchdog you should be able to get the switchover down to a small multiple of the cost to context switch between processes. If possible use the native kernel event interface to monitor the watchdog for changes although old-fashioned polling will work if necessary.
If you were writing in C I'd suggest running multiple processes rather than faffing about with pThreads, but with Go you should be fine spinning off tasks in goroutines and communicating via channels: the marginal cost of launching a goroutine is negligible and I've had more than 900k running concurrently in 2GB or RAM so their base memory consumption is also very acceptable for massively parallel worker architectures. This approach means you've also only got the cost of tearing down a couple of processes when you need to load a new code instance, and likewise when a failover occurs the total costs of recovery are about as minimal as they're going to get.
Rob Heittman
I'm glad GoLightly's proving useful to someone :)
One of the ideas I'm toying with for a future stable version of the library is compilation of Go to a virtual machine implemented in GoLightly, allowing hot-loading of multiple Go applications within a single process space and replacing conventional IPC with channels. Something similar to OSGi could then be built atop this, with the added benefit that code could be moved fairly easily between processes and/or hosts by supporting operations from the pi-calculus extensions to CSP - an area I definitely want to explore as part of making GoLightly cloud-friendly.
If you were writing in C I'd suggest running multiple processes rather than faffing about with pThreads, but with Go you should be fine spinning off tasks in goroutines and communicating via channels: the marginal cost of launching a goroutine is negligible and I've had more than 900k running concurrently in 2GB or RAM so their base memory consumption is also very acceptable for massively parallel worker architectures. This approach means you've also only got the cost of tearing down a couple of processes when you need to load a new code instance, and likewise when a failover occurs the total costs of recovery are about as minimal as they're going to get.
Eleanor McHugh
> That is exactly what I was imagining; again here I almost want to avoid saying "OSGi" out loud, because it's a dramatically different (and I think much better) solution to the same problem set. Join this putative model with a expressive dynlang (e.g. RubyGoLightly) to provide runtime configuration and direction, and you've got something that looks not at all like any existing application server, but does all the same tricks and is operable at the same level of abstraction. Yet at a couple of orders of magnitude better efficiency, with a lot more flexibility. So exciting!
Well even when it's mature and tightly optimised I doubt RubyGoLightly will be even an order of magnitude faster than standard Ruby for most day-to-day uses. Contrary to popular perception Ruby isn't slow and responds very well to exactly the same multiprocess patterns as any other language that integrates with its host environment (at least when that environment's Unix), and the main speedups I hope to realise in RubyGoLightly are mostly to do with the implicit parallelism in a lot of standard library operations such as Enumerators.
However I do agree the prospect is exciting. Sufficiently so that I ditched commercial work at the end of last year to focus exclusively on GoLightly and related projects >8o
> Loading a new bundle in OSGi is brutally expensive compared to the cost of spawning a new process; there's just a lot of work to be done to open and examine the bundle, construct its classloader and connect up its plumbing. A typical process running a Go program would be up and actually talking over IPC before OSGi got done reading the jar headers. Hm. I feel a little demo project and some slides coming on. This is probably better shown than told.
And another Go addict is born ;)
André Moraes
philip
communicate across processes. Processes could start and stop and
communicate, programs could be built from source and started and
stopped and talk, or talk to a central service bus, SOA.
