Status Updates
Esko Luontola
just done. Give a short status report after each day of developing the
program, that what did you complete that day and what will you be doing
next. So basically this is a daily stand-up meeting
(http://martinfowler.com/articles/itsNotJustStandingUp.html)
accommodated for open source projects (instead of a traditional morning
face-to-face meeting, this is in the evenings on a mailing list).
Here is the first status update as an example. Please always include a
link to the commits in question.
I just pushed the following commits:
http://github.com/orfjackal/dimdwarf/compare/2594511...ada9bc9
This includes the infrastructure for registering "services" and
"controllers" using Guice modules. It's not yet fully integrated - the
new code is not being used by the end-to-end tests - so next I'll be
working on updating ServerStarter to use the new module system. There
are also some additional tests which I need to write in
InstallingServicesSpec.
The idea is that each "service" runs in its own thread, where it reads
messages from a message queue and processes them one-by-one in FIFO
order. One of the services is the ControllerHub class, which contains
"controllers". Each controller is connected to 0 or 1 services, and
nobody else has access to that service - all access goes through the
service's controller. The ControllerHub is a special service, in that
anybody can send messages to it (by asking for injection of a @Hub
MessageSender<Object>), and it will delegate processing the message to
every controller (in unspecified order). The controllers should process
everything quickly and without any blocking I/O, because that would
easily block the whole server. Any slow operations should be done in the
services.
Examples of how the service modules look like (in this first version)
are RelayModule and ControllerModule. They need to call some methods in
the ServiceModule class to register a service or controller, and also
provide a ServiceRegistration or ControllerRegistration instance which
will be used to bootstrap the services. The ServiceModule is actually a
PrivateModule
(http://google-guice.googlecode.com/svn/trunk/javadoc/com/google/inject/PrivateModule.html),
so all bindings in the module are by default not seen by other modules,
but ServiceModule will expose the controller and service endpoints so
that the server can use them. Using them by accident from another module
or another thread will not be possible.
--
Esko Luontola
www.orfjackal.net
Esko Luontola
their own threads:
http://github.com/orfjackal/dimdwarf/compare/43a8ed9b...33a991a3
I also wrote authenticator and network modules which use the new
ServiceModule system, but I'm having problems with Guice sometimes non-
deterministically injecting the wrong dependencies, so they are in the
ctrl-bug branch instead of master. I've posted a question on the Guice
mailing list and I'll try to figure out later that what is the
problem:
http://groups.google.com/group/google-guice/browse_thread/thread/5f61266829554993#
Esko Luontola
I managed to fix the bug which I mentioned last night. It had to do
with one controller constructor-injecting the concrete class of
another controller, in which case an implicit binding to the
controller would be created inside the current private module. Having
direct dependencies between controllers is something that should be
allowed (and disallowing it by convention would be risky - dependency
checking via Guice SPI would be needed), so the solution was to expose
the controller classes from the service modules.
Now the authenticator and network have their own modules
(AuthenticatorModule and NetworkModule). This made ServerStarter very
simple - maybe it can be removed soon. There is still the issue of
passing parameters to the services from the command line. Now
ServerStarter is using a dirty hack ("public static int port"), so
that's the next one I'll refactor.
I also refactored ServiceModule to minimize the boilerplate in service
modules. To create a service, extend ServiceModule, hard-code the
service name as constructor parameter, call the bind* utility methods
in the ServiceModule class and provide a ServiceRunnable
implementation. Most of the services should do fine with
ServiceMessageLoop, but the ControllerHub requires some special
handling, so I did not yet figure out a good way to remove that
duplication. Finally the module needs to be passed as a parameter to
ServiceInstallerModule when creating the Injector.
Esko Luontola
Passing the port parameter to the network service happens now cleanly.
I decided that the server will be configured by setting the parameters
in the modules of the services which will be affected by that
configuration. That will make them private to the service.
I also inlined some of the startup code, until it's better seen what
it should be like. The module configuration might be later extracted
to some other class. One idea is that the applications which run on
the server would be responsible for at least some of the configuration
(i.e. they provide a top-level aggregation module).
http://github.com/orfjackal/dimdwarf/compare/fdf045c27f...60018d43
I wrote some unit tests to the network service. Earlier it was covered
only by the end-to-end tests (which are slow). I extracted a utility
class DeterministicMessageQueues for unit testing services.
Next I'll begin working on
ClientConnectionTest.login_and_logout_successfully()
Esko Luontola
Now the system checks the username and password which are provided by
the client. It is done by a CredentialsChecker running in the same
thread as AuthenticatorService, so blocking operations are OK (for
example connecting to an external database). The CredentialsChecker
implementation which does this needs to be provided by the application
(see EchoApp). That means that now has also been implemented the class
loading of the application.
ClientConnectionTest.login_and_logout_successfully() does not yet
pass, because the system does not support logging out. I'll work on
that next. After that I'll implement session messages, and after that
probably support for more than one concurrent client. :)
Esko Luontola
I renamed services to actors, because that's what is the common name
for the concept (http://en.wikipedia.org/wiki/Actor_model). So now the
three main component types of the architecture are actors, controllers
and tasks.
http://github.com/orfjackal/dimdwarf/compare/e757613b03...ad2d2203
I implemented logging out. Now the end-to-end test, where a client
logs in and then logs out, passes.
Next I will do some clean up of the NetworkSpec class. It needs a
utility class to help asynchronous tests. Now if the client does not
receive a message (or the message is too short), the test will
deadlock. A timeout is needed.
After that I will implement support for more than one client. I should
be able to do that already before implementing session messages.
http://blog.orfjackal.net/2010/09/lets-code-dimdwarf.html
http://www.orfjackal.net/lets-code
I've started making screencasts of me coding. The first topic is
Dimdwarf. :) The first episode of the series explains the architecture
of the system, so that should be useful even as documentation. I'll be
releasing the second episode soon. Episodes 2-4 will cover the changes
that I mentioned in this post (except for the renaming of services to
actors - that would anyways have been boring to watch).
Jussi Mononen
> http://github.com/orfjackal/dimdwarf/commit/e757613b03d869b40ffdcc5236a0ed24a7561b28
>
> I renamed services to actors, because that's what is the common name
> for the concept (http://en.wikipedia.org/wiki/Actor_model). So now the
> three main component types of the architecture are actors, controllers
> and tasks.
Everytime someone mentions Actors and Scala I just start to think about
Akka framework (http://akkasource.org/).
Would love to see if it could be of any use in DimDwarf since that
project is amzaingly interesting.
And apologies for being silent for a long time, as a family guy I don't
seem to have as much free time as I would like to. I'll try to come up
with something as soon as possible. Maybe the tiny proof-of-concept game
using DimDwaf or smth else. We'll see!
--
- Agile Poodle
- http://www.jussimononen.info/
- http://www.twitter.com/agilepoodle
Esko Luontola
> Everytime someone mentions Actors and Scala I just start to think about
> Akka framework (http://akkasource.org/).
>
> Would love to see if it could be of any use in DimDwarf since that
> project is amzaingly interesting.
I decided not use Scala or Akka actors in Dimdwarf, because I wanted to
minimize the complexity and understand thoroughly what is happening
inside the system (and be able to modify it freely). Also to make the
system as deterministic as possible, I wanted to enforce that actors
process messages in FIFO order - with Scala actors it's possible to
process selectively only some of the messages.
And anyways, it was not much work to create the minimal actor library
which Dimdwarf has. Mostly it's just these classes which do all the work:
net.orfjackal.dimdwarf.actors.ActorStarter
net.orfjackal.dimdwarf.actors.ActorRunner
net.orfjackal.dimdwarf.actors.ActorMessageLoop
net.orfjackal.dimdwarf.mq.MessageQueue
Creating the ActorModule which configures the actors using Guice was
much harder than the code for starting and running the actors.
> And apologies for being silent for a long time, as a family guy I don't
> seem to have as much free time as I would like to. I'll try to come up
> with something as soon as possible. Maybe the tiny proof-of-concept game
> using DimDwaf or smth else. We'll see!
No worries. I'm also occupied by many other things, so it'll take some
time until Dimdwarf is in a shape where it can run applications. Of
course applications for Dimdwarf can be developed before it, by mocking
Dimdwarf's application APIs.
--
Esko Luontola
www.orfjackal.net
Esko Luontola
I'm creating some utility classes for making it easier to write
asynchronous tests (it can be seen starting from Let's Code Dimdwarf
#5 http://www.orfjackal.net/lets-code). It's still work in progress.
I'll continue working on that, until it is generic enough to be used
in the rest of the application. Then I'll be able to refactor the
NetworkSpec and some other test classes to use it. Also NetworkSpec
needs to be split so that there will be two test classes: one focusing
on the Apache MINA integration and another focusing on the login/
logout logic.
Apache MINA 2.0.0 was just released, so I upgraded from 2.0.0-RC1 to
it.
Esko Luontola
I upgraded to Hamcrest 1.2 and Maven 3. I solved some warnings that
Maven 3 was making due to relying on deprecated features, and moved
some common plugin configurations to the parent POM's pluginManagement
element.
http://github.com/orfjackal/dimdwarf/compare/c6e3f403...6b1e2eddf3
Work on the asynchronous testing tool continues. I created a class for
working with byte streams (ByteSink), after which I extracted the
common parts with the previous work (EventSink), resulting in a
hopefully reusable base class (AbstractSink). This work will show up
in Let's Code Dimdwarf episodes 9-12 (http://www.orfjackal.net/lets-
code). Next I'll work on the exception messages of that tool, after
which I'll start using it in the tests (which might result in more
modifications).
Esko Luontola
The asynchronous testing tool is now just about done. Any further work
on it depends on discovering new use cases.
I split NetworkSpec into two test classes, LoginLogoutSpec and
NetworkActorSpec. Now they are more cohesive and testing the server
logic is easier (only NetworkActor's tests are asynchronous).
Next I will implement support for multiple clients. It will drive the
design to implement a way to identify clients, client session
management, and maybe also some way to monitor the internal state of
the server.
After that I'll start working on session messages, which will drive
the design for at least running tasks and committing transactions. I
think that session messages won't need persistence, so that
persistence will be then the following topic.
When those are done, then the current architecture change will be
complete and I'll be able to delete a bunch of old classes. Around
that time I will also change the name of this project to something
easier to pronounce and more descriptive than "Dimdwarf". My current
favorite is OneShard (I've already reserved the oneshard.org domain).
What do you think about that name? Do you have other suggestions?
Esko Luontola
I implemented support for multiple clients. This work will show up in
Let's Code Dimdwarf #22-30: http://www.orfjackal.net/lets-code
The NetworkController identifies the clients based on a SessionHandle
instance from the NetworkActor and only the NetworkActor knows how to
unwrap the SessionHandle and get access to the underlying network
session. Also only NetworkController should be aware of the
SessionHandle - later I'll implement a domain specific SessionId which
the other server modules (and application code) can use to identify
clients. NetworkController shall be responsible for the conversions
between SessionId and SessionHandle.
Next I'll refactor and write some tests for the application loading
code. Now it's covered only by end-to-end tests. After that I'll start
working on session messages.
Esko Luontola
I started using type parameters for MessageQueue and also made the
Actor trait type parameterized. This should make Guice configuration
errors more visible.
When I was working on commit
https://github.com/orfjackal/dimdwarf/commit/f2e8ada478d926e0adf4710b477b11cf79944e06
of the previous updates, I at first forgot to copy the @Hub annotation
and wrote "toHub: MessageSender[Any]" instead of "@Hub toHub:
MessageSender[Any]". The unit tests passed fine, but the end-to-end
tests caused weird behaviour - the NetworkActor was processing
messages which it itself sent - and it took many minutes to figure out
the reason. That should not happen anymore. When every MessageQueue is
parameterized with an Actor specific type, a similar mistake will
produce a Guice configuration error message.
Esko Luontola
I went crazy with generics-reflection and eliminated nearly all
boilerplate from the ActorModule implementations. Now the message type
is a type parameter for ActorModule and everything else is automatic.
A minimal ActorModule needs to declare only the controller and actor
classes and the name of the actor.
Hmm... Actually the name could be figured out automatically...
*restarts his IDE*
https://github.com/orfjackal/dimdwarf/commit/83e6dadb48841b7bd3532580a07e858c167fcf85
Tadaah! Now also the name is automagic. :)
Esko Luontola
I'm now writing unit tests for some parts which were only covered by
end-to-end tests. I wrote a test that Guice gives to errors about the
DI configuration, and next I'll write tests for loading applications.
Before I could do that, I had to move test utilities to their own
module, so that I could access them both from the core module and end-
to-end test module.
My principle is that as much as possible to covered by unit tests,
because (1) that will give the fastest feedback and confidence when
refactoring, and (2) it puts more design pressure on the code - it
requires the code to be more decoupled. The main classes should have
only a couple of lines of trivial code which is covered only by end-to-
end tests.
Esko Luontola
Renamed AbstractThreadContext to BaseContext. This was to avoid
confusion with the ThreadContext class which contains the ThreadLocal.
I tried to think of a better name for ThreadContext, because it's not
a Context implementation, but did not come up with any.
https://github.com/orfjackal/dimdwarf/compare/32d27ee6...3b09045c29
I wrote some unit tests for application loading. Making it testable
required extracting it to its own class. Earlier this was covered only
by end-to-end tests and was one of the few remaining things from the
original walking skeleton which did not yet have unit test coverage.
These changes will show up in Let's Code Dimdwarf #35-41 at
http://www.orfjackal.net/lets-code
Next I will write some proper command line argument parsing, to get
rid of the last hack from the original walking skeleton. I will
probably use https://args4j.dev.java.net/ - it seems simpler to use
than Commons CLI.
After that I think the most critical cleanups have been done, and I
will continue towards the next feature: sending and receiving session
messages.