reX-NG from scratch, or collaborate with the Intensity Engine?

[kripken]

Hello reX community,

As you probably already know, realXtend plans to stop using the Second
Life viewer, due to both legal and technical reasons. Instead,
realXtend plans to develop its next-gen viewer - called reX-NG - from
scratch. I have suggested to the realXtend team an alternative to
that: Instead of writing a virtual worlds client/game engine entirely
from scratch, which will take a lot of time and involve non-trivial
risk, I suggested that reX-NG be built upon a project of mine. My
project - the Intensity Engine - already has a working client and
server. It does not suffer from the legal and technical drawbacks of
the Second Life client. While not yet compatible with Second Life, it
already has various advantages, such as a lag-free experience, client-
side physics, clean object oriented development API, and requires
significantly fewer resources to run.

Basically, I am offering realXtend my project - comprised of over
75,000 lines of working code - to form the basis for reX-NG, as well
as my own continuing coding efforts. Collaborating on this is a good
idea, since (1) it would give the community a working version of reX-
NG far faster, and (2) it would focus the efforts of the open source
virtual worlds community, to the benefit of us all. The alternative to
collaboration is that we will have separate projects, with all the
usual duplication of effort and loss of momentum; think KDE/GNOME/
etc., git/bzr/hg/etc., Ogre/CrystalSpace/Irrlicht/etc., etc. This post
is my attempt to promote open source unity.

I approached realXtend with this idea to collaborate almost two months
ago. On both the management and technical side, the people I've spoken
to at realXtend seem thus far to show little interest. With their
agreement I'm posing the issue to the community here.

Why joining forces is a good idea

realXtend currently intends to write a client entirely from scratch.
Virtual worlds clients are very large and extremely complicated
software projects, typically taking years to write, even for talented
teams (look how long it took other projects). They also involve
significant risk, in the sense that the timeline is unpredictable -
components may need to be rewritten, even with the best planning.
Consequently, quite a lot of time may pass before the realXtend
community receives reX-NG, if reX-NG is to be written from scratch.

I agree with realXtend's decision to stop using the Second Life
viewer. It is both technically problematic - tends to crash, hard to
extend - and legally complicated - GPL, with copyrights held by a
competitor. The Intensity Engine, while not compatible with Second
Life yet, has none of those drawbacks: The great majority of the code
is stable and extensible (see details below), and has no legal issues.
Given that, I think that adding Second Life compatibility to the
Intensity Engine is a far better idea than writing a client entirely
from scratch. The same is true for the various features being
discussed on realxtend-arch at the moment: It is faster and safer to
add advanced capabilities (say, voice) to a working client rather than
to write a client entirely from scratch.

I would also argue that the Intensity Engine server should be used
instead of OpenSim, because that would give additional benefits, but
this can be a separate issue.

History and Background

Last spring I began to work on a project combining elements of both
virtual worlds and games. I considered using realXtend as the platform
for my project, as I share many if not all of realXtend's overall
goals: open source, interoperable worlds, etc. Some of you here might
remember me from the work I did for realXtend last summer, in order to
get the viewer to run on Linux. I did that in order to see if
realXtend could be cross-platform, which was (and is) a requirement of
mine.

A little while after that, it became clear that realXtend did not
intend to focus on the issues most pressing to me, in particular
dealing with lag, reducing the high resource requirements, and making
realXtend truly cross-platform. So, I considered my options. Upon
reflection, my main problems were in essence fundamental issues with
both the Second Life client and the OpenSim server, namely:

- Licensing: GPL viewer. Code and programmers can't be shared between
server and client, stalling development.
- Stability: The Second Life viewer is quite prone to crashes.
- Resource usage: Very high, on both client and server.
- Network/physics model: Physics is done on the server, leading to
lag, and making many games impossible.
- Development model: The Second Life scripting approach is deficient
in various fundamental ways.

Since writing a client+server from scratch is a daunting task, I
evaluated existing alternative open source code. After much research
and testing, I decided to build upon the open source game engine Cube
2, a.k.a Sauerbraten. I have been working on this since last summer,
and the result is the Intensity Engine.

Meanwhile I heard that realXtend has come to similar conclusions as I
did last summer, regarding the viewer at least. When I saw that
realXtend was planning to write a client entirely from scratch, I
decided to offer my project to realXtend, with the results mentioned
above.

Note: A word about the Second Life client and OpenSim. While issues
with the Second Life client and OpenSim led me to start my own
project, that does not mean I do not respect and admire the work that
has gone into those two projects. Both are impressive in their own
way. They just do not provide what *I* would like in a virtual worlds
solution. Instead of complaining, I have gone and worked on a virtual
worlds solution of my own. So, please do not see this message as
flamebait, it isn't. And do not see it as mere complaints without code
(the kind open source developers hate to get), while I do state
detriments in Second Life and OpenSim, I have actually worked to solve
those issues. If I believed that I could solve them by contributing
code to the upstream projects, I would have. But the simple fact is
that a separate project was needed (as also realXtend has come to
realize, again, so far regarding the client).

Intensity Engine Overview

http://www.intensityengine.com/

There are some screenshots and videos on the screenshots page, and you
can download the source code or binaries. Note that the examples are
all done using non-professional artwork (I am a coder, not an artist),
so don't expect to be impressed by the artwork. For more aesthetically
appealing stuff, you can look at http://sauerbraten.sourceforge.net/screenshots.html
- these are Sauerbraten screenshots, which has much of the same
underlying code (and in particular the same rendering engine, at least
for now).

The Intensity Engine works and can already be used to create
applications/worlds. It's still in alpha, so there are bugs, and a few
features remain to be developed. I currently plan a stable 1.0 release
for this summer, assuming I continue separately from realXtend. (If,
as I hope happens, realXtend agrees to collaborate on this, then the
timeline would naturally be otherwise as the goals would change.)

Here is a list of already working features (not a wishlist!):

* Lag-free. Physics can be run on both server and client; this is
easily customizable. Currently it uses a first person shooter-type
model which leads to a pretty much lag-free experience. The network
protocol is also built for speed and efficiency. As proof of its
capabilities, you can play Sauerbraten, a great game which as
mentioned above is also built using Cube 2 ( http://sauerbraten.org/
).

* Quite a lot of the code (networking, physics, actions/events, etc.)
is shared between the client and server, which is very beneficial for
development. In particular, there is no need to write stuff twice.
Another advantage is that there is no limitation on which programmers
can look at what, which is the current sad situation with OpenSim (see
below for more comments on this).

* CEGUI is used for the GUI, with the ability to use Lua for
scripting. I replaced the original Cube 2 in-house GUI with CEGUI.
This was very easy and other GUIs would also be simple to add.

* A unique (cube-based - stemming primarily from the Cube 2 engine)
world geometry system, which leads to very efficient rendering,
physics and editing (which is done in-world and in realtime, in that
respect like Second Life, but the details are different). You can also
place arbitrary meshes in the world. Animated characters are currently
done with the MD5 (Doom 3) model format, which includes skeletal
animation and attachments, but see below regarding Ogre.

* Already cross-platform, runs on both Windows and Linux, both server
and client, natively - no Wine. OS X should be easy to add, Cube 2
works on it flawlessly, but I don't own an Apple machine so I can't
test that. We currently build using SCons on Windows and Linux, with
VC2008 on Windows and GCC on Linux.

* Currently uses the Cube 2 rendering engine, which is OpenGL/SDL
based. However, I have already gotten a proof of concept working with
Ogre instead (in particular, most of the hard parts have been
completed, like using the existing system of static lightmaps/shadows;
there are some screenshots on the development blog if you want to take
a look). So this should not be a problem. Combined with using CEGUI,
this means that 100% Ogre rendering - which would allow complete
DirectX output, and so forth - is within reach. Existing realXtend
code that uses Ogre (the avatar library, etc.) should be able to be
integrated at that point.

* Currently uses a specialized physics system, which is simple but
efficient and robust (and even includes some nice stuff like ragdoll
physics, etc.). It would be possible to move to a more general-purpose
physics engine, in fact a proof of concept with Bullet has already
been done by other people in the Cube 2 community. Furthermore the
code is written in a way that moving to a pluggable physics system
will not be hard.

* A custom entity system is used, based on modern OO principles.

Perhaps the main issue that remains to be done, from the perspective
of Second Life users, is Second Life compatibility. This will be added
to the Intensity Engine; the timing depends on whether my proposal to
collaborate with realXtend is accepted or not.

Intensity Engine Code Overview

The Intensity Engine is comprised of a core of some 25,000 lines of
code written entirely by me, and which I can license however I want
(so I can relicense it however realXtend wants, if we collaborate on
this). In addition there are some 50,000 lines of code that originate
from Cube 2, which I have modified and relicensed (the original
license was zlib, which allows such relicensing). Required libraries
at this point are CEGUI, SDL, Python, Storm, and some smaller ones
(ENet, Boost.Python, Lua, libpng, libjpeg, zlib).

Both client and server are written mostly in C++ (in particular this
makes it significantly lighter on the server side than OpenSim).
Applications/scripts are written in Python (C++/Python integration is
via Boost.Python).

Status

As I said above, I approached realXtend with this idea almost two
months ago. Given that the initial response was unfavorable, I've been
exploring other options for my code meanwhile. At the moment I am in
the process of forming a venture around the Intensity Engine, called
Syntensity. Bottom line, for now the Intensity Engine code is still
mine, so collaboration of any kind is possible with realXtend, but
fairly soon the code will probably be the property of the startup.

Summary and Conclusion

I've tried here to present in an open and transparent manner the case
for collaboration between realXtend and my project, in particular, for
basing reX-NG on the Intensity Engine. By working together on this, I
believe that reX-NG can be finished far faster and with far less risk
along the way. If we collaborate, the end result will be a better
product, available sooner for the realXtend community to benefit from.

Opinions are welcome. Aside from talking here, I'm also available on
IRC (#realxtend or #intensityengine).