Opencast full rewrite in 2025

[Lukas Kalbertodt]

Sup,

I sure know how to get your attention with a subject line :)


In a meeting yesterday, the Spring Security update came up briefly. And
it got me thinking: is it actually time to rewrite?

I know, rewriting Opencast is a meme [1]. One that has been around for
longer than I've been part of this community. But let's actually have a
proper conversation about this for once. Where everyone can bring their
arguments, experiences and assessments to the table in order for us to
make an informed decision at the end.

But wait! Despite what most of you were probably suspecting, I am _not_
bringing this up to replace Java. This is truly not my point! So let's
please completely set aside the question of programming language, to
narrow the scope of this discussion.

Also note: while I'm leaning toward "yes, rewrite" right now, I don't
want to convince you – I merely want to start a discussion and hear from
all of you.

With that meta stuff out of the way, let me lay out my view of the
situation.


---------------------

# Reasons for

Coming back to Spring Security: the outdated version has been a problem
for many years and no one managed to update it yet, not even Greg! The
amount of time sunk into maintenance work like this is enormous. The
state of the codebase significantly slows down the implementation of new
features and other changes. Opencast often requires _arcane knowledge_
to work with and code in: I have lost count of how many times, in a
meeting, a part of Opencast was mentioned, only for many experienced
devs to chuckle and say "no one understands that, I wouldn't touch it
with a ten foot pole".

Random crashes, unexplained behavior and inconsistent data are common.
There is so much unneeded complexity, so many code smells, edge cases
and bugs. I cannot remember when I last tried an Opencast feature that
works flawlessly, or when I last looked at Opencast code that looked
reasonably fine to me.

As you know, Opencast recently received a large fund in order to
modernize, refactor and generally improve it. We had a couple of
meetings about this and agreed on some good improvements already, but
the coding work has not been started yet.

**I am more and more convinced that the "incremental improvements" route
is actually slower than a full rewrite.** That's the main reason I'm
writing this mail. Yes, rewrites take a ton of time, but the velocity is
way faster. Imagining trying to implement the discussed "data model"
changes gives me shivers. I don't see how this can be merged in less
than half a year, and this is just a first step.

A fresh start means we can learn from our mistakes, get rid of
unnecessary complexity, part with historical baggage, and build a more
robust and maintainable product in the end. This funding is the perfect
opportunity: if not now, it will never happen.


# Learning from mistakes and backward-compatibility

Of course, this requires us to actually learn from our mistakes. Just
starting a rewrite aimlessly will get us nowhere. We need to really
understand the pain points of devs, admins and users of Opencast. And
find out what ultimately causes them and how we can improve. We don't
want to *just* rewrite the code, but also improve on Opencast's data
model and semantics.

But we cannot throw out _everything_ either. We want current users of
Opencast to eventually switch to the rewritten version, so a certain
amount of backwards compatibility is required. We probably should keep
the ingest API for capture agents, for example. This requires careful
consideration, but in the end, there needs to be a _reasonable_
migration path for admins of installations and for developers of
third-party integrations!


# Reasons against

My biggest worry is: I am still too naive and just wrong about a rewrite
being faster than incremental improvements. This is what I'm most
interested in hearing your assessment of and opinion on!

Even if the rewrite is faster, it will take a while, during which we
have two parallel codebases. That means some feature might need to be
developed twice and/or people are discouraged from putting any work into
the old codebase since it will be thrown away at some point. We had a
similar problem with the admin UI before.

I also want to touch on a topic not commonly talked about: a rewrite
means that experienced members of the community have to re-learn several
aspects and some of their knowledge will become useless. It's easy to
dismiss this as irrational, but Opencast is worthless without its
community and disregarding "human problems" like that doesn't get us
anywhere. I personally think that a rewrite is possible without
alienating or splintering parts of the community, but I understand if
others feel differently about this.

Finally, a rewrite is scary: if we run out of money before reaching a
usable state, we have essentially nothing. With incremental
improvements, we can stop at "any time" and have still gained something.
We'd be leaving the safe and familiar shores.


---------------------

Thank you for taking the time to read this. I am truly curious to hear
what you think.

Have a nice Christmas and see you in 2025!


Lukas






[1]: https://github.com/opencast/opencast/pull/822