Update: rewrite and support slowdowns

[Jason Priem]

Hi all,

Here’s a quick progress update: we’re still working on our migration to Databricks. It’s codenamed “Walden” because it will “simplify, simplify” our infrastructure in ways that’ll let us diagnose and repair data quality issues 100x faster than now.

We know that we’ve been a bit slow on customer support lately and I apologize for that. OpenAlex is tightly integrated into Unpaywall and that infrastructure has accumulated so much technical debt over the last decade that it's became real tough to maintain—every little brick is a load-bearing wall and every little fix risks breaking two other things.

This is why we’re doubling down on the rewrite, so that we can get back to delivering the great support experience we’re really committed to. The downside is that support will continue to be a bit slow (and we certainly won’t be shipping features) till that’s done.

I don’t want to commit to any granular milestones because The Full Rewrite is a notoriously unruly beast, but we will be done this quarter with Unpaywall, and next quarter with OpenAlex.

We’re proud that so many folks rely on OpenAlex data, and we’re committed to keeping it accurate, comprehensive, and responsive. Thanks for your support and patience while we fix things up.

Best, Jason

PS If you’d like to hear more about our Q2 plans, check out the video of the town hall we did last week.