Following up on yesterday's API outage

[Jason Priem]

Hi everyone,

I want to follow up on yesterday's API outage with more context on what happened and what we're doing about it.

First, an apology

The API was down for about six hours yesterday morning, which is a long time, especially for teams who depend on OpenAlex for production systems, demos, and customer work. I didn’t do a good job of communicating on this list how seriously we take that problem.

The reality is, we spent the whole day in a grueling, front-to-back system analysis to make sure this never happens again—comms took a back seat. We take this outage very seriously, and we’re committed to learning from it and improving.

What we’re changing

Here are the concrete steps we’ve implemented since the outage:

1. We shipped a status page: You can now check https://status.openalex.org anytime to see if the API is up. It's automatically monitored and updated every few minutes.
2. We doubled our API capacity: we went from about 160 concurrent web server processes to 380. This gives us much more headroom to handle traffic spikes. This is more expensive but needs to be done to cover our continuing growth.
3. We shipped several performance improvements in code, to make sure we make better use of the compute capacity we have. We're continuing this work over the next few days.
4. We're setting up automated tests that run after every deploy to catch issues before they affect users.
5. We’re temporarily limiting anonymous API access to 10,000 request per day, per IP address. This only affects about 0.1% of users. (On February 13th, as announced earlier, you’ll just get 100 credits per day of anonymous use.)

Rate limit change

We’re also going to be tightening the limits on free accounts, even those with an API key, starting on February 13th. Or rather, we’ll be tightening some of them…others will increase. We've done a lot of research to determine which API calls and which types of calls cost us the most, and we want our API limits to reflect those costs. We love the POSI approach: our data is free, but services are not. API is a service, and it costs money. If we charge the heavy users, the vast majority can continue to use the API for free—and we think we can offer a fantastic value for the paying users, too.

Stay tuned for announcements on the revised limits.

Looking ahead

OpenAlex is open infrastructure, and we're committed to keeping it open and free (as in speech). To do that we have to make sure usage costs are sustainable and shared fairly. We have to make sure that the free API stays up while at the same time offering great service to paying users.

We didn’t get that balance right yesterday.

We also need to invest harder in unsexy features like continuous integration, telemetry, QA, monitoring, performance assessments, and testing. If we want to be truly infrastructure—and we do—these are nonnegotiable. We fell a bit behind on these, and it bit us, and it bit the community. Lesson learned. We’ve made concrete improvements already and will continue to integrate more.

We really appreciate this community for giving us honest feedback and being patient as we learn and improve.

If you have questions or concerns, please reply to this thread or reach out directly.

Jason