API working normally; machine access to content pages temporarily blocked

[Brendan O’Connell]

Dear API users,

Since late June, we've successfully restored access to our API and OAI-PMH feed by directing all machine traffic to dedicated machines. Since these machines are still experiencing a high volume of bot traffic, there may be occasional slowdowns for the API and OAI-PMH feed.

However, to preserve human access to our public site, we've had to turn off machine access to the main content pages of DOAJ and endpoints by implementing a Cloudflare Managed Challenge that generally blocks machine requests. For example, https://doaj.org/search/articles is available to human users, but not machine access.

We've heard from some of you that this is causing issues for your workflows. Unfortunately, this is the best solution we can provide right now, because turning machine access back on for our main content pages could again result in major slowdowns and downtime across our site for human users.

We are currently working with our developers on a longer-term solution that will craft more specific rules around which bots can access content pages, so that "good" bots are allowed and "bad" bots excluded. 

I will provide updates to this list as I have them. In the meantime, please continue to use the API and OAI-PMH feed for all machine requests to DOAJ. 

Best,

Brendan O'Connell

Platform Manager, DOAJ