Help with SEC 403 Error While Using Arelle with Threading

[Faisal Anjum]

Hi Austin,

Hope you're doing well!

I have Python code that successfully fetches XBRL data using the Arelle library. However, when I try to run it on multiple SEC XBRL instances simultaneously using threading, I keep encountering a 403 Forbidden error from the SEC.

I'm struggling to understand how to bypass this issue to make it work. Do you have any insights?

Many thanks,
Faisal

[Austin Matherne]

Hi Faisal,

You should declare an email address identifying your application in the user agent header and limit requests to 10 per second.

https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data

Kind regards,

Austin Matherne

[Faisal Anjum]

 

Hi Austin,

 

Following your advice, I've implemented:

·      SEC-compliant user agent: "EventTrader-XBRL/1.0 (faia...@gmail.com)"

·      Conservative rate limiting: 2-second delay between requests (only 0.5 req/sec, well below the 10/sec limit)

·      Retry logic with exponential backoff (2, 4, 8 seconds)

 

The first XBRL document loads successfully, but subsequent requests still receive 403 Forbidden errors, even with these conservative settings. Is there a cooldown period required between successful requests, or another header beyond User-Agent that I'm missing?

 

Many thanks,

Faisal

--

---
You received this message because you are subscribed to a topic in the Google Groups "Arelle-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/arelle-users/pO-gJQwiOd4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to arelle-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/arelle-users/f7ed5c5d-543b-41b1-bb73-a115935e0872n%40googlegroups.com.

[Austin Matherne]

Hi Faisal,

I would expect that to work. You can try reaching out to the SEC for more details using the email addresses available from the page I previously linked.

Best,

Austin