Querying the locally installed DB via arelle web services

[asker12345]

Hi, I am a complete no-ob to arelle and XBRL. I just discovered this group and am looking for help.

My final goal, is to execute, for example "show me all companies with market cap > $1B in 2013" ( just an example). But I want to do this using a locally installed DB.

Here is what I have done so far (I am on OS X)

a) Installed postgres, installed the arelle app

b) Inside arelle, installed the XBRL DB plugin and connected it to my local postgres server

c) Downloaded arelle's 20GB pgdump file http://arelle.org/documentation/xbrl-database/ and used psql to populate it into my db. It is currently at 200GB - will take a few more hours to reach 417GB

d) I've started arelle's web service and confirmed I can access it via my browser

Now when I look at arelle's web service API documentation, the examples seem to all refer to individual xml files I may have downloaded from SEC. How do I use it to query the files already populated in my local DB and execute the query that I listed above via their web service?

thanks

[Herm Fischer]

you are correct that the Arelle python app and web service process the XML files of XBRL submissions to the SEC.  Arelle does populate a database which could do the query  (after normalizing the differences in tagging).  The database that Arelle loads probably can support your query.  We are working on statement detection and normalization to some degree.

If you want to play with our copy of the loaded-up database send a private e-mail to sup...@arelle.org (to let us know who you are).  A public backup of this database is posted for download (but it is very large and a pain to install).  Others have explored maybe syncing local copies from it.

   Herm

--

---
You received this message because you are subscribed to the Google Groups "Arelle-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to arelle-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arjun Roychowdhury]

Hi Herm, thanks for the response. I am afraid my level of understanding is still too basic to fully comprehend what you mean ( I am new both to XBRL as well as the entire financial world)

 I sucessfully downloaded your xbrl_db_SEC.pg.zip from the site via wget. It was around 20GB. I am in the process of importing it into my local postgres server --> its still populating and is around 250GB so far. Per your email to me, it will reach around 500GB. I am fine with that size. My understanding is that this database is basically going to  give me a local  copy of all SEC filings without me having to grab them 1 by 1 from SEC (which is very slow)

Now the reason I downloaded this DB was I was hoping once populated, I can use some arelle web services API (I have your webservices instance running on my mac and arelle has the pg XBRL db plugin installed and connected to my pg server) to query basic financial data

(the example I like quoting is "get me a list of all companies that have market cap >$1B in 2013")

I understand the statement above is a human readable statement and that it will need to be translated to a specific set of parameters through the web services API.

Is this understanding correct on my behalf? If this understanding is correct, can someone please help  me with

a) what is the exact web services url construct that will return the list of companies for my above human readable query?

If instead you are saying that the webservices APIs that arelle exposes are not intended to work on the local database, then fine, I don't mind executing direct post gres queries either, in whih case my question translates to

a) what is the exact postgres sql construct that will return the list of companies for my above human readable query?

Thank you for your patience

You received this message because you are subscribed to a topic in the Google Groups "Arelle-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/arelle-users/84QFhzKeHuI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to arelle-users...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Arjun Roychowdhury

[asker12345]

Hi, I've made progress directly querying the local DB provided by Arelle to get at certain attributes I want via direct SQL statements. I assume the Arelle web services component is not geared to work with the preloaded DB that Arelle provides at http://arelle.org/documentation/xbrl-database/

Has anyone had any success connecting the web services component to extract data from the preloaded DB? Any tips? That would save me a lot of time rather than having to decipher the complex XBRL structure/taxonomy