Follow up: Open Data vs. opensecrets.org industry amount mismatch

[Chris]

Dear all,

First of all, thank you very much open secrets for sharing your data on politicians and elections. 

I have a follow up question regarding an industry amount mismatch for individual contributions to specific candidates between the opensecrets.org website and the bulk data. My question relates to an earlier thread by Skye, which can be found here:

https://groups.google.com/d/topic/opensecrets-open-data/bDV-5HTINvY/discussion

I followed the instructions given in the open data manual and in the above thread to calculate individual contributions by industry to individual politicians, so basically I am replicating the member profiles on opensecrets.org. However, my industry amounts are usually less than the amounts given on the website and I wonder whether there has been an update to the website or whether I have still a coding error.

Here are two examples:

1) In the above thread, Skye replicated individual contributions from the Oil & Gas industry for representative Randy Neugebauer for the 2006 cycle. Here is the member page:

http://www.opensecrets.org/politicians/industries.php?cycle=2006&type=I&cid=N00026043&newMem=N&recs=20

Interestingly, I am able to replicate the amount of $39,000, which should have been the amount listed on the website in 2010 (as indicated by the above thread). However, the website currently lists a total of $44,600 for the Oil & Gas industry for the 2006 cycle. So I guess that there has been a revision that is not included in the bulk data?

2) In the 2006 election cycle, representative Spencer Bachus received a total of $425,331 individual contributions as can be found at the bottom of the member page:

http://www.opensecrets.org/politicians/summary.php?cid=N00008091&cycle=2006

However, when I sum over all individual contributions received by Spencer Bachus in 2006 (even with and without applying the inclusion/exclusion criteria as outlined in the data manual), I get a total amount of individual contributions of only $407,465.

I would be very grateful for any help on whether this might be a coding error or whether this is only resulting from an updated webpage (which would be a minor concern for me). If the answer is already in one of the threads in this group, I must have missed it and would be thankful if you could provide me with the reference.

Thanks and best regards,

Chris

[Susi Alger]

Hello Chris,

First of all, you're very welcome.  It's wonderful to see folks dig into this treasure trove.  We're very proud of the work we do and enjoy helping folks use it correctly.

1)  First of all, while Skye's question did have to do with trying to use the OpenData to match OpenSecrets.org numbers -- the answer I provided to his question is definitely not the same as the answer to your questions.  In Skye's case, he was trying to match numbers on the site that included money we used not only in the individual contribution table, but also the PACs and Pac_Other table.  In your case, you're trying to match numbers in the Indivs column only, which does come only from the individual contributions table - so you don't need to include other tables.  I believe that the difference is simply time lag.  We are constantly updating and improving our data for all cycles, while we do concentrate on the current cycle more than others.  We last updated the 2006 cycle page for Randy Neugeberger on July 10, 2011 -- or 14 months after the last 2006 data set was released in OpenData.  We do intend to update older cycles sometime this year, but we have a lot of backend work to do before we can begin that process.  There are other more technical reasons why things may not match, but generally those reasons would cause our totals to be somewhat lower for Oil & Gas and other non-ideological codes than yours and not higher.

2)  Spencer Bachus source of funds.  The individual contribution total (actually, all the totals) come from the member's summary filing with the FEC.  It is NOT the total of the itemized contributions provided, which are limited to those over $200.  In later cycles (check 2012, for example), the data is broken down into itemized and non-itemized individual contributions.  In theory, the total in the individual contribution table should match the "Itemized Contributions".

Hope this helps and good luck with your project.

Susi

--
 
 

[Chris]

Dear Susi,

Thank you very much for your quick reply and the clarifications. I guess my differences are then simply resulting from time lag, which is good news.

Thank you very much!

Chris