Luke, not sure if this helps, but entries to the delayed_jobs log end on 8/26 (the same day exports stopped working). Here are the last few entries, followed by our attempts to re-start the job worker:
2015-08-26T13:17:12-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] Question#add_prompt_to_queue completed after 0.0062
2015-08-26T13:17:12-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] Question#add_prompt_to_queue completed after 0.0058
2015-08-26T13:17:12-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] 2 jobs processed at 84.6761 j/s, 0 failed ...
2015-08-26T13:17:17-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] Question#add_prompt_to_queue completed after 0.0284
2015-08-26T13:17:17-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] 1 jobs processed at 16.7299 j/s, 0 failed ...
2015-08-26T16:37:28-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2445)] Question#export completed after 6.9199
2015-09-11T12:53:39-0400: [Worker(host:ip-172-31-17-4.ec2.internal pid:2858)] New Relic Ruby Agent Monitoring DJ worker host:ip-172-31-17-4.ec2.internal pid:2858
2015-09-11T12:53:40-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2858)] Starting job worker
2015-09-11T13:34:58-0400: [Worker(host:ip-172-31-17-4.ec2.internal pid:3938)] New Relic Ruby Agent Monitoring DJ worker host:ip-172-31-17-4.ec2.internal pid:3938
2015-09-11T13:34:58-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:3938)] Starting job worker
2015-09-17T11:23:26-0400: [Worker(host:ip-172-31-17-4.ec2.internal pid:6074)] New Relic Ruby Agent Monitoring DJ worker host:ip-172-31-17-4.ec2.internal pid:6074
2015-09-17T11:23:27-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:6074)] Starting job worker
2015-09-17T11:36:40-0400: [Worker(host:ip-172-31-17-4.ec2.internal pid:2149)] New Relic Ruby Agent Monitoring DJ worker host:ip-172-31-17-4.ec2.internal pid:2149
2015-09-17T11:36:40-0400: [Worker(delayed_job host:ip-172-31-17-4.ec2.internal pid:2149)] Starting job worker
And below are the entries from the production.log when the 404 errors began:
Processing QuestionsController#export to xml (for 54.208.117.91 at 2015-08-26 16:37:17) [POST]
Parameters: {"question"=>{"active"=>false, "choices_count"=>9, "created_at"=>2015-08-21 15:23:45 UTC, "creator_id"=>4527, "id"=>51, "inactive_choices_count"=>0, "information"=>nil, "it_should_autoactivate_ideas"=>false, "local_identifier"=>"1", "name"=>"[QUESTION REDACTED]", "prompts_count"=>72, "show_results"=>true, "site_id"=>1, "tracking"=>nil, "updated_at"=>2015-08-21 15:23:56 UTC, "uses_catchup"=>true, "version"=>2, "votes_count"=>71293, "item_count"=>9}, "key"=>"wikisurvey_51_ideas_2015-08-26T20:37:17Z_PySo", "type"=>"ideas", "id"=>"51"}
Processing QuestionsController#show to xml (for 54.208.117.91 at 2015-08-26 16:37:23) [GET]
Parameters: {"id"=>"51"}
Processing QuestionsController#export to xml (for 54.208.117.91 at 2015-08-26 16:37:23) [POST]
Parameters: {"question"=>{"active"=>false, "choices_count"=>9, "created_at"=>2015-08-21 15:23:45 UTC, "creator_id"=>4527, "id"=>51, "inactive_choices_count"=>0, "information"=>nil, "it_should_autoactivate_ideas"=>false, "local_identifier"=>"1", "name"=>"[QUESTION REDACTED]", "prompts_count"=>72, "show_results"=>true, "site_id"=>1, "tracking"=>nil, "updated_at"=>2015-08-21 15:23:56 UTC, "uses_catchup"=>true, "version"=>2, "votes_count"=>71293, "item_count"=>9}, "key"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn", "type"=>"votes", "id"=>"51"}
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:37:56) [GET]
Parameters: {"id"=>"wikisurvey_51_ideas_2015-08-26T20:37:17Z_PySo"}
Sending data
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:37:59) [GET]
Parameters: {"id"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn"}
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:38:59) [GET]
Parameters: {"id"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn"}
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:39:59) [GET]
Parameters: {"id"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn"}
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:42:00) [GET]
Parameters: {"id"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn"}
Processing ExportsController#show (for 54.208.117.91 at 2015-08-26 16:47:00) [GET]
Parameters: {"id"=>"wikisurvey_51_votes_2015-08-26T20:37:23Z_IdAn"}
Processing QuestionsController#show to xml (for 54.208.117.91 at 2015-08-26 16:47:34) [GET]
Parameters: {"id"=>"51"}
As you'll see, I ran a successful Ideas export at 2015-08-26 16:37:56 followed by a Votes export attempt at 2015-08-26 16:37:59. The Votes export failed, and all exports have failed since.
Does this help? We truly appreciate your input, Luke!
Thanks,
Kyle