Activerecord models not reliably being indexed on save with sunspot 2.1.1

[Edward Rudd]

We have upgraded from Sunspot 1.3.0 to 2.1.1 recently and are having lots of issues with documents not being indexed when changed.    This is actually the second time the development team has tried this ans the same issues occurred both times.

So, the web-app is a Rails 3.2 application which has been using sunspot 1.3 for nearly 3 years (against a Solr 4.0 server) and everything worked perfectly fine in that scenario.  So now we have upgrade to Sunspot 2.1.1 / Solr 4.2 and our AR models are not reliably being indexed..   What I have so far been able to figure out is that it is not anything to do with the solr instance..as that is working fine..  Instead it seems that for some reason sunspot just decides to not send a model over to solr to be reindexed when saved..   But it does it randomly..   

If I restart the main rails application everything is fine for a few hours. (example.. I restarted it 4 hours ago today).  and it's been indexing happily fine until about 40 minutes ago.. when it "skips" indexing things again.    I had a wireshark trace on the communication earlier this morning and watched as sunspot issued a commit (due to the commit_if_dirty at the and of a web request) but no document was indexed before hand!!.    Right now I have the solr log in debug mode and am tailing all the calls to /update and for most of the day it would be doing

INFO: [production] webapp=/solr path=/update params={wt=ruby} {add=[Document 286622 (1490215034256621568)]} 0 5
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 15
INFO: [production] webapp=/solr path=/update params={wt=ruby} {add=[Communication 630465 (1490215038080778240)]} 0 11
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 23

But then ~ 40 minutes ago it started doing this.

INFO: [production] webapp=/solr path=/update params={wt=ruby} {add=[Task 607347 (1490213291601952768)]} 0 4
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 28
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 23
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 9
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 9
INFO: [production] webapp=/solr path=/update params={wt=ruby} {commit=} 0 12

I am at a loss as to where to look to diagnose why in the world sunspot is not sending things over to Solr.

One specific example is I have a "test task" in the system and all I do is complete it.. and then uncomplete it..  (essentially changing the status and setting/clearing the completed_at column). And it'll work fine reliably as it did 4 hours ago.. but then 40 minutes ago.. solr would never see the update request come through. and I would just see a bare commit with no add before.