This was a result of the following conditions:
The data generation process used multiple threads to each stream bulk insert operations to a stand-alone instance of mongod using write_concern of k_unacknowledged. Using k_unacknowledged prevents the driver from throttling requests, so mongod fell behind. It appears to cache the bulk operations. When this exception occurred the cxx driver will throw bulk_write_exception.
additionally in my case, I was catching bulk_write_exception but continuing my loop using the same connection without closing/reopening. This caused the number of connections displayed by mongostat to continually grow as my application continued to run in this way. Killing my process resulted in mongod slowly "draining" the open connections back down to zero, apparently by executing the operations associated with them.