In the end we implemented timeouts both on the pdb and pg sides, though the pg side timeouts aren't as precise because pg's statement_timeouts appear to apply to each underlying fetch batch for a top-level select, rather than the select as a whole. After some discussion, including CS, we also settled on a default statement timeout of 10m, which we expect to be long enough to avoid any significant disruption of existing environments (that aren't already likely in related trouble). Ideally, it might be even lower, something we can revisit later. Assuming everything works right, that timeout should also substantially mitigate (or outright avoid) the causes of a number of serious escalations we've seen in the past, since no queries will be allowed to remain open for (sometimes) hours, unintentionally. |