| We will need a script that can run periodically via cron (nightly?) which will: - Keep track of the sha of the last build it tested in a file - Checkout the pdb repo and compare the tip commit with last tested, if diff run tests - Run a/b tests with the different versions of pdb and report results via email/slack. (possibly only report on tests if perf diff is over a certain threshold) - Tests could be as simple as running queries stored in a file via curl while tracking the query time. (may want to consider running queries multiple times in random order and taking average time of query) - At first use a couple of queries that have caused issues in the past. (possibly audit the console and add the built-in queries it uses) Additional considerations: - Make sure data doesn't get gc'd when pdb starts. Either increase node-ttl or adjust timestamps to far in the future in PDB-4771 - Check that skewed timestamps don't cause results to be excluded in query engine - We may want to consider using cgroups to limit the available ram PG has access to when running tests for more realistic results |
