ERROR: parameter "autovacuum" cannot be changed now
SQL state: 55P02
I should have been more clear. I virtually never delete or do updates, but I insert a lot. So the table does change quite a bit, but only in one direction.
I was unable to disable autovacuum universally (due to the cant_change_runtime_param error) but I was able to disable it on individual tables. Still, I know this is heavy handed and sub-optimal. I tried set autovacuum_naptime='6min' but got the same 55P02 error. Should/can I set that per table?
I did look at autovacuum_vacuum_threshold and autovacuum_vacuum_scale_factor but couldn't make sense out of them. (Besides, I'd probably get the same 55P02 error if I tried to change them.)
The documentation has information like "This parameter can only be set in the postgresql.conf file or on the server command line." that will tell you in advance which settings will fail when you attempt to set them through SQL statements.
But autovacuum is pretty smart about not vacuuming tables until reasonably necessary. And beware that autovacuum is also controlling when to analyze a table. Mass inserts are probably changing the characteristics of your table such that it needs to be analyzed to allow the planner to properly optimize your queries.
Have you identified that vacuum is actually causing a problem? If not, I'd leave it alone. The system tables have a lot of information on table vacuuming and analyzing:
select
relname,
last_vacuum,
last_autovacuum,
last_analyze,
last_autoanalyze,
vacuum_count,
autovacuum_count,
analyze_count,
autoanalyze_count
from
pg_stat_user_tables;
On Thu, Feb 23, 2012 at 10:01 AM, Steve Crawford <scra...@pinpointresearch.com> wrote:
The documentation has information like "This parameter can only be set in the postgresql.conf file or on the server command line." that will tell you in advance which settings will fail when you attempt to set them through SQL statements.
Ah. I missed that. Sorry for asking stupid questions.
But autovacuum is pretty smart about not vacuuming tables until reasonably necessary. And beware that autovacuum is also controlling when to analyze a table. Mass inserts are probably changing the characteristics of your table such that it needs to be analyzed to allow the planner to properly optimize your queries.
Okay, that makes more sense to me; because the stats would be changing quickly and so while vacuuming may not be necessary, analyzing would be. At the same time, I can't afford to analyze if it's causing my inserts to take over 50 ms. Something else I should add: if my selects are slow, that's annoying; but if my inserts are slow, that could be disastrous...
Then you are not running a current version of PostgreSQL so the first step to performance enhancement is to upgrade. (As a general rule - there are occasionally specific cases where performance decreases.)...Apparently the last four columns don't exist in my database. As for the first four, that is somewhat illuminating....
So if it's not auto-vacuuming that's making my inserts so slow, what is it? I'm batching my inserts (that didn't seem to help at all actually, but maybe cause I had already turned off synchronous_commit anyway).
I've gotten rid of a bunch of indices (especially those with low cardinality–that I did around the same time as disabling auto-vacuum, so that could account for the coincidental speed up).
I'm not sure what else I could be doing wrong. It's definitely better than it was a few days ago, but I still see "LOG: duration: 77.315 ms statement: COMMIT" every minute or two.
That depends on if you have triggers that are doing selects. But in general you are correct, analyze wont help inserts.
checkpoint_segments can help insert speed, what do you have that set to?
Also how you insert can make things faster too. (insert vs prepared vs COPY)
Also, if you have too many indexes on a table that can cause things to slow down.
Your IO layer needs to be fast too. Have you watched vmstat and iostat?
Have you read up on synchronous_commit?
You need to rethink things a bit. Databases can fail in all sorts of ways and can slow down during bursts of activity, data dumps, etc. You may need to investigate some form of intermediate buffering.
Then you are not running a current version of PostgreSQL so the first step to performance enhancement is to upgrade. (As a general rule - there are occasionally specific cases where performance decreases.)...Apparently the last four columns don't exist in my database. As for the first four, that is somewhat illuminating....
How are you batching them? Into a temp table that is copied to the main table? As a bunch of insert statements within a single connection (saves probably considerable time due to eliminating multiple connection setups)? With one PREPARE and multiple EXECUTE (to save repeated planning time - I'm not sure this will buy you much for simple inserts, though)? With COPY (much faster as many records are inserted in a single statement but if one fails, all fail)?
And what is the 50ms limit? Is that an average? Since you are batching, it doesn't sound like you need every statement to complete in 50ms. There is always a tradeoff between overall maximum throughput and maximum allowed latency.
I recommend you abandon this thread as it presupposes a now seemingly incorrect cause of the problem and start a new one titled something like "Tuning for high insert rate" where you describe the problem you want to solve. See http://wiki.postgresql.org/wiki/Guide_to_reporting_problems for a good guide to the information that will be helpful in diagnosis.
It's possible that you might get a nice boost by wrapping the inserts into a transaction:
begin;
insert into...;
insert into...;
insert into...;
...
commit;
This only requires all that disk-intensive stuff that protects your data once at the end instead of 1000 times for you batch of 1000.
COPY is even better. I just ran a quick test by restoring a table on my desktop hacking db (untuned, few years old PC, single SATA disk, modest RAM and lots of resource competition). The 22+ million rows restored in 282 seconds which is a rate somewhat north of 78,000 records/second or about 0.13ms/record.
My hunch is still that your issue is lock contention.
We have many customers who do much more than this throughput, thoughI'm not sure what level of resourcing you're current at. You might
consider experimenting with a larger system if you're having
performance problems.