| The loop resulting in this is here: https://github.com/puppetlabs/puppetdb/blob/6.3.4/src/puppetlabs/puppetdb/scf/migrate.clj#L1543-L1558 There are a couple items that lead to the INSERT being executed once per row:
- Reduce will iterate over each row in isolation, so there is only one row passed to jdbc/insert-multi!, instead of a batch of many rows.
- We’re passing a seq of hashes to jdbc/insert-multi! and the docs for that function indicate that will cause a separate database operation per hash of data.
Item 1 might be addressed by using Clojure's partition function to chunk the list of rows into batches: http://clojuredocs.org/clojure.core/partition For item 2, the java.jdbc docs indicate that we should pas a seqence of lists of column values to jdbc/insert-multi!: https://github.com/clojure/java.jdbc/blob/java.jdbc-0.7.7/src/main/clojure/clojure/java/jdbc.clj#L1574-L1577 Also, it sounds like getting JDBC to do batched inserts with Postgres requires the :reWriteBatchedInserts option to be set on the database connection: https://github.com/clojure/java.jdbc/blob/java.jdbc-0.7.10/src/main/clojure/clojure/java/jdbc.clj#L1626-L1631 |
