Trying to figure out what the conn.transaction bit of code is taking so long?

8 views
Skip to first unread message

ch...@altonymous.com

unread,
Oct 21, 2014, 11:32:05 PM10/21/14
to rub...@googlegroups.com
I have a worker where I'm using the pg gem.  There appears to be an issue with the `conn.transaction do` block taking a long time.  

Here's a screenshot of an example run time.  There are some where the transaction bit is much worse, but this gives you an idea.


Without going into the details of every method here's the parent class which all my sidekiq workers inherit.

class Worker
  include PgImport
  def perform(thread_hash)
    class_name = self.class.name.split("::").last.underscore
    ActiveSupport::Notifications.instrument("perform.#{class_name}") do
      temp_table_name, built_data = ActiveSupport::Notifications.instrument("data.#{class_name}") do
        data(thread_hash)
      end
      $connection_pool['reporting'].with do |conn|
        ActiveSupport::Notifications.instrument("reset_connection.#{class_name}") do
          conn.reset if conn.status != 0
        end
        ActiveSupport::Notifications.instrument("create_transaction.#{class_name}") do
          conn.transaction do
            ActiveSupport::Notifications.instrument("create_temp_table.#{class_name}") do
              conn.exec(temp_table_sql)
            end
            ActiveSupport::Notifications.instrument("import_data.#{class_name}") do
              import_data(built_data, conn, 'pg_temp', temp_table_name, "STDIN WITH (FORMAT 'csv', DELIMITER E'\t')")
            end
            ActiveSupport::Notifications.instrument("create_indexes.#{class_name}") do
              conn.exec(temp_table_indexes_sql)
            end
            ActiveSupport::Notifications.instrument("analyze_table.#{class_name}") do
              conn.exec(analyze_sql)
            end
            child_partition_dates = ActiveSupport::Notifications.instrument("child_partition_sql.#{class_name}") do
              conn.exec(child_partition_table_sql)
            end
            # Update the master table
            ActiveSupport::Notifications.instrument("write_data.#{class_name}") do
              child_partition_dates.field_values('partition_date').each do |partition_date|
                ActiveSupport::Notifications.instrument("lock_sql.#{class_name}") do
                  conn.exec(lock_sql(partition_date))
                end
                ActiveSupport::Notifications.instrument("update_sql.#{class_name}") do
                  conn.exec(update_sql(partition_date))
                end
                ActiveSupport::Notifications.instrument("insert_sql.#{class_name}") do
                  conn.exec(insert_sql(partition_date))
                end
              end
            end
          end
        end
      end
    end
  end

If anyone has some ideas where I should look to figure out why this is the case I'm all ears.   I highly doubt it's the PG gem itself because I have googled and don't see many others with the issue, but I have no idea how to figure out what the issue is.

Altonymous

unread,
Oct 21, 2014, 11:35:00 PM10/21/14
to rub...@googlegroups.com
Should have also noted the versions I'm using...

pg (0.17.1)
ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
postgres (PostgreSQL) 9.3.5
Reply all
Reply to author
Forward
0 new messages