I have a worker where I'm using the pg gem. There appears to be an issue with the `conn.transaction do` block taking a long time.
Here's a screenshot of an example run time. There are some where the transaction bit is much worse, but this gives you an idea.
Without going into the details of every method here's the parent class which all my sidekiq workers inherit.
class Worker
include PgImport
def perform(thread_hash)
class_name = self.class.name.split("::").last.underscore
ActiveSupport::Notifications.instrument("perform.#{class_name}") do
temp_table_name, built_data = ActiveSupport::Notifications.instrument("data.#{class_name}") do
data(thread_hash)
end
$connection_pool['reporting'].with do |conn|
ActiveSupport::Notifications.instrument("reset_connection.#{class_name}") do
conn.reset if conn.status != 0
end
ActiveSupport::Notifications.instrument("create_transaction.#{class_name}") do
conn.transaction do
ActiveSupport::Notifications.instrument("create_temp_table.#{class_name}") do
conn.exec(temp_table_sql)
end
ActiveSupport::Notifications.instrument("import_data.#{class_name}") do
import_data(built_data, conn, 'pg_temp', temp_table_name, "STDIN WITH (FORMAT 'csv', DELIMITER E'\t')")
end
ActiveSupport::Notifications.instrument("create_indexes.#{class_name}") do
conn.exec(temp_table_indexes_sql)
end
ActiveSupport::Notifications.instrument("analyze_table.#{class_name}") do
conn.exec(analyze_sql)
end
child_partition_dates = ActiveSupport::Notifications.instrument("child_partition_sql.#{class_name}") do
conn.exec(child_partition_table_sql)
end
# Update the master table
ActiveSupport::Notifications.instrument("write_data.#{class_name}") do
child_partition_dates.field_values('partition_date').each do |partition_date|
ActiveSupport::Notifications.instrument("lock_sql.#{class_name}") do
conn.exec(lock_sql(partition_date))
end
ActiveSupport::Notifications.instrument("update_sql.#{class_name}") do
conn.exec(update_sql(partition_date))
end
ActiveSupport::Notifications.instrument("insert_sql.#{class_name}") do
conn.exec(insert_sql(partition_date))
end
end
end
end
end
end
end
end
If anyone has some ideas where I should look to figure out why this is the case I'm all ears. I highly doubt it's the PG gem itself because I have googled and don't see many others with the issue, but I have no idea how to figure out what the issue is.