db:migrate ManageAccountBanksBackfill unknown attribute singleton error (canvas-lms/stable 2021-10-21)

[Skylar]

We're testing canvas-lms/stable from 2021-10-21 and running into a db:migration error for ManageAccountBanksBackfill.

Steps to reproduce:

1. Upgrading from canvas-lms/stable 2020-11-30 to canvas-lms/stable 2021-10-21.
2. Running db:migrate:predeploy results in an error for ManageAccountBanksBackfill

Already tried clearing redis cache and re-running migrate with no luck. Looking forward to any help on what the issue could be. Any files we can tweak to get passed this migration error? Thanks!

Error returned: unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job.

-----

==  ManageAccountBanksBackfill: migrating =====================================

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:201: warning: The called method `in_batches' is defined here

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/performable_method.rb:5: warning: The called method `initialize' is defined here

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job.

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:52:in `_assign_attribute'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/core.rb:324:in `initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:34:in `block in initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `block in instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:33:in `initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/active_record.rb:38:in `create'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/base.rb:89:in `enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block (2 levels) in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:6:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:40:in `block in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:39:in `enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68:in `method_missing'

/var/canvas/app/models/role_override.rb:43:in `clear_caches'

/var/canvas/app/models/role_override.rb:38:in `clear_caches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `invoke_after'

/var/canvas/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:181:in `run_callbacks'

/var/canvas/config/initializers/active_record.rb:1747:in `block in run_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'

/var/canvas/config/initializers/active_record.rb:1747:in `run_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `create_or_update'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:128:in `create_or_update'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:503:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/validations.rb:53:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `block in save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:43:in `block in transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:38:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/suppressor.rb:48:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `block in save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `save!'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:46:in `block (2 levels) in run'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `each'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `block in run'

/var/canvas/config/initializers/active_record.rb:779:in `block in find_in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `loop'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call_super'

/var/canvas/config/initializers/active_record.rb:793:in `block in in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'

/var/canvas/config/initializers/active_record.rb:793:in `in_batches'

/var/canvas/config/initializers/active_record.rb:778:in `find_in_batches'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:35:in `run'

/var/canvas/db/migrate/20210715175216_manage_account_banks_backfill.rb:25:in `up'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:831:in `exec_migration'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:811:in `block in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:810:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/active_record/migration.rb:16:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:624:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1002:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1363:in `ddl_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'

/var/canvas/config/initializers/active_record.rb:1611:in `execute_migration_in_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `migrate_without_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `block in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:51:in `block in with_advisory_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `block in with_advisory_lock_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `with_advisory_lock_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:47:in `with_advisory_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `migrate'

/var/canvas/lib/tasks/canvas.rake:175:in `block (3 levels) in <top (required)>'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (6 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (5 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:87:in `block (4 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:362:in `block (2 levels) in with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:360:in `block in with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:356:in `with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:72:in `block (3 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:69:in `block (2 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/guard_rail.rb:24:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:8:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:68:in `block in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Caused by:

ActiveModel::UnknownAttributeError: unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job.

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:52:in `_assign_attribute'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/core.rb:324:in `initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:34:in `block in initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `block in instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `instrument'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:33:in `initialize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/active_record.rb:38:in `create'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/base.rb:89:in `enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block (2 levels) in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:6:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:40:in `block in enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:39:in `enqueue'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68:in `method_missing'

/var/canvas/app/models/role_override.rb:43:in `clear_caches'

/var/canvas/app/models/role_override.rb:38:in `clear_caches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `invoke_after'

/var/canvas/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:181:in `run_callbacks'

/var/canvas/config/initializers/active_record.rb:1747:in `block in run_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'

/var/canvas/config/initializers/active_record.rb:1747:in `run_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `create_or_update'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:128:in `create_or_update'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:503:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/validations.rb:53:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `block in save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:43:in `block in transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:38:in `transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/suppressor.rb:48:in `save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `block in save!'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `save!'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:46:in `block (2 levels) in run'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `each'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `block in run'

/var/canvas/config/initializers/active_record.rb:779:in `block in find_in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `loop'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call_super'

/var/canvas/config/initializers/active_record.rb:793:in `block in in_batches'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'

/var/canvas/config/initializers/active_record.rb:793:in `in_batches'

/var/canvas/config/initializers/active_record.rb:778:in `find_in_batches'

/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:35:in `run'

/var/canvas/db/migrate/20210715175216_manage_account_banks_backfill.rb:25:in `up'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:831:in `exec_migration'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:811:in `block in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:810:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/active_record/migration.rb:16:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:624:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1002:in `migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1363:in `ddl_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'

/var/canvas/config/initializers/active_record.rb:1611:in `execute_migration_in_transaction'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `migrate_without_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `block in migrate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:51:in `block in with_advisory_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `block in with_advisory_lock_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `with_advisory_lock_connection'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:47:in `with_advisory_lock'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `migrate'

/var/canvas/lib/tasks/canvas.rake:175:in `block (3 levels) in <top (required)>'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (6 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (5 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:87:in `block (4 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:362:in `block (2 levels) in with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:360:in `block in with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:356:in `with_each_shard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:72:in `block (3 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:69:in `block (2 levels) in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/guard_rail.rb:24:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:8:in `activate'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:68:in `block in shardify_task'

/var/canvas/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:migrate:predeploy

(See full trace by running task with --trace)

[Skylar]

While I haven't heard back, I was able to workaround this by moving db/migrate/20210715175216_manage_account_banks_backfill.rb out of that folder, running db:migrate:predeploy and letting it complete all the predeploy migrations, moving it back in, and running db:migrate:predeploy again. So something with the order of the migration scripts was causing this to fail.

Hopefully by running it last (out of order) that doesn't cause any issues - haven't seen any issues so far though!

[Msan]

Hi Skylar,

I faced the problem too, but I found a way to fix it.

I have the same error `unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job` when migrating `db/migrate/20111111214312_create_account0.rb`, so i change the `db/migrate/20210812210129_add_singleton_column.rb` to `db/migrate/20111111214311_add_singleton_column.rb`, make sure it before create_account0.rb date and it work.

In your case, I think change `20210812210129_add_singleton_column.rb` to `20210715175215_add_singleton_column.rb` may work.