Maier queue in production stuck on Enqueued

[Russell Roberts]

I have tried to work through many of the goggle searches & also studied the links at https://github.com/mperham/sidekiq/wiki

Sidekiq works fine ion my local Mac with redis but not on my Mac

I would really appreciate sone help

here is my sidekiq.yml

#config/sidekiq.yml:

---

:concurrency: 1

:queues:

  - default

  - mailers

====================================================

1. 
   

[Eric Chapman]

Hey Russell,

This looks like your jobs are getting submitted fine but your Sidekiq background process either isn't running or isn't connected to that particular Redis instance.  Could you share some info on your production setup?  Your original description isn't very clear

"Sidekiq works fine ion my local Mac with redis but not on my Mac"

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/17c0b41f-d5d6-46dd-b30f-f833d16a61fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rus...@drusound.com]

Thank you Erik

Sorry for my lack of clarity

Indeed I am on Mac OS for development

And for production I have Ubuntu 18.04 running on a linode

I use nginx & passenger and deploy with Capistrano, for my rails 6 app

I pretty much use the passenger website guides for deployment.

I just cd in to my current deployment directory on the server and ran

 Bundle exec sidekiq all works well my emails were  sent and the queue processed

But I guess this is not a log erm solution

I have added a sidekick.service to /lib/systemd/system

I think my path to bundle  is /var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production

But this dose not work if I run start & status I get the following

russell@linode_drusound_6:~$ sudo systemctl start  sidekiq

russell@linode_drusound_6:~$ sudo systemctl status  sidekiq

● sidekiq.service - sidekiq

   Loaded: loaded (/lib/systemd/system/sidekiq.service; enabled; vendor preset: enabled)

   Active: failed (Result: exit-code) since Mon 2019-06-24 21:25:44 UTC; 6s ago

  Process: 15890 ExecStart=/var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production (code=exited, status=42)

 Main PID: 15890 (code=exited, status=42)

Jun 24 21:25:43 linode_drusound_6 systemd[1]: sidekiq.service: Main process exited, code=exited, status=42/n/a

Jun 24 21:25:43 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 5.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: Stopped sidekiq.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: sidekiq.service: Start request repeated too quickly.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 24 21:25:44 linode_drusound_6 systemd[1]: Failed to start sidekiq.

russell@linode_drusound_6:~$ 

If I run /var/www/drusound_beta_6/current/bin on the terminal I get the following

russell@linode_drusound_6:~$ /var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production

2019-06-24T21:32:39.222Z 16274 TID-grzsdodfi INFO: ==================================================================

2019-06-24T21:32:39.222Z 16274 TID-grzsdodfi INFO:   Please point sidekiq to a Rails 4/5 application or a Ruby file  

2019-06-24T21:32:39.222Z 16274 TID-grzsdodfi INFO:   to load your worker classes with -r [DIR|FILE].

2019-06-24T21:32:39.222Z 16274 TID-grzsdodfi INFO: ==================================================================

2019-06-24T21:32:39.222Z 16274 TID-grzsdodfi INFO: sidekiq [options]

    -c, --concurrency INT            processor threads to use

    -d, --daemon                     Daemonize process

    -e, --environment ENV            Application environment

    -g, --tag TAG                    Process tag for procline

    -i, --index INT                  unique process index on this machine

    -q, --queue QUEUE[,WEIGHT]       Queues to process with optional weights

    -r, --require [PATH|DIR]         Location of Rails application with workers or file to require

    -t, --timeout NUM                Shutdown timeout

    -v, --verbose                    Print more verbose output

    -C, --config PATH                path to YAML config file

    -L, --logfile PATH               path to writable logfile

    -P, --pidfile PATH               path to pidfile

    -V, --version                    Print version and exit

    -h, --help                       Show help

russell@linode_drusound_6:~$ 

My cab deploy file is as follows

# server-based syntax

# ======================

# Defines a single server with a list of roles and multiple properties.

# You can define all roles on a single server, or split them:

# server "example.com", user: "deploy", roles: %w{app db web}, my_property: :my_value

# server "example.com", user: "deploy", roles: %w{app web}, other_property: :other_value

# server "db.example.com", user: "deploy", roles: %w{db}

# role-based syntax

# ==================

# Defines a role with one or multiple servers. The primary server in each

# group is considered to be the first unless any hosts have the primary

# property set. Specify the username and a domain or IP for the server.

# Don't use `:all`, it's a meta role.

 role :app, %w{rus...@139.162.192.62}

 role :web, %w{rus...@139.162.192.62}

 role :db,  %w{rus...@139.162.192.62}

# Configuration

# =============

# You can set any configuration variable like in config/deploy.rb

# These variables are then only loaded and set in this stage.

# For available Capistrano configuration variables see the documentation page.

# http://capistranorb.com/documentation/getting-started/configuration/

# Feel free to add new variables to customise your setup.

set :repo_url, "g...@bitbucket.org:RussellRoberts/drusound_6.git"

set :branch, 'beta'

set :application, "drusound_beta_6"

#  custom deploy_to options

set :deploy_to, '/var/www/drusound_beta_6'

set :rails_env, :production

set :pty,  false

#set :rvm_ruby_version, '2.5.1@drusound_5_2'

# Custom SSH Options

# ==================

# You may pass any option but keep in mind that net/ssh understands a

# limited set of options, consult the Net::SSH documentation.

# http://net-ssh.github.io/net-ssh/classes/Net/SSH.html#method-c-start

#

set :ssh_options, {

port: 30861,

  forward_agent: true,

  keys: %w(~/.ssh/id_rsa.pem),

  auth_methods: %w(publickey),

  user: 'russell'

}

# The server-based syntax can be used to override options:

# ------------------------------------

# server "example.com",

#   user: "user_name",

#   roles: %w{web app},

#   ssh_options: {

#     user: "user_name", # overrides user setting above

#     keys: %w(/home/user_name/.ssh/id_rsa),

#     forward_agent: false,

#     auth_methods: %w(publickey password)

#     # password: "please use keys"

#   }

=======================================================================

My cap file 

# Load DSL and set up stages

require "capistrano/setup"

# Include default deployment tasks

require "capistrano/deploy"

# Load the SCM plugin appropriate to your project:

#

# require "capistrano/scm/hg"

# install_plugin Capistrano::SCM::Hg

# or

# require "capistrano/scm/svn"

# install_plugin Capistrano::SCM::Svn

# or

require "capistrano/scm/git"

install_plugin Capistrano::SCM::Git

# Include tasks from other gems included in your Gemfile

#

# For documentation on these, see for example:

#

#   https://github.com/capistrano/rvm

#   https://github.com/capistrano/rbenv

#   https://github.com/capistrano/chruby

#   https://github.com/capistrano/bundler

#   https://github.com/capistrano/rails

#   https://github.com/capistrano/passenger

#

require "capistrano/rvm"

require "capistrano/bundler"

require "capistrano/rails/assets"

require "capistrano/rails/migrations"

require "capistrano/passenger"

require 'capistrano/rake'

#require 'rvm1/capistrano3'

#require 'capistrano/sidekiq'

#require 'capistrano/sidekiq/monit' #to require monit tasks # Only for capistran

#require "capistrano/webpacker/precompile"

# Load custom tasks from `lib/capistrano/tasks` if you have any defined

Dir.glob("lib/capistrano/tasks/*.rake").each { |r| import r }

 Thanks so much Russell

On 24 Jun 2019, at 19:48, Eric Chapman <cha...@e2tec.com> wrote:

Hey Russell,

This looks like your jobs are getting submitted fine but your Sidekiq background process either isn't running or isn't connected to that particular Redis instance.  Could you share some info on your production setup?  Your original description isn't very clear

"Sidekiq works fine ion my local Mac with redis but not on my Mac"

On Sun, Jun 23, 2019 at 4:13 AM Russell Roberts <rus...@drusound.com> wrote:

I have tried to work through many of the goggle searches & also studied the links at https://github.com/mperham/sidekiq/wiki

Sidekiq works fine ion my local Mac with redis but not on my Mac

I would really appreciate sone help

here is my sidekiq.yml

#config/sidekiq.yml:

---

:concurrency: 1

:queues:

  - default

  - mailers

====================================================

<Screenshot 2019-06-23 at 11.11.14.png>

<Screenshot 2019-06-23 at 11.10.55.png>

1. 
   

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/17c0b41f-d5d6-46dd-b30f-f833d16a61fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAMnb1%3DsZCJM%3D%3DeHUO%2BkH333AMEJkRh4UeNOSWbObngt%2BB0hp7Q%40mail.gmail.com.

[rus...@drusound.com]

As an update

My current sidekiq.service

==============================================

#

# systemd unit file for CentOS 7, Ubuntu 15.04

#

# Customize this file based on your bundler location, app directory, etc.

# Put this in /usr/lib/systemd/system (CentOS) or /lib/systemd/system (Ubuntu).

# Run:

# - sudo mv  sidekiq.service /lib/systemd/system/sidekiq.service

#   - sudo systemctl daemon-reload

#   - sudo systemctl enable sidekiq

#   - sudo systemctl {start,stop,restart} sidekiq

#   - sudo systemctl stop sidekiq

#   - sudo journalctl -u sidekiq.service

# This file corresponds to a single Sidekiq process.  Add multiple copies

# to run multiple processes (sidekiq-1, sidekiq-2, etc).

#

# See Inspeqtor's Systemd wiki page for more detail about Systemd:

# https://github.com/mperham/inspeqtor/wiki/Systemd

#

[Unit]

Description=sidekiq

# start us only once the network and logging subsystems are available,

# consider adding redis-server.service if Redis is local and systemd-managed.

After=syslog.target network.target

# See these pages for lots of options:

# http://0pointer.de/public/systemd-man/systemd.service.html

# http://0pointer.de/public/systemd-man/systemd.exec.html

[Service]

Type=simple

WorkingDirectory=/var/www/drusound_beta_6/current

# If you use rbenv:

# ExecStart=/bin/bash -lc '/home/deploy/.rbenv/shims/bundle exec sidekiq -e production'

# If you use the system's ruby:

ExecStart=/var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production 

User=russell

Group=russell

UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage

# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/

Environment=MALLOC_ARENA_MAX=2

# if we crash, restart

RestartSec=1

Restart=on-failure

# output goes to /var/log/syslog

StandardOutput=syslog

StandardError=syslog

# This will default to "bundler" if we don't specify it

SyslogIdentifier=sidekiq

[Install]

WantedBy=multi-user.target

=============================================

This still fails   if I run   sudo journalctl -u sidekiq.service

==============================================

-- Logs begin at Thu 2019-06-06 20:16:58 UTC, end at Tue 2019-06-25 08:57:26 UTC. --

Jun 19 07:48:27 linode_drusound_6 systemd[1]: Started sidekiq.

Jun 19 07:48:27 linode_drusound_6 systemd[15401]: sidekiq.service: Failed to determine user credentials: No such process

Jun 19 07:48:27 linode_drusound_6 systemd[15401]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No s

Jun 19 07:48:27 linode_drusound_6 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER

Jun 19 07:48:27 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 19 07:48:28 linode_drusound_6 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.

Jun 19 07:48:28 linode_drusound_6 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 1.

Jun 19 07:48:28 linode_drusound_6 systemd[1]: Stopped sidekiq.

Jun 19 07:48:28 linode_drusound_6 systemd[1]: Started sidekiq.

Jun 19 07:48:28 linode_drusound_6 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER

Jun 19 07:48:28 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 19 07:48:29 linode_drusound_6 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.

Jun 19 07:48:29 linode_drusound_6 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 2.

Jun 19 07:48:29 linode_drusound_6 systemd[1]: Stopped sidekiq.

Jun 19 07:48:29 linode_drusound_6 systemd[1]: Started sidekiq.

Jun 19 07:48:29 linode_drusound_6 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER

Jun 19 07:48:29 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 19 07:48:30 linode_drusound_6 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.

Jun 19 07:48:30 linode_drusound_6 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 3.

Jun 19 07:48:30 linode_drusound_6 systemd[1]: Stopped sidekiq.

Jun 19 07:48:30 linode_drusound_6 systemd[1]: Started sidekiq.

Jun 19 07:48:30 linode_drusound_6 systemd[1]: sidekiq.service: Main process exited, code=exited, status=217/USER

Jun 19 07:48:30 linode_drusound_6 systemd[1]: sidekiq.service: Failed with result 'exit-code'.

Jun 19 07:48:31 linode_drusound_6 systemd[1]: sidekiq.service: Service hold-off time over, scheduling restart.

Jun 19 07:48:31 linode_drusound_6 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 4.

Jun 19 07:48:31 linode_drusound_6 systemd[1]: Stopped sidekiq.

===============================================

I can start up manually follows

===============================================

russell@linode_drusound_6:~$ /var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production -r /var/www/drusound_beta_6/current

2019-06-25T08:58:53.707Z 26552 TID-grk9f9714 INFO: Booting Sidekiq 5.2.7 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"app3_sidekiq_production", :id=>"Sidekiq-server-PID-26552"}

2019-06-25T08:58:56.148Z 26552 TID-grk9f9714 INFO: Running in ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

2019-06-25T08:58:56.149Z 26552 TID-grk9f9714 INFO: See LICENSE and the LGPL-3.0 for licensing details.

2019-06-25T08:58:56.149Z 26552 TID-grk9f9714 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org

2019-06-25T08:58:56.151Z 26552 TID-grk9f9714 INFO: Starting processing, hit Ctrl-C to stop

===============================================

Still stumbled thanks Russell

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/40A00036-509D-4DEF-9F1E-23D053E0254F%40drusound.com.

[Eric Chapman]

Hey Russell,

This is appearing to be an issue outside of Sidekiq and not with Sidekiq itself.  Do you know what is throwing the following error?  This looks like a Linux permission issue given you can manually start the process.

Jun 19 07:48:27 linode_drusound_6 systemd[15401]: sidekiq.service: Failed to determine user credentials: No such process

Jun 19 07:48:27 linode_drusound_6 systemd[15401]: sidekiq.service: Failed at step USER spawning /usr/local/bin/bundle: No s

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/A9928262-A137-46C8-BBA7-44D537369E2F%40drusound.com.

[rus...@drusound.com]

Hi Eric

I realise this might not be a specific sidekiq issue but its related to using sidekiq in production.

I don’t know how to work out what the issue is, which is why I posted on the forum

My user exists but which group do I assign in the configuration file?

Why is /usr/local/bin/bundle running in the config file I stated some thing else

Is there an issue with the configuration file.

If I can manually start the service using the same user I assume there is something wrong With the sidekiq.service file 

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAMnb1%3DvERiXxae-qoBDNZQ9GhweJ5pJoWN%3DqHg%2BO%3DC3ifejAsA%40mail.gmail.com.

[rus...@drusound.com]

Ok I ask a different way to my issue

I cannot start sidekiq.service on my ubuntu 18.04 but I can manually by login as Russell with ssh

 Groups on linux =  russell sudo rvm 

If I run the following, sidekiq starts

/var/www/drusound_beta_6/current/bin/bundle exec sidekiq -e production -r /var/www/drusound_beta_6/current

Is the following correct

WorkingDirectory=/var/www/drusound_beta_6/current

ExecStart=/var/www/drusound_beta_6/shared/bundle exec sidekiq -e production 

User=russell

Group= what do I put here? sudo ram or both?

Could some one also say if  the following service file is  ok

#
	# Sidekiq auto start using systemd unit file for Ubuntu 18.04
	#
	# Put this in /lib/systemd/system (Ubuntu).
	# Run:
	# 1. systemctl enable sidekiq (to enable sidekiq service)
	# 2. systemctl {start,stop,restart} sidekiq (to start sidekiq service)
	#

	# This file corresponds to a single Sidekiq process. Add multiple copies
	# to run multiple processes (sidekiq-1, sidekiq-2, etc).
	#
	# See Inspeqtor's Systemd wiki page for more detail about Systemd:
	# https://github.com/mperham/inspeqtor/wiki/Systemd
	#
	[Unit]
	Description=sidekiq
	# start us only once the network and logging subsystems are available,
	# consider adding redis-server.service if Redis is local and systemd-managed.
	After=syslog.target network.target
	# See these pages for lots of options:
	# http://0pointer.de/public/systemd-man/systemd.service.html
	# http://0pointer.de/public/systemd-man/systemd.exec.html
	[Service]
	Type=simple

	WorkingDirectory=/home/Public/Rails/your-rails-app
	# If you use rbenv or rvm:
	# ExecStart=/bin/bash -lc 'bundle exec sidekiq -e production'

# If you use the system's ruby:

	# ExecStart=bundle exec sidekiq -e production
	ExecStart=/home/deploy/.rbenv/bin/rbenv exec bundle exec sidekiq -e production -c 3 -q mailers -q default
	User=deploy
	Group=deploy
	UMask=0002

	# if we crash, restart
	RestartSec=1
	Restart=on-failure
	# output goes to /var/log/syslog
	StandardOutput=syslog
	StandardError=syslog
	# This will default to "bundler" if we don't specify it
	SyslogIdentifier=sidekiq
	[Install]
	WantedBy=multi-user.target

Thanks Russell