Help With Some Puma Warnings

[Rob Whittaker]

Hey, everybody!

I have a side project of getting my first proper Rails app up to date. The last time I touched the project was about six years ago. Ruby, Rails, and all the other gems are out of date. I'm trying to be sensible and "safe" in my approach to each upgrade. This is more of an exercise in approach than anything else. The repo is public for folks who want to have a look.

After a chat with Will at the previous NWRUG, I decided to fix any warnings before moving on. This is a sensible and logical plan. The latest thing on my list is this list of warnings from Puma.

07:33:22 web.1      |  [25567] ! WARNING: Detected 5 Thread(s) started in app boot:

07:33:22 web.1      |  [25567] ! #<Thread:0x000000013acd2090@/Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:300 sleep> - /Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:302:in `sleep'

07:33:22 web.1      |  [25567] ! #<Thread:0x000000013d392360@listen-wait_thread@/Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/listen-3.8.0/lib/listen/thread.rb:17 sleep_forever> - /Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/2.5.0/forwardable.rb:229:in `pop'

07:33:22 web.1      |  [25567] ! #<Thread:0x000000013b7d1ba8@listen-worker_thread@/Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/listen-3.8.0/lib/listen/thread.rb:17 sleep> - /Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/rb-fsevent-0.11.2/lib/rb-fsevent/fsevent.rb:44:in `select'

07:33:22 web.1      |  [25567] ! #<Thread:0x000000013bef6b90@listen-run_thread@/Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/listen-3.8.0/lib/listen/thread.rb:17 sleep> - /Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/listen-3.8.0/lib/listen/record/entry.rb:44:in `realpath'

07:33:22 web.1      |  [25567] ! #<Thread:0x000000013bef46b0@listen-wait_thread@/Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/gems/2.5.0/gems/listen-3.8.0/lib/listen/thread.rb:17 sleep_forever> - /Users/purinkle/.asdf/installs/ruby/2.5.9/lib/ruby/2.5.0/forwardable.rb:229:in `pop'

I only see these warnings when I start the server using heroku local. If I start the server using rails s, then everything is okay.

There is no pressure here; I want to use this repo as a learning space. I want to understand the problem more than anything else. Is this something that I can fix? Is it something that a future gem bump will fix? How would you approach debugging this problem?

—Rob

[Lee Hambley]

WARNING: Detected 5 Thread(s) started in app boot:

I suspect this is because threads behave like file handles, and forked processes inherit and share file handles. In the Puma case, I'd imagine that this means you have `n` processes sharing the same thread for AR connection pool, some file watcher (4x?).

In theory if none of the child processes close the file handle it's probably fine. After all stdout/err are file handles, and they're shared. The warning is probably however that if any of the child processes close or otherwise mangle the file handles then the other processes are starved.

In a setup where you want 10 connections to AR per worker, this would also cause problems you'd get 10 total to share.

I might be overthinking, and it's just a warning because in _theory_ this is a antipattern because your processes are less independent.

As for how to fix it, I'd guess "don't' use Puma" or at least don't use a pre-forking web server? Or upgrade rails and/or the Gems... I think Rails did better in later versions at giving a "post-fork" hook for thread- and worker-pools to be initialized.

Ahoy, HTH

Lee Hambley

http://lee.hambley.name/

+49 (0) 170 298 5667

--
You received this message because you are subscribed to the Google Groups "North West Ruby User Group (NWRUG)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nwrug-member...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nwrug-members/eb3ea539-eee2-49ce-a378-e9eb6e74ccb5n%40googlegroups.com.

[Tekin Süleyman]

I wouldn’t worry too much about this. Puma warns about threads that are spawned during app boot and before it forks workers as those threads can be unsafe/lead to issues, but the threads listed are almost all from the listen gem, which is a development dependency, and I can't imagine ActiveRecord will be spawning threads that are not safe for forking. I wouldn’t be surprised if the issue goes away when you update dependencies.

It’s interesting that you don’t see the same warnings with `rails server`. Is it actually booting using Puma and not another web server?

Tekin

[Rob Whittaker]

Thank you both for your comments.

Much of what Lee said went over my head and made me realise how little I know about Puma. I have always accepted that it's the tool to use since it gained popularity. Some stuff went in, but now I want to learn more.

After Tekin's comments about rails s, I did some more investigating and was wrong. Running rails s produces no warnings. Running bundle exec puma as per the Procfile gives warnings, though.

Puma loads the config/puma.rb file by default. I combed through that file and commented out lines until the warnings disappeared. The offending line has something to do with workers. If I set the value to zero, then the warnings disappear.

I set the number of workers from environment variables. When we have no value set, it defaults to two workers. I have this value set to zero in my .env file. I thought it might be that I was using 0, and the environment saw this as no value set. I tried setting it to "0", but still no dice. It was time for a trusty raise. No matter what value I put in .env, there was nothing in my error. Weird.

Then, I found this question on Stack Overflow. It makes sense that we load dotenv after Puma. The gem is part of the Rails stack, after all. What could I do?

My next step was to create a separate Procfile.dev with the following line and a bin/dev wrapper.

web: WEB_CONCURRENCY=0 bundle exec puma -p $PORT -C ./config/puma.rb

This approach seems like it could be better. I'll have to duplicate the commands between my two files and now use bin/dev to start my server. With the extra information I've provided, can anybody give a better solution?

[Will Jessop]

I highly recommend installing direnv (https://direnv.net/, available in homebrew) instead of using the dotenv gem and .env files. I've seen the sort of confusion you have had /repeatedly/ over the years with dotenv as it doesn't actually provide an environment at all, it's mis-named. direnv actually provides a first-class environment to your application so any command you run is in that env from program start:

will@lentil ~/www/oas/master (on 1e5da09)% cat .envrc
export REDIS_URL=redis://localhost:6379
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
export RUBYOPT=-w
# Temporary for segfault https://github.com/ged/ruby-pg/issues/538
export PGGSSENCMODE="disable"
will@lentil ~/www/oas/master (on 1e5da09)% env | grep RUBYOPT
RUBYOPT=-w

[Rob Whittaker]

You make some excellent points, Will. Your approach is something I will consider over the longer term. I plan to pinch these lines from Rails' default Puma config. People more intelligent than me have already solved the problem.

[Lee Hambley]

+10 to what Will said. Using a _Gem_ to load env config is a pretty big violation of the "outside in" principal of environment variables. In principal environment variables are mapped into a processes memory space.

From Wikipedia:

In all Unix and Unix-like systems, as well as on Windows, each process has its own separate set of environment variables. By default, when a process is created, it inherits a duplicate run-time environment of its parent process, except for explicit changes made by the parent when it creates the child. At the API level, these changes must be done between running fork and exec. Alternatively, from command shells such as bash, a user can change environment variables for a particular command invocation by indirectly invoking it via env or using the ENVIRONMENT_VARIABLE=VALUE <command> notation. A running program can access the values of environment variables for configuration purposes.
-- https://en.wikipedia.org/wiki/Environment_variable

In theory if you argue that you load env vars before the fork on a preforking webserver, maybe it's _not_ breaking core principles of unix-like OSs but it's really bad practice for a process to read it's own "env vars" (it can load config files all day long, but env config is a specific thing with specific semantics)

Lee Hambley

http://lee.hambley.name/

+49 (0) 170 298 5667

--
You received this message because you are subscribed to the Google Groups "North West Ruby User Group (NWRUG)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nwrug-member...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/nwrug-members/32bb2225-1a24-43ca-803b-52173fe2649fn%40googlegroups.com.

[Rob Whittaker]

It may be the Baader-Meinhof effect, but I've seen more mentions of direnv over the last day than I have ever before. I've added a card to my backlog to investigate swapping out dotenv.

Thank you all for all your help. You've pointed me in the right direction.

If any of you wanted to give a talk at NWRUG about Puma I would love it. If you wanted to explain Puma to me, I would love to give that talk to aid my learning.

[Will Jessop]

To save you some time it should just be a case of:

1. installing direnv (brew install direnv)
2. move the .env file contents to the .envrc in the project root (with some minor formatting tweaks)
3. running `direnv allow` in the project root

Unless you're using envrc in production (which I wouldn't, and can't remember ever seeing) then that should be everything you need to change. In production there would be no changes if you're setting env vars in the normal way (eg. "settings" in Heroku, systemd config etc.). If you /are/ using envrc in production then you might find the config to be better moved to those "normal" ways of setting the env, initialisers or the credentials store.

Will.