Phusion Passenger 4.0.57 - Smart spawning method issue?

[Suresh Karuppannan]

Recently we upgraded Phusion Passenger from 3.0.19 to 4.0.57, Ruby from 1.9.3 to 2.1.5 and our Rails (3.2.19) application started to behave in weired manner on activerecord part. From the application log, we could see there are some invalid SQL queries like following got generated while the app runs on full traffic. We tried to reproduce and the issue is consistent on heavy traffic but not on low traffic.

1. ActiveRecord::StatementInvalid (Mysql2::Error: [34816] syntax error: improper query string termination: SELECT (End of query)
   
2. ActiveRecord::StatementInvalid (Mysql2::Error: [8192] Relation not found: no such relation ": SELECT ` (End of query)
   
3. Mysql2::Error: [32768] Translate error: SQL-ERROR: unable to resolve relation "": SELECT agents.* FROM agents WHERE ``.id = 22241082 LIMIT 1
   
4. ActiveRecord::StatementInvalid (Mysql2::Error: [32769] Unknown column: SQL-ERROR: unable to resolve column "agents.: SELECT  agents.* FROM agents WHERE agents.` (end of query)
   
5. One update query with where 0 = 0
   
6. One select query without where clause whereas app code supplies the conditions.
   

After spending hours with Passenger configurations, we narrowed down the issue to "PassengerSpawnMethod". After setting this value to "direct" instead of "smart", the issue is no more. In 3.0.19, we didn't set this config and app was working great. I couldn't see this issue reported anywhere. My guess is multiple processes are writing into the same database socket connection leads to invalid SQL statement(?). Does anyone else has encountered this issue? Do I have to open a bug?

[Hongli Lai]

The smart spawning method indeed makes it possible for multiple processes to write to the same database socket connection, causing protocol corruption. The issue is described here: https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#_smart_spawning_caveat_1_unintentional_file_descriptor_sharing

However, Passenger in smart spawning mode will automatically reestablish the ActiveRecord connection whenever a new application process is spawned. So this issue should normally not occur. Maybe you are using ActiveRecord in such a way that ActiveRecord creates multiple database connections. Passenger only reestablishes the main ActiveRecord connection, not additional ones.

--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phusion-passen...@googlegroups.com.
To post to this group, send email to phusion-...@googlegroups.com.
Visit this group at http://groups.google.com/group/phusion-passenger.
To view this discussion on the web visit https://groups.google.com/d/msgid/phusion-passenger/557da8a8-eff5-4dd1-9638-a1681934b31d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Phusion | Web Application deployment, scaling, and monitoring solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Suresh Karuppannan]

Hi Hongli, thanks for the response.

The problem continues to persist even in direct spawn method after running load test for 3x time than smart span method.

As we upgraded Ruby & Passenger together, to isolate between ruby and passenger we setup load test and executed load tests for each of the following combinations multiple times in order to reproduce the issue consistently.

New Ruby (2.1.5) & New Passenger(4.0.57)	Reproducible
Old Ruby (1.9.3) & New Passenger(4.0.57)	Reproducible
New Ruby (2.1.5) & Old Passenger(3.0.19)	Not Reproducible
Old Ruby (1.9.3) & Old Passenger(3.0.19)	Not Reproducible

 

With the help of above matrix and its corresponding results, we identified that the latest version of Passenger was causing these issues. But we are clueless about whether the issue is internal to Passenger or some of our app configurations that trigger latest Passenger to behave weirdly. Could you tell us on how to debug this further?

[Hongli Lai]

I can't remember a change between 3.x and 4.x that would cause something like this. 3.x was a long time ago and as far as I remember, nothing changed w.r.t. file descriptor inheritance.

Do you know whether your app uses multiple ActiveRecord database connections? You can scan for establish_connection in your codebase. It might even be caused by a gem.