Possible bug since 0.16.

577 views
Skip to first unread message

Fernando Lujan

unread,
Oct 3, 2013, 10:30:35 AM10/3/13
to rub...@googlegroups.com
Hello, 

I am having error messages like:

usr/local/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec': PG::QueryCanceled: ERROR:  canceling statement due to user request (ActiveRecord::StatementInvalid)

Since I upgrade from 0.15 to 0.16. Version 0.17 also have the same error. I check the commits on bitbucket but cannot find what is causing this issue. 

Thanks in advance, 

Fernando Lujan

Lars Kanis

unread,
Oct 3, 2013, 3:05:19 PM10/3/13
to rub...@googlegroups.com
Am 03.10.2013 16:30, schrieb Fernando Lujan:
> usr/local/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in
> `exec': PG::QueryCanceled: ERROR: canceling statement due to user
> request (ActiveRecord::StatementInvalid)

Could you please describe your environment and when the error is raised
somewhat more? Is there a way to reproduce the issue?

--
Regards,
Lars

Fernando Lujan

unread,
Oct 3, 2013, 4:13:23 PM10/3/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Hello Lars, 

Thank you for pg and for the quick reply. It does not occurs consistently. I have a script that each 2 seconds connects to the database and issue a query using active record. I can easy reproduce it, just start the script and after sometime it break. 


Do you need me to run it with some especial parameter for debug purposes? On version 0.15.0 it is pretty stable.

Best Regards,

Fernando Lujan

Fernando Lujan

unread,
Oct 4, 2013, 8:18:49 AM10/4/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Lars, 

Do you have a procedure to install ruby-pg from source? I will try to isolate witch commit cause the problem. 

Thanks, 

Fernando Lujan

Fernando Lujan

unread,
Oct 15, 2013, 12:49:58 PM10/15/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Lars, 

I found that two commits are causing the isssue: 

d4cf78e1f1ae66ad199fc8cc227f9578f1542278

All versions of PostgreSQL starting from 8.4 know DIAG_STATEMENT_POSITION and 8.3 is EOL.

and ad1b1794bc5f9ac0dd9784379996e99430029a4d

So, the last version of pg that supports Posgtresql 8.3 and 8.4 will be 0.15.1? 

Thanks in advance, 

fabian....@gmail.com

unread,
Nov 13, 2013, 6:42:40 AM11/13/13
to rub...@googlegroups.com
The same error occurs for me when I'm doing zero-downtime deployment of a rails app with unicorn. My versions are pg (0.17.0) and postgresql 
9.2.4. The following code is executed when a new version is deployed:

before_fork do |server, worker|
  # Disconnect since the database connection will not carry over
  if defined? ActiveRecord::Base
    ActiveRecord::Base.connection.disconnect!
  end

  # Quit the old unicorn process
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  # Start up the database connection again in the worker
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
end

I think the error occurs when disconnecting from the database: `ActiveRecord::Base.connection.disconnect!` while a query is carried out. I'm not sure if this is a bug in the pg or rails.

Thanks in advance!

Lars Kanis

unread,
Nov 14, 2013, 5:41:14 AM11/14/13
to rub...@googlegroups.com

I think the error occurs when disconnecting from the database: `ActiveRecord::Base.connection.disconnect!` while a query is carried out. I'm not sure if this is a bug in the pg or rails.

That makes sense. It is new since pg-0.16.0, that a query is canceled, when the thread in which the query is executed is about to be killed. The exec method then throws a PG::QueryCanceled error.

Regards,
Lars

joev...@gmail.com

unread,
Nov 18, 2013, 9:36:56 PM11/18/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Is there a way to work around this? If I have an active query running when using unicorn to do a zero-downtime deployments, I don't really want to kill the active queries. Ideally the page requests should finish.

Joe

Michael Granger

unread,
Nov 19, 2013, 1:57:36 PM11/19/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de, joev...@gmail.com
On Monday, November 18, 2013 6:36:56 PM UTC-8, joev...@gmail.com wrote:
Is there a way to work around this? If I have an active query running when using unicorn to do a zero-downtime deployments, I don't really want to kill the active queries. Ideally the page requests should finish. 

There might be, but it would have to come from Unicorn, since it's managing the termination of the old process and startup of the new one. There's no way the database driver can manage that. 

 

joev...@gmail.com

unread,
Nov 19, 2013, 2:07:57 PM11/19/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de, joev...@gmail.com
Hm, earlier versions of ruby-pg didn't have this problem though, as far as I can tell.

Joe 

Sam Saffron

unread,
Dec 7, 2013, 8:06:35 AM12/7/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de, joev...@gmail.com
I discussed the ubf being used now by pg with tmm1 and ko1 both think it is buggy. (https://github.com/SamSaffron/pg/commit/2b87400c9ddb576e91984a0318ace09b75e88644

A ubf is unconditionally invoked if ANY signal fires on a process (eg even: USR1 or PROF) and Ruby has handlers registered. Cancelling queries in those conditions does no make sense. The standard ubf (RUBY_UBF_IO) does not terminate the running thread it just issues it VTALRM which forces scheduling.  

There must be another api out there for handling Thread.kill, in fact http://ruby-doc.org/core-2.0.0/Thread.html#method-c-handle_interrupt would probably do for the purpose the ubf was introduced for.

It is totally messing up my flamegraphs http://samsaffron.com/archive/2013/03/19/flame-graphs-in-ruby-miniprofiler (even though I have a much cleaner way in 2.1 to get the stack traces) 

Lars Kanis

unread,
Dec 7, 2013, 9:03:56 AM12/7/13
to rub...@googlegroups.com
Am 07.12.2013 14:06, schrieb Sam Saffron:
> I discussed the ubf being used now by pg with tmm1 and ko1 both think
> it is buggy.
> (https://github.com/SamSaffron/pg/commit/2b87400c9ddb576e91984a0318ace09b75e88644)
>
> A ubf is unconditionally invoked if ANY signal fires on a process (eg
> even: USR1 or PROF) and Ruby has handlers registered. Cancelling
> queries in those conditions does no make sense. The standard ubf
> (RUBY_UBF_IO) does not terminate the running thread it just issues it
> VTALRM which forces scheduling.
>
> There must be another api out there for handling Thread.kill, in
> fact http://ruby-doc.org/core-2.0.0/Thread.html#method-c-handle_interrupt would
> probably do for the purpose the ubf was introduced for.
>
> It is totally messing up my
> flamegraphs http://samsaffron.com/archive/2013/03/19/flame-graphs-in-ruby-miniprofiler (even
> though I have a much cleaner way in 2.1 to get the stack traces)

Thank you for this great analysis and discussion you did! I already was
afraid that canceling in the UBF is somewhat too overeager...

I'll look for an alternative approach to this issue. Do you have an
proposal already? Or did tmm1 or ko1 propose something?

--
Regards,
Lars

Sam Saffron

unread,
Dec 9, 2013, 1:28:33 AM12/9/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Had a discussion today with ko1 and tmm about this, based on this sample: https://gist.github.com/SamSaffron/7868096 

The very unfortunate thing is that there is no way to use ubf's in the way they are attempted to be consumed. To quote ko1 "I never think about it" 

The recommended way to resolve this is to use the non-blocking apis in libpq as opposed to the blocking ones. 

If that is done after issuing a query we could rb_wait_for_single_fd on the socket before calling the non blocking checks. 

see: PQconnectStart PQconnectPoll and PQsendQuery , PQisBusy etc.

Lars Kanis

unread,
Dec 9, 2013, 2:14:27 AM12/9/13
to rub...@googlegroups.com

Can you forward your discussions? It would be nice to get somewhat more information regarding to the UBF and how ko1 thought it should be used.

Regards,
Lars

Lars Kanis

unread,
Dec 9, 2013, 10:26:14 AM12/9/13
to rub...@googlegroups.com
I did a closer look at the issue now. For the sample you wrote, it is easy to solve: https://gist.github.com/larskanis/7872229 But I couldn't find a similar mechanism we could use in libpq except cancelation of the query.

There seems to be no reasonable way to check for the presence of a trap handler within the UBF. In this case we could just ignore the UBF-request, so that it's delivered after the query finished. The issue with Thread#handle_interrupt is obviously the notice "signal trap (not supported yet)" in the documentation. But anyway, both variants are not really correct, IMHO.

The async API is used for ruby-1.8 but still works on 1.9+, of course. We switched to the synchronous API because of speed and reliability. This is why I'm not really pleased to switch back.

So for what I can tell now, the only way we can correctly work together with signal handling, is actually to switch back to the async API at least for Connection#async_exec. Nevertheless it would be nice if you could forward the responses of ko1 and tmm1.

--
Regards,
Lars

Lars Kanis

unread,
Dec 9, 2013, 12:03:48 PM12/9/13
to rub...@googlegroups.com
I pushed https://bitbucket.org/ged/ruby-pg/pull-request/18 for review.

--
Regards,
Lars

Sam Saffron

unread,
Dec 9, 2013, 7:13:32 PM12/9/13
to rub...@googlegroups.com, la...@greiz-reinsdorf.de
Lars, 

Thanks heaps, 

I think UBFs are horribly misunderstood, took me ages to figure this out. The only intention there is to unblock custom io, once unblocked you need to be able to re-aquire it is not designed for "black hole" api calls. 

Not much in the conversation to share really, when I explained we need the cancel reason ko1 said that he never thought about it, it was a reasonable request, but pg should not cancel, I have pointed them back at here. 

As to the perf and stability issues, do you have any examples? I would like to have a look.

If we need some extra messaging from ubfs I think it makes sense to raise a ticket in redmine with all the details. 

There has been an unfortunate flow through effect in unicorn see: http://rubyforge.org/pipermail/mongrel-unicorn/2013-December/001979.html

I will point Eric back here so he can have full context.
Reply all
Reply to author
Forward
0 new messages