Threads and ActiveRecord

[Matt Smith]

Hi,

We have a system that is behaving somewhat abnormally or better yet
unexpectedly. We are using the rufus-scheduler to schedule tasks
within a rails application. These tasks each interact with
ActiveRecord going against Postgres. We see two things. One is
Postgres runs out of memory and two sometimes the inserts get mixed
up, i.e. one insert to table A actually tries the insert to table B.
I am thinking this is do to each task running in a separate thread but
utilizing a shared memory when it comes to ActiveRecord. Does this
make sense to you and have you experienced this before?

Not sure about the out of memory error, but the crossing of inserts
seems to be thread related.

Thanks,

Matt

[John Mettraux]

Hello Matt,

On Wed, May 21, 2008 at 10:46 PM, Matt Smith <codea...@gmail.com> wrote:
>
> We have a system that is behaving somewhat abnormally or better yet
> unexpectedly. We are using the rufus-scheduler to schedule tasks
> within a rails application. These tasks each interact with
> ActiveRecord going against Postgres. We see two things. One is
> Postgres runs out of memory and two sometimes the inserts get mixed
> up, i.e. one insert to table A actually tries the insert to table B.

What's the name of your ActiveRecord based class and what are the
names of table A and B ?

> I am thinking this is do to each task running in a separate thread but
> utilizing a shared memory when it comes to ActiveRecord. Does this
> make sense to you and have you experienced this before?

I haven't experienced that before. Your description of the "mixed up"
problem is really puzzling.

Have you considered using a Mutex ? Have you also checked the doc of
ActiveRecord::Base.allow_concurrency ?
http://github.com/rails/rails/tree/master/activerecord/lib/active_record/base.rb#L426-431

> Not sure about the out of memory error, but the crossing of inserts
> seems to be thread related.

Do you have a trace for this out of memory error ? Maybe there is
something to google about it.

Best regards,

--
John Mettraux - http://jmettraux.wordpress.com

[Matt Smith]

Hi John,

I will reply with more detail this evening.

Thanks,

Matt

On May 21, 9:04 am, "John Mettraux" <jmettr...@openwfe.org> wrote:
> Hello Matt,
>

> On Wed, May 21, 2008 at 10:46 PM, Matt Smith <codeaspe...@gmail.com> wrote:
>
> > We have a system that is behaving somewhat abnormally or better yet
> > unexpectedly.  We are using the rufus-scheduler to schedule tasks
> > within a rails application.  These tasks each interact with
> > ActiveRecord going against Postgres.  We see two things.  One is
> > Postgres runs out of memory and two sometimes the inserts get mixed
> > up, i.e.  one insert to table A actually tries the insert to table B.
>
> What's the name of your ActiveRecord based class and what are the
> names of table A and B ?
>
> > I am thinking this is do to each task running in a separate thread but
> > utilizing a shared memory when it comes to ActiveRecord.  Does this
> > make sense to you and have you experienced this before?
>
> I haven't experienced that before. Your description of the "mixed up"
> problem is really puzzling.
>
> Have you considered using a Mutex ? Have you also checked the doc of

> ActiveRecord::Base.allow_concurrency ?http://github.com/rails/rails/tree/master/activerecord/lib/active_rec...

[Matt Smith]

Hi John,

ActiveRecord::Base.allow_concurrency = true

That did the trick. We had multiple jobs/threads using ActiveRecord.
Without this setting we saw many errors. Thanks for the tip.

Also, what are your thoughts on http://www.infoq.com/news/2008/05/wfxml-r
?

Thanks,

Matt

[John Mettraux]

On Fri, May 30, 2008 at 8:37 AM, Matt Smith <codea...@gmail.com> wrote:
>
> ActiveRecord::Base.allow_concurrency = true
>
> That did the trick. We had multiple jobs/threads using ActiveRecord.
> Without this setting we saw many errors. Thanks for the tip.

Hello Matt,

cool, thanks for sharing the news :)

> Also, what are your thoughts on http://www.infoq.com/news/2008/05/wfxml-r
> ?

discussions :
http://groups.google.com/group/openwferu-dev/search?q=restful&start=0&scoring=d&
http://groups.google.com/group/openwferu-users/search?q=restful&start=0&scoring=d&

My vision of a RESTful thinggy :
http://jmettraux.wordpress.com/2008/05/07/restful-workflow-engine-on-sinatra/
My first attempt : http://github.com/jmettraux/kisha
Pat Cappelaere's work : http://github.com/cappelaere/ruote-web

And you, what do you think about wfxml-r ?

[Pat Cappelaere]

Matt,

There has been many attempts to integrating workflow engines.
WfXML-R is a RESTFul version of WfXML (SOAP).
The big drive is interoperability and leverage of other tools.
We (or may be John) will eventually support XPDL for model interchange from
BPMN tool designers. More work is happening at the Stevens Institute to
standardize the auditing function... More on that later...

There is an agreement in place between the Workflow Management Coalition and
the Open Geospatial Consortium to work together towards that end.
So hopefully this is not another stovepipe development and this brings
OpenWFE to the Enterprise.

Any feedback is very welcome: http://groups.google.com/group/wfxml

Pat.

[John Mettraux]

On Fri, May 30, 2008 at 10:35 AM, Pat Cappelaere <cappe...@gmail.com> wrote:
>
> We (or may be John) will eventually support XPDL for model interchange from
> BPMN tool designers.

"support" is a big word, especially with the word "enterprise" in the
same post. Here's what I've got on that front :

http://jmettraux.wordpress.com/2008/05/18/xpdl-for-fun/
http://github.com/jmettraux/ruote-transition

[Arjan van Bentem]

> ActiveRecord::Base.allow_concurrency = true
> That did the trick. We had multiple jobs/threads using ActiveRecord.

Just for the archives: apart from ActiveRecord::Base.allow_concurrency I
also explicitly needed to add some Mutex things to code using Net::HTTP
(to be more precise: OAuth), when executed by rufus-scheduler. Though I
was only using a single instance of the scheduler, my workflow
application also uses the same code. When an application user would
trigger this code while the scheduler was also running it, I happily
found the following in the log, making me realise I forgot about thread
safety:

Processing TaskController#update
...
Net::HTTPBadResponse (wrong status line: "e_end>2037-12-31
00:00:00</pub_date_end>"):
/usr/local/lib/ruby/1.8/net/http.rb:2031:in `read_status_line'
/usr/local/lib/ruby/1.8/net/http.rb:2018:in `read_new'
/usr/local/lib/ruby/1.8/net/http.rb:1059:in `request'

/usr/local/lib/ruby/gems/1.8/gems/oauth-0.2.4/lib/oauth/consumer.rb:101:
in `request'

/usr/local/lib/ruby/gems/1.8/gems/oauth-0.2.4/lib/oauth/token.rb:44:in
`request'

/usr/local/lib/ruby/gems/1.8/gems/oauth-0.2.4/lib/oauth/token.rb:79:in
`get'

Not caused by rufus-scheduler, of course!

Arjan.

[Arjan van Bentem]

As an additional note: this weekend I also needed to set

config.cache_classes = true

in development.rb on my Mac, as suddenly ActiveRecord was giving me
trouble. While rufus-scheduler was still running some major update, then
as soon as I would open the first screen in my browser, rufus-scheduler
would immediately get

trigger() caught exception
stack level too deep

or

trigger() caught exception
undefined method `countries=' for #<BusinessPlan:0x38a0b58>

(here, "countries=" is an ActiveRecord auto-generated method for an
HABTM association; the exception also occurred for other auto-generated
methods -- until the exception occurred ActiveRecord surely knew about
those methods, as it was populating the MySQL database just fine)

I could also see that some module I use in both the web pages and the
scheduler was being initialised a second time. That is probably some
major flaw in my code, but the problem took me to a Rails 2.0.2 bug at
http://dev.rubyonrails.org/ticket/10896 The scary workaround to enable
caching helps, as far as I can tell...

Now, maybe this might be relevant for the Oracle sequence problem? Maybe
ActiveRecord is also caching its sequences?

A few more maybes, just for the archives:

Maybe for me the sudden change is due to starting rufus-scheduler from
enviroment.rb (earlier I started it in ApplicationController, which of
course needs someone to request some web page to get it). Or, maybe it
is caused by running the scheduler much earlier now (rather than only
after 1 hour, the first run is now started before anyone probably
requested some web page). Or, maybe it is because I added
"ActiveRecord::Base.allow_concurrency = true" last week and since had
not opened a web page while rufus-scheduler was running. Or, maybe I
only noticed now because I set up a specific log file for the
scheduler...

Arjan.

-----Original Message-----
From: Arjan van Bentem
Sent: Friday, May 30, 2008 10:46 AM
To: OpenWFEru users
Subject: RE: [openwferu-users] Re: Threads and ActiveRecord

> ActiveRecord::Base.allow_concurrency = true
> That did the trick. We had multiple jobs/threads using ActiveRecord.

Just for the archives: apart from ActiveRecord::Base.allow_concurrency I