running background jobs with neo4j.rb rails?

163 views
Skip to first unread message

arikan

unread,
Jan 25, 2012, 12:42:02 PM1/25/12
to neo...@googlegroups.com
Hi there, 

To run background jobs in rails/neo4j.rb I tried the Redis-backed library "resque" https://github.com/defunkt/resque

worker:
class BackgroundTest
  @queue = :background_test_queue
  
  def self.perform(id)
    puts(file)
    hub = Person.find id
    puts person.name
  end
end

When a test worker (above) is called from a controller (below), a second Neo4j instance starts in a readonly mode, so you can not update any models this way. 

in controller:
Resque.enqueue(BackgroundTest, @person.id)

console:
Enable remote shell at port port=9332
Starting Neo4j in readonly mode since the /Users/arikan/Sites/graphcp/db/neo4j-development is locked
Peter

Is there any way to use background jobs in Neo4j.rb?

Thanks,
Burak

Peter Neubauer

unread,
Jan 25, 2012, 1:30:16 PM1/25/12
to neo...@googlegroups.com
Mmh,
in order to get different processes to work together, I think Neo4j HA
would be the best option. Set up a cluster, and then put the updater
job on one of them, and the rails app on the other.

Would that work?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/neo4jrb/-/GoyLm9GNhYAJ.
> To post to this group, send email to neo...@googlegroups.com.
> To unsubscribe from this group, send email to
> neo4jrb+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/neo4jrb?hl=en.

arikan

unread,
Jan 25, 2012, 1:50:34 PM1/25/12
to neo...@googlegroups.com
Tried batch_insert as a background job, which also didn't work. 

class BackgroundInsert
  @queue = :background_insert_queue
  
  def self.perform
    Neo4j.shutdown
    inserter = Neo4j::Batch::Inserter.new    
    src_node = inserter.create_node({'name'=>'Peter'}, Person)
    trg_node = inserter.create_node({'name'=>'Nokia'}, Organization)
    inserter.create_rel("likes", src_node, trg_node)
    inserter.shutdown
    Neo4j.start
  end
end

It can shut down Neo4j, but the Inserter throws the error below. So "resque" background jobs always start another Neo4j instance (since they load the Rails environment).  
  
java.lang.IllegalStateException: Unable to lock store [/Users/arikan/Sites/graphcp/db/neo4j-development/neostore], this is usually a result of some other Neo4j kernel running using the same store.
org/neo4j/kernel/impl/nioneo/store/CommonAbstractStore.java:174:in `checkStorage'
org/neo4j/kernel/impl/nioneo/store/CommonAbstractStore.java:102:in `<init>'
org/neo4j/kernel/impl/nioneo/store/AbstractStore.java:119:in `<init>'
org/neo4j/kernel/impl/nioneo/store/NeoStore.java:78:in `<init>'
org/neo4j/kernel/impl/batchinsert/BatchInserterImpl.java:118:in `<init>'
sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
sun/reflect/NativeConstructorAccessorImpl.java:39:in `newInstance'
sun/reflect/DelegatingConstructorAccessorImpl.java:27:in `newInstance'
java/lang/reflect/Constructor.java:513:in `newInstance'
org/jruby/javasupport/JavaConstructor.java:291:in `newInstanceDirect'
/Users/arikan/.rvm/gems/jruby-1.6.4/gems/neo4j-1.3.1-java/lib/neo4j/batch/inserter.rb:24:in `initialize'
/Users/arikan/Sites/graphcp/app/workers/background_insert.rb:7:in `perform'

Peter Neubauer

unread,
Jan 25, 2012, 1:53:09 PM1/25/12
to neo...@googlegroups.com
Yes,
if you have an instance running against the same store files, the
second instance cannot start in embedded mode. Try setting up a
cluster and then let Neo4j do the background synch work.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/neo4jrb/-/4i6_duwUvIsJ.

Andreas Ronge

unread,
Jan 25, 2012, 3:17:57 PM1/25/12
to neo...@googlegroups.com

Hi

I suggest using threads instead of processes to run background jobs. In one project we use http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger

Another approach is to expose a rest api which an external  background job could use.

sent from my phone

--
You received this message because you are subscribed to the Google Groups "neo4jrb" group.
To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/GoyLm9GNhYAJ.

Dmytrii Nagirniak

unread,
Jan 25, 2012, 8:01:47 PM1/25/12
to neo...@googlegroups.com, neo...@googlegroups.com
I second Andreas here.
The easiest option is to run embedded scheduler. If you need higher reliability, then you can also use TorqueBox.

But the moment you need to scale out, you will have to go with neo4j HA.

arikan

unread,
Jan 26, 2012, 7:19:12 AM1/26/12
to neo...@googlegroups.com
Simply tried to run the Batch::Inserter in a Thread from a rails controller:  

def import
   @hub = Hub.find(params[:id])
   t1 = Thread.new do
      puts "opened a thread"
      Hub.batch_import(params[:csv_file], @hub.id.to_i) # batch inserts here 
      puts "closing the thread"
   end
   t1.join
   respond_to do |format|
     format.html
    end
end
    
Hub.batch_import method shuts down Neo4j, starts Batch::Inserter, does the inserts, shuts down the Inserter, and starts Neo4j again. This thread ends successfully. But then while trying to redirect_to or render a view, it throws an error. 

Any idea how this happens, or recommendations for thread based background job gems for Jruby (there seems to be many process based ones)?
ActionView::Template::Error (java.lang.NullPointerException: null):
    11: 
    12:   <div id="main">
    13:     
    14:     <div id="<%= user_signed_in? ? "toolbar" : "toolbar-notloggedin" %>"></div>
    15:       
    16:     <div id="header">
    17:       <div class="left">
 app/views/layouts/application.html.erb:14:in `_app_views_layouts_application_html_erb__774776397_13334'
 app/controllers/hubs_controller.rb:112:in `import'

Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/_trace.erb (30.0ms)
Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (19.0ms)
Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/template_error.erb within rescues/layout (65.0ms)

Matthew Ford

unread,
Jan 30, 2012, 5:51:26 PM1/30/12
to neo...@googlegroups.com
Hi, 

I was wondering if there are any examples of getting a Neo4J HA cluster setup with Rails 3. Currently running into an issue where I'd like to run a background task on collecting data from an API.

Cheers,
Matt

Andreas Ronge

unread,
Jan 31, 2012, 2:54:16 AM1/31/12
to neo...@googlegroups.com
Hi

Yes, there is a very simple example here -
https://github.com/andreasronge/neo4j/tree/master/example/ha-cluster
Regarding Rails 3 and Neo4j you probably should check the HAProxy docs
- http://docs.neo4j.org/chunked/snapshot/ha-haproxy.html
Not sure if it's a good idea to use neo4j HA cluster just because you
need to run background tasks.
Maybe it's better to provide your own rest API or using
http://torquebox.org/ (see previous mails).

Cheers
Andreas

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/neo4jrb/-/OvzgA1pJ5JQJ.

Vivek Prahlad

unread,
Jan 31, 2012, 3:35:40 AM1/31/12
to neo...@googlegroups.com
On my current project, we used Resque for a background jobs. We built a simple REST Api for the neo4j.rb application, and we invoke the API as part of the Resque job. That way, we avoid having to worry about things like multiple connections to the same database. This approach can also be extended in the future to support database sharding, for example.

Cheers,
Vivek
Reply all
Reply to author
Forward
0 new messages