running background jobs with neo4j.rb rails?

Showing 1-10 of 10 messages
running background jobs with neo4j.rb rails? arikan 1/25/12 9:42 AM
Hi there, 

To run background jobs in rails/neo4j.rb I tried the Redis-backed library "resque" https://github.com/defunkt/resque

worker:
class BackgroundTest
  @queue = :background_test_queue
  
  def self.perform(id)
    puts(file)
    hub = Person.find id
    puts person.name
  end
end

When a test worker (above) is called from a controller (below), a second Neo4j instance starts in a readonly mode, so you can not update any models this way. 

in controller:
Resque.enqueue(BackgroundTest, @person.id)

console:
Enable remote shell at port port=9332
Starting Neo4j in readonly mode since the /Users/arikan/Sites/graphcp/db/neo4j-development is locked
Peter

Is there any way to use background jobs in Neo4j.rb?

Thanks,
Burak
Re: running background jobs with neo4j.rb rails? Peter Neubauer 1/25/12 10:30 AM
Mmh,
in order to get different processes to work together, I think Neo4j HA
would be the best option. Set up a cluster, and then put the updater
job on one of them, and the rails app on the other.

Would that work?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/neo4jrb/-/GoyLm9GNhYAJ.
> To post to this group, send email to neo...@googlegroups.com.
> To unsubscribe from this group, send email to
> neo4jrb+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/neo4jrb?hl=en.

Re: running background jobs with neo4j.rb rails? arikan 1/25/12 10:50 AM
Tried batch_insert as a background job, which also didn't work. 

class BackgroundInsert
  @queue = :background_insert_queue
  
  def self.perform
    Neo4j.shutdown
    inserter = Neo4j::Batch::Inserter.new    
    src_node = inserter.create_node({'name'=>'Peter'}, Person)
    trg_node = inserter.create_node({'name'=>'Nokia'}, Organization)
    inserter.create_rel("likes", src_node, trg_node)
    inserter.shutdown
    Neo4j.start
  end
end

It can shut down Neo4j, but the Inserter throws the error below. So "resque" background jobs always start another Neo4j instance (since they load the Rails environment).  
  
java.lang.IllegalStateException: Unable to lock store [/Users/arikan/Sites/graphcp/db/neo4j-development/neostore], this is usually a result of some other Neo4j kernel running using the same store.
org/neo4j/kernel/impl/nioneo/store/CommonAbstractStore.java:174:in `checkStorage'
org/neo4j/kernel/impl/nioneo/store/CommonAbstractStore.java:102:in `<init>'
org/neo4j/kernel/impl/nioneo/store/AbstractStore.java:119:in `<init>'
org/neo4j/kernel/impl/nioneo/store/NeoStore.java:78:in `<init>'
org/neo4j/kernel/impl/batchinsert/BatchInserterImpl.java:118:in `<init>'
sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
sun/reflect/NativeConstructorAccessorImpl.java:39:in `newInstance'
sun/reflect/DelegatingConstructorAccessorImpl.java:27:in `newInstance'
java/lang/reflect/Constructor.java:513:in `newInstance'
org/jruby/javasupport/JavaConstructor.java:291:in `newInstanceDirect'
/Users/arikan/.rvm/gems/jruby-1.6.4/gems/neo4j-1.3.1-java/lib/neo4j/batch/inserter.rb:24:in `initialize'
/Users/arikan/Sites/graphcp/app/workers/background_insert.rb:7:in `perform'
Re: running background jobs with neo4j.rb rails? Peter Neubauer 1/25/12 10:53 AM
Yes,
if you have an instance running against the same store files, the
second instance cannot start in embedded mode. Try setting up a
cluster and then let Neo4j do the background synch work.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/neo4jrb/-/4i6_duwUvIsJ.

>
> To post to this group, send email to neo...@googlegroups.com.
> To unsubscribe from this group, send email to
> neo4jrb+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/neo4jrb?hl=en.

Re: running background jobs with neo4j.rb rails? ronge 1/25/12 12:17 PM

Hi

I suggest using threads instead of processes to run background jobs. In one project we use http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger

Another approach is to expose a rest api which an external  background job could use.

sent from my phone

--
You received this message because you are subscribed to the Google Groups "neo4jrb" group.
To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/GoyLm9GNhYAJ.

To post to this group, send email to neo...@googlegroups.com.
To unsubscribe from this group, send email to neo4jrb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/neo4jrb?hl=en.
Re: running background jobs with neo4j.rb rails? dnagir 1/25/12 5:01 PM
I second Andreas here.
The easiest option is to run embedded scheduler. If you need higher reliability, then you can also use TorqueBox.

But the moment you need to scale out, you will have to go with neo4j HA.
Re: running background jobs with neo4j.rb rails? arikan 1/26/12 4:19 AM
Simply tried to run the Batch::Inserter in a Thread from a rails controller:  

def import
   @hub = Hub.find(params[:id])
   t1 = Thread.new do
      puts "opened a thread"
      Hub.batch_import(params[:csv_file], @hub.id.to_i) # batch inserts here 
      puts "closing the thread"
   end
   t1.join
   respond_to do |format|
     format.html
    end
end
    
Hub.batch_import method shuts down Neo4j, starts Batch::Inserter, does the inserts, shuts down the Inserter, and starts Neo4j again. This thread ends successfully. But then while trying to redirect_to or render a view, it throws an error. 

Any idea how this happens, or recommendations for thread based background job gems for Jruby (there seems to be many process based ones)?
ActionView::Template::Error (java.lang.NullPointerException: null):
    11: 
    12:   <div id="main">
    13:     
    14:     <div id="<%= user_signed_in? ? "toolbar" : "toolbar-notloggedin" %>"></div>
    15:       
    16:     <div id="header">
    17:       <div class="left">
 app/views/layouts/application.html.erb:14:in `_app_views_layouts_application_html_erb__774776397_13334'
 app/controllers/hubs_controller.rb:112:in `import'

Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/_trace.erb (30.0ms)
Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (19.0ms)
Rendered /Users/arikan/.rvm/gems/jruby-1.6.4/gems/actionpack-3.1.3/lib/action_dispatch/middleware/templates/rescues/template_error.erb within rescues/layout (65.0ms)

Re: running background jobs with neo4j.rb rails? Matthew Ford 1/30/12 2:51 PM
Hi, 

I was wondering if there are any examples of getting a Neo4J HA cluster setup with Rails 3. Currently running into an issue where I'd like to run a background task on collecting data from an API.

Cheers,
Matt
Re: running background jobs with neo4j.rb rails? ronge 1/30/12 11:54 PM
Hi

Yes, there is a very simple example here -
https://github.com/andreasronge/neo4j/tree/master/example/ha-cluster
Regarding Rails 3 and Neo4j you probably should check the HAProxy docs
- http://docs.neo4j.org/chunked/snapshot/ha-haproxy.html
Not sure if it's a good idea to use neo4j HA cluster just because you
need to run background tasks.
Maybe it's better to provide your own rest API or using
http://torquebox.org/ (see previous mails).

Cheers
Andreas

> --
> You received this message because you are subscribed to the Google Groups
> "neo4jrb" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/neo4jrb/-/OvzgA1pJ5JQJ.

>
> To post to this group, send email to neo...@googlegroups.com.
> To unsubscribe from this group, send email to
> neo4jrb+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/neo4jrb?hl=en.

Re: running background jobs with neo4j.rb rails? Vivek Prahlad 1/31/12 12:35 AM
On my current project, we used Resque for a background jobs. We built a simple REST Api for the neo4j.rb application, and we invoke the API as part of the Resque job. That way, we avoid having to worry about things like multiple connections to the same database. This approach can also be extended in the future to support database sharding, for example.

Cheers,
Vivek