Problem inserting 150,000 nodes

[Dave Troy]

Hi folks,

I'm trying to import a dataset with roughly 150,000 people nodes into Neo4j.rb; running OS X (10.8.2) and jruby 1.6.7.2 with neo4j 2.2.1, and neo4j-community 1.8.

I have a primary key defined on the people nodes. I'm trying to load these people in and create these nodes using a rake task.

1) I have some duplicates in the source; I can't seem to use get_or_create because I get a ProxyNode object on an existing node, and can't seem to use that for anything. What's up with that?

2) If I use a strategy like this -- Person.find_by_myid(p['id']) || Person.create(:raw_profile => p.to_json) -- then it ends up crashing after ~10k people saying "too many open files" on the Lucene index.

3) I read up on the Lucene index problem and it sounds like something happening in the neo4j library (not closing objects). How can I deal with this?

4) What happened to the bulk_insert functionality and should I be thinking about something like that for this task?

5) Lucene is using a ton of open files -- growing at a rate of about 1,000/minute (lsof | grep java | grep lucene -c == 10127). Why is Lucene holding so many open files?

Help?

Thanks and happy holidays.

Dave

[Dave Troy]

Here are some details on the code I'm having trouble with.

The insertion loop looks like this at the moment. I've tried many variations. This is in a method called "add_connections" on a Person instance. "list" is a parsed data structure.

      list.each do |p|

        next if p['id']=='private' || p['firstName'].blank? || p['lastName'].blank?

        Person.transaction do |tx|

          friend = Person.find_by_linkedin_id(p['id'])

          friend ||= Person.create(:raw_profile => p.to_json)

          self.find_or_create_friendship(friend)

          self.save

        end

        count += 1

      end

The Person model is defined essentially as follows:

class Person < Neo4j::Rails::Model

  

  has_n(:friends)

  

  property :firstname

  property :lastname

  property :headline

  property :industry

  property :company

  property :raw_profile

  property :oauth_token

  property :oauth_secret

  property :updated_at

  property :created_at

  property :imported_connections_at, :type => DateTime

  property :imported_edges_at, :type => DateTime

  

  property :linkedin_id, :index => :exact, :unique => true

  validates :linkedin_id, :exclusion => { :in => %w(private) }

  validates :firstname, :presence => true, :allow_blank => false

  validates :lastname, :presence => true, :allow_blank => false

  

  before_validation :update_properties

end

Any insights appreciated!

Dave

[Andreas Ronge]

Hi

Regarding  lucene and open files, you must call the close method, see http://rdoc.info/github/andreasronge/neo4j-core/Neo4j/Core/Index/Indexer#find-instance_method

Maybe this is not clear in the documentation (feel free to update the wiki pages).

When used from Rails, this is done automatically for you after each request, but since you use it from Rake you have to call close manually.

I will come back to the other questions later.

Cheers

Andreas.

Dave

--
You received this message because you are subscribed to the Google Groups "neo4jrb" group.
To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/RO0vhxZInzUJ.
To post to this group, send email to neo...@googlegroups.com.
To unsubscribe from this group, send email to neo4jrb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/neo4jrb?hl=en.

[Andreas Ronge]

Hi again

If you use the Neo4j::Rails::Model you can close all lucene connection with

  Neo4j::Rails::Model.close_lucene_connections

Regarding performance, you should try to only create a new transaction for each 1000-3000 write. This means that you either have to manually trigger updates to the lucene index (since it is updated after the transaction finished) or keep the linkedin_ids which has not been written to the lucene database in memory, which I think is the best solution.

Something like this, maybe

# check first in memory for the linkedin id

friend = not_commited_linkedin_ids.find ... # use a hashmap

new_friend = !friend && Person.create(:raw_profile => p.to_json)

if new_person

  # if new person was created store the linkedin_id

  not_commited_linkedin_ids << new_friend.linkedin_id 

  self.find_or_create_friendship(new_friend)

  self.save

end

# maybe we should handle a friend has been created in the same transaction, !!friend == true

# only commit after 1000 updates

if count % 1000 == 0

  tx.success

  tx.finish

  not_commited_linkedin_ids = [] # can now be found using lucene

  tx = Neo4j::Rails::Transaction.new

end

[Dave Troy]

Thanks Andreas. I am using Neo4j::Rails::Model.close_lucene_connections and am getting improved filehandle performance. Thanks!

I'll try the strategy of committing around 1000-3000 items at a time also.

Right now I have a new problem: java is becoming horribly slow at about 820MB of heap usage. I know it's possible (and desirable) to increase the Java heap size, but,

a) I don't know how to do that for a rake task (I'm using rvm),

b) *should* a simple import job be consuming 800+MB of memory, when I am not needing to retain objects as the loop is processed?

Any insights appreciated.

Best,

Dave

[Andreas Ronge]

Interesting. No, it sounds like a memory leak.

I would suggest that you don't use the Neo4j::Rails API (Neo4j::Rails::Model and Neo4j::Rails::Relationship) but instead use the raw java api/neo4j-core API (Neo4j::Node and Neo4j::Relationship). The reason is that you will get better performance because each Java object is not wrapped in a Ruby object. Another reason of doing this is to narrow down where a possible memory leak is located. 

I would also suggest using the VisualVM tool monitoring the memory consumption (use the -J-Djruby.reify.classes=true, see https://blog.engineyard.com/2010/monitoring-memory-with-jruby-part-1-jhat-and-visualvm/)

To access the lucene neo4j-core API you can use the _indexer method which returns a Neo4j::Core::Index::Indexer object, see

http://rdoc.info/github/andreasronge/neo4j-core/Neo4j/Core/Index/Indexer

To query using the Neo4j::Core::Index::Indexer and the Lucene query Syntax, example:

    result = Person._indexer.find("name: andreas", :wrapped => false) # wrapped false means don't load the Ruby wrapper around the java node object

    result.first  # an Enumerable, returns nil if nothing found

    result.close # don't forget to close the connection

To add an index on an existing node:  Person._indexer.add_index(a, "name", "sune") 

Notice, you don't have to wait till the transaction finished in order to query the index in this case. Not sure you need this method.

To create new raw node: 

    Neo4j::Node.new(:_classname => "Person", :name => 'andreas')  

Notice, if there is a declared index on :name then the lucene index will be updated after the transaction is completed.

To load the Ruby wrapper around the node (Java::OrgNeo4jKernelImplCore::NodeProxy) , use the wrapper method, example:

    Person._indexer.find("name: kalle", :wrapped => false).first.wrapper.class # => Person

I could also have a quick look at it. Can I have a look at the complete example together with a sample of the data you are trying to load ?

Cheers

Andreas

To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/4qUU6P5e7XYJ.

[Michael Hunger]

Dave, usually inserting 150k nodes is a no-brainer, there seems to be something really weird going on.

Can you perhaps share your project somehow so that we could look into it?

Thanks a lot

Michael

To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/4qUU6P5e7XYJ.

[Dave Troy]

I am relying on several of the Neo4j::Rails::Model features so I'm hoping to solve this in that context before resorting to using neo4j-core.

I went ahead and used jmap to get a sense of what was using up the memory. I'm a bit outside of my knowledge on this, so consider this a naive first pass.

At the moment the memory leak is occurring, the full list looks like this:

 num     #instances         #bytes  class name

----------------------------------------------

   1:        628997      326402088  [B

   2:        574226       22969040  org.jruby.util.ByteList

   3:        510731       20429240  org.jruby.RubyHash$RubyHashEntry

   4:        560446       17934272  org.jruby.RubyString

   5:         63654       13124856  [C

   6:         71988       10956824  <constMethodKlass>

   7:         71988        9801360  <methodKlass>

   8:          9748        8879536  <instanceKlassKlass>

   9:          9748        8331112  <constantPoolKlass>

  10:        115582        7399240  [Lorg.jruby.RubyHash$RubyHashEntry;

  11:        242891        6841080  [Ljava.lang.Object;

  12:        115578        6472368  org.jruby.RubyHash

  13:        102662        5696312  <symbolKlass>

  14:          9213        4921416  <constantPoolCacheKlass>

  15:        195806        4699344  java.lang.Long

  16:         87162        2789184  org.jruby.RubyFixnum

  17:          5425        2641392  <methodDataKlass>

Grepping out the jRuby objects (with the reify option passed):

 146:          2407          57768  rubyobj.Neo4j.Core.Index.LuceneQuery

 225:           560          13440  rubyobj.Person

 229:           545          13080  rubyobj.Gem.Version

 233:           511          12264  rubyobj.Gem.Requirement

 256:           383           9192  rubyobj.Gem.Dependency

 390:           122           2928  rubyobj.URI.HTTPS

 392:           121           2904  rubyobj.Net.HTTP

 394:           121           2904  rubyobj.Net.HTTPOK

 395:           121           2904  rubyobj.OAuth.Consumer

 397:           121           2904  rubyobj.Neo4j.Rails.Relationships.Storage

 398:           121           2904  rubyobj.DateTime

 399:           121           2904  rubyobj.OAuth.AccessToken

 400:           120           2880  rubyobj.ActiveModel.Errors

 404:           114           2736  rubyobj.Rails.Initializable.Initializer

 467:            78           1872  rubyobj.Monitor

 476:            73           1752  rubyobj.Rake.Task

 493:            63           1512  rubyobj.Gem.Specification

 505:            56           1344  rubyobj.Bundler.LazySpecification

 578:            34            816  rubyobj.Journey.Nodes.Cat

 702:            22            528  rubyobj.Mime.Type

The Ruby objects look reasonable. The biggest chunk of memory is obviously being used by "[B" but I don't know what that means.

I've posted my model file here: https://gist.github.com/4384778

If you have any guesses about what is causing the memory leak I think that's the primary issue at the moment.

BTW, if I don't include the Neo4j::Rails::Model.close_lucene_connections call after each import, those objects do start to run away in jmap, so that was definitely a problem and one which was addressed with that call. I'd expect the Rails model to close those connections for me. Also, for that pattern of "find in index, or create" I'd expect "get_or_create" to work; some kind of more intuitive Unique Factory API at this level would be good.

Dave

[Michael Hunger]

This still looks as if the lcuene query results from your Person.find() are not closed:

 146:          2407          57768  rubyobj.Neo4j.Core.Index.LuceneQuery

[B are imho byte arrays, I would really abstain from your raw_profile thingy and json strings and work with the individual properties.

Michael

To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/kdVLrCWel3UJ.

[Andreas Ronge]

Aha, I think I know what's the problem is: the IdentityMap see http://rdoc.info/github/andreasronge/neo4j-wrapper/Neo4j/IdentityMap.  (the number of org.jruby.RubyHash$RubyHashEntry looks suspicious)

Try disable the cache:

  Neo4j::Config[:identity_map] = false

Just like with closing the lucene connection, cleaning the cache occurs after each request when used from Rails (rack middleware). Since you use it from Rake the cache will not work.

I should really only enable identity map when used from rails, will add a github issue.

Cheers

To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/kdVLrCWel3UJ.

[Dave Troy]

Michael,

While there were a number of open LuceneQuery objects in that list, they get freed quickly and weren't building up. Before, there were tends of thousands, constantly accumulating.

The storage of the raw JSON strings is a bit of a temporary hack while I develop the graph further. Eventually that will go away and be replaced with discrete properties, meantime I wouldn't expect Neo4j to have a problem persisting arbitrary ~3k byte strings.

But I did find that I was able to keep memory growth under control by clearing memoized properties @profile and @access_token -- these were being held for some reason. VisualVM helped me try a few things and see what would curb growth.

Best,

Dave

...
Show original

[Dave Troy]

Thanks Andreas. As I mentioned in my last message to Michael I was able to make the import work by clearing the memoized @profile and @access_token ivars; VisualVM helped me figure that much out.

I'll try the Neo4j::Config[:identity_map] = false setting too. Thanks!

And as for a UniqueFactory API, why am I unable to make get_or_create to work for this? Seems like it's the perfect thing, but I guess I don't understand how it works.

Best,

Dave

<blockquote class="gmail_quote" style="margin:...
Show original

[Andreas Ronge]

I guess the problem with "get_or_create" method is that it does not return the wrapped object, but the java node.

I've added an github issue for this: https://github.com/andreasronge/neo4j-wrapper/issues/8

To get your wrapper (Person) object use the wrapper method (as a workaround).

Example

  Person.get_or_create(:linkedin_id => 123123, some properties for a new person).wrapper

Regarding the IdentityMap, maybe that is not a problem because it's also cleaned after each transaction, so that should not be a memory leak for you. It just consumes some memory without giving you much performance benefits since you probably don't often load nodes more than once (?).

--

You received this message because you are subscribed to the Google Groups "neo4jrb" group.

To view this discussion on the web visit https://groups.google.com/d/msg/neo4jrb/-/BmDHRgK4nGoJ.

[Andreas Ronge]

I think I know why you had a problem with get_or_create. It must not be called in a transaction.

But since you are using it from Rake and you don't have concurrent access to the database, it should be enough to use the find_or_create_by_xxx rails methods.

Happy new year

[Michael Hunger]

Andreas,

why mustn't it be called within a tx ? Does it the use the underlying Neo4j mechanism for get_or_create ? Which is transactionally safe and threadsafe and HA safe.

Michael

[Andreas Ronge]

Michael, maybe because the way I've implemented it using putIfAbsent

I noticed there is a better way of impl. it - http://docs.neo4j.org/chunked/stable/transactions-unique-nodes.html#transactions-get-or-create

I call the putIfAbsent method here: https://github.com/andreasronge/neo4j-core/blob/master/lib/neo4j-core/index/unique_factory.rb#L40

[Michael Hunger]

Do you also create index-locks?

Sent from mobile device

[Andreas Ronge]

No

sent from my phone

[Mattias Persson]

index#putIfAbsent is an atomic operation so that's fine.