Reading millions of lines from a RDF file to Postgres

177 views
Skip to the first unread message

Rohit Mishra

unread,
28 Mar 2013, 03:11:1728/03/2013
to bangal...@googlegroups.com
Hi, 

I am working on importing DBpedia RDF dumps in my postgres database. The title dump file that I am working on has > 9 million rows. I know that the right approach to use this kind of data is to use a graph db, but I do not want to complicate my stack with something like Neo4j now. 

I am reading the RDF dump using the RDF gem and doing a couple of ActiveRecord transactions for insertion and association. But this process is slow, and will take many days to complete. How can I speed up this process?

Thanks,

Swanand Pagnis

unread,
28 Mar 2013, 03:24:3328/03/2013
to bangal...@googlegroups.com
Hadoop. :-) It is built for this purpose.


--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Anand Chitipothu

unread,
28 Mar 2013, 04:44:1528/03/2013
to bangal...@googlegroups.com


On Thursday, March 28, 2013, Rohit Mishra wrote:
Hi, 

I am working on importing DBpedia RDF dumps in my postgres database. The title dump file that I am working on has > 9 million rows. I know that the right approach to use this kind of data is to use a graph db, but I do not want to complicate my stack with something like Neo4j now. 

I am reading the RDF dump using the RDF gem and doing a couple of ActiveRecord transactions for insertion and association. But this process is slow, and will take many days to complete. How can I speed up this process?

If your tables have any indexes, remove them. That'll speed up your inserts considerably fast. You can add them back  later  once you are done with loading the data.

Anand


--
Anand
http://anandology.com/

Rohit Mishra

unread,
4 Apr 2013, 06:40:2604/04/2013
to bangal...@googlegroups.com
Thanks. I have removed the indexes, but it is still pretty slow. 

I read more on this, and one of the suggested methods to speed things up is to disable autocommits. I haven't been able to find any documentation on how to do this till now. If anyone of you have any pointers on how to do this, please share.  

Anand


--
Anand
http://anandology.com/

Thanks,
Rohit

--
Rohit Mishra

Hemant Kumar

unread,
4 Apr 2013, 07:01:5704/04/2013
to bangal...@googlegroups.com
Hi Rohit,

On Thu, Apr 4, 2013 at 4:10 PM, Rohit Mishra <ro...@rohitmishra.me> wrote:

Thanks. I have removed the indexes, but it is still pretty slow. 

I read more on this, and one of the suggested methods to speed things up is to disable autocommits. I haven't been able to find any documentation on how to do this till now. If anyone of you have any pointers on how to do this, please share.  


You can disable autocommits via:


 

Rohit Mishra

unread,
4 Apr 2013, 07:17:0004/04/2013
to bangal...@googlegroups.com
This page says that autocommits are disabled for the entire session. Since, I have to do this import on production, is this the right thing to do? Preferably, if I can do this only for the particular rake task, it will be great. 

My apologies if I am missing something obvious. 

Rohit

 

--
You received this message because you are subscribed to a topic in the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bangalorerug/o392x1sFtEE/unsubscribe?hl=en-GB.
To unsubscribe from this group and all of its topics, send an email to bangalorerug...@googlegroups.com.

Dhruva Sagar

unread,
4 Apr 2013, 07:47:5304/04/2013
to bangal...@googlegroups.com


--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Thanks & Regards,
Dhruva Sagar

----------------------------
Senior Ruby/Rails Consultant

Rohit Mishra

unread,
4 Apr 2013, 11:15:4104/04/2013
to bangal...@googlegroups.com
I need to process the text lines with the RDF gem before inserting them in to Postgres, so using the copy or prepare command won't work in this case. 

Leena S N

unread,
4 Apr 2013, 13:19:5104/04/2013
to bangal...@googlegroups.com
Hi Rohit,

If am not wrong, grouping the entire code in rake task with Model.transaction block should disable the autocommit. This way the transaction is  started by code, so DB should wait on the code to commit or rollback.

You can explicitly disable the autocommit at the connection level, by using ActiveRecord::Base.connection.execute method with 'set the autocommit=0' which should disable the autocommit only for the current session. 

Hope this helps.


--
Leena S N
http://www.multunus.com/

Rohit Mishra

unread,
5 Apr 2013, 05:23:3005/04/2013
to bangal...@googlegroups.com
Hi Leena,

Thanks for your help. I will try using transactions. I don't want to disable autocommit for the whole session because this will effectother production code. 

Rohit

Leena S N

unread,
5 Apr 2013, 14:35:4105/04/2013
to bangal...@googlegroups.com
Rohit,

I think there is some confusion with the second option I was suggesting and I would like to clarify. What I meant was to turn off the autocommit from the rake task. In that way it would affect only the the session initiated by rake task, not the sessions initiated by the production app. I believe you can do that with ActiveRecord::Base.connection.execute( 'SET AUTOCOMMIT=0' ) or with Model.connection.execute( 'SET AUTOCOMMIT=0' ) 

Rohit Mishra

unread,
6 Apr 2013, 09:45:0706/04/2013
to bangal...@googlegroups.com, lee...@gmail.com
Leena, 

Thanks. This should be a great option. Let me try this out and report back the results. 

Rohit
Hi Rohit,

To unsubscribe from this group and all of its topics, send an email to bangalorerug+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Thanks & Regards,

--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
Leena S N
http://www.multunus.com/

--
You received this message because you are subscribed to a topic in the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bangalorerug/o392x1sFtEE/unsubscribe?hl=en-GB.
To unsubscribe from this group and all of its topics, send an email to bangalorerug...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply all
Reply to author
Forward
0 new messages