Loading ~86M quads taking hours on Bolt and LevelDB backend

[Sol Eun]

Hello all,

I am trying to load a decent size graph (~86M quads) to Bolt but I am experiencing very slow insert performance.
The machine has quad core and 16GB ram.
I sorted the triples if there is any additional performance gain from it but nothing noticeable.
It has been 5 hours and it has written 18M quads.

Is this the kind of insert performance I should expect? or am I doing it wrong?

Any guidance will be helpful.


Sol

[Denys Smirnov]

Hi Sol,

LevelDB backend in master is noticeably slower than Bolt, but such insert performance (~1000 quads/sec) is still unexpected. For example, lowest performance for postgres is ~5000 quads/sec. I will investigate it for Bolt.

среда, 23 марта 2016 г., 0:13:04 UTC+2 пользователь Sol Eun написал:

[Sol Eun]

Thanks for looking into it Denys.

So here is some additional information.
Both gets gradually slower as more quads are being written.

Not sure breaking the large file (~10GB) into smaller files will do any better.

Bolt

I0325 07:45:32.643736 12620 cayley.go:175] Setting GOMAXPROCS to 4
I0325 07:45:32.644739 12620 db.go:52] Opening quad store "bolt" at db/store_bolt
I0325 07:45:32.646745 12620 db.go:62] Opening replication method "single"
I0325 07:46:40.470120 12620 db.go:91] Wrote 100000 quads.
I0325 07:46:46.441794 12620 db.go:91] Wrote 200000 quads.
I0325 07:46:53.647915 12620 db.go:91] Wrote 300000 quads.
I0325 07:47:02.400353 12620 db.go:91] Wrote 400000 quads.
I0325 07:47:16.743401 12620 db.go:91] Wrote 500000 quads.
I0325 07:47:37.274729 12620 db.go:91] Wrote 600000 quads.
I0325 07:48:03.876154 12620 db.go:91] Wrote 700000 quads.
I0325 07:48:31.274279 12620 db.go:91] Wrote 800000 quads.
I0325 07:49:01.687662 12620 db.go:91] Wrote 900000 quads.
I0325 07:49:35.355296 12620 db.go:91] Wrote 1000000 quads.

Postgres

I0325 08:35:49.635586 06600 cayley.go:175] Setting GOMAXPROCS to 4
I0325 08:35:49.636586 06600 db.go:52] Opening quad store "sql" at postgres://postgres:password@

localhost/cayley?sslmode=disable
I0325 08:35:49.653587 06600 db.go:62] Opening replication method "single"
I0325 08:35:56.922314 06600 db.go:91] Wrote 100000 quads.
I0325 08:36:06.300252 06600 db.go:91] Wrote 200000 quads.
I0325 08:36:18.685490 06600 db.go:91] Wrote 300000 quads.
I0325 08:36:37.143336 06600 db.go:91] Wrote 400000 quads.
I0325 08:36:56.546276 06600 db.go:91] Wrote 500000 quads.
I0325 08:37:14.173038 06600 db.go:91] Wrote 600000 quads.
I0325 08:37:48.268448 06600 db.go:91] Wrote 700000 quads.
I0325 08:38:21.667787 06600 db.go:91] Wrote 800000 quads.
I0325 08:38:42.275848 06600 db.go:91] Wrote 900000 quads.
I0325 08:39:11.215742 06600 db.go:91] Wrote 1000000 quads.

[matthew...@gmail.com]

How are you getting that output? I've added --alsologtostderr to my load command but am not getting those statistics?

[rajmalhot...@gmail.com]

On Thursday, June 23, 2016 at 11:07:36 PM UTC+5:30, matthew...@gmail.com wrote:
> How are you getting that output? I've added --alsologtostderr to my load command but am not getting those statistics?

Hi ,

you can add -v=2 in the command to get those logs.

./cayley load -config=confname.conf -quads=filename -v=2 --alsologtostderr