Adding large file to a db

[nemanja.vu...@sbgenomics.com]

I've downloaded Stardog Community Edition to test it out. I have a dataset in RDF/XML format of ~2GB and I can't add it to an already created database. First I tried using the admin but as the file is too large I switched to command line import. So I ran ./stardog data add -f RDF/XML knowledge_base kb.xml and after ~30 seconds or so Stardog only gives out the following: "There was a fatal failure during preparation of ea60e75b-2d71-4cd9-b87f-e200d36aeb81". Every time I try to run it it gives out the same error with a different hash. I tried to import an other file which was considerably smaller using the same approach and it worked. How can I import a large dataset I have?

Thanks

[Evren Sirin]

Community edition has a limit of 25 million triples in a database.
Your file probably has more triples in it. If the error message client
display is not informative you can always look at stardog.log where
there will be a more detailed error message. If you would like to test
Stardog with more than 25M triples then you should get the 30-day
evaluation license.

Best,
Evren

> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com
> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Marcel Karnstedt-Hulpus]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

We observed similar issues with the Enterprise edition. As it turned
out, they were apparently due to the fact that we ran a single-node
instance in cluster mode.

We were able to load our data in two different ways:

a) turn off cluster mode
b) add data at creation time of the DB (also worked in cluster mode)

Still, Evren's explanation sounds like the more reasonable cause.


Cheers,
Marcel

- --
Dr. Marcel Karnstedt-Hulpus
www.marcel.karnstedt.com

Senior Data Architect
IT & Production Corporate Markets / Databases
Springer Nature
www.springernature.com

tel +49-6221-4878868
mob +49-151-23125052
Tiergartenstrasse 17, 69121 Heidelberg, Germany

- ---
Springer Nature is a major new force in scientific, scholarly,
professional and educational publishing, created in May 2015 through
the combination of Nature Publishing Group, Palgrave Macmillan,
Macmillan Education and Springer Science+Business Media.
- ---
Branch of Springer-Verlag GmbH, Heidelberger Platz 3,
14197 Berlin, Germany
Registered Office: Berlin / Amtsgericht Berlin-Charlottenburg,
HRB 91881 B
Directors: Derk Haank, Martin Mos, Petrus W. J. Hendriks
- ---
- ---

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJXPY6SAAoJELjmZAjIQprmIggIAJF5TxAoiqSb8QIWtsi9d0Xi
fmyltqTBcdCv7r4NkIoDX2wyLXwqwe1yDMeW/y0XoBEYWqOqCvKD9jRNdqldjjjM
+F4DP14lxJm3Y/gYstuHn8G5hoE4qmoDaA5uKXopkHXpk+QWult3zlI9vhlHzjfn
RFNqkFN3QOab4VghG99FOyWuYSxP76sIKJPmQwOvr6iRpeAYRQiPCThgOSMhBuUT
taSx2eOXrNQYnLuUGnAxMgdPnAUVqKhk25wbMl70d3bhGBjpWD8B6oMO7pKYOb7h
yr2n8bnTYWcX79OXotB0nVJ+sdiA2PnVX5QGXQo6+LtMJdvwRAvslTXtOYh1PuA=
=07bc
-----END PGP SIGNATURE-----

[nemanja.vu...@sbgenomics.com]

Well, in my db I have ~17 million triples so I presume that's not the issue?

Some other course of action?

Thanks,

Nemanja

[Michael Grove]

Can you provide your stardog.log file?

Cheers,

Mike

[nemanja.vu...@sbgenomics.com]

This worked. The part of adding the data at the time of creation. Thanks

[Michael Grove]

On Thu, May 19, 2016 at 6:55 AM, <nemanja.vu...@sbgenomics.com> wrote:

This worked. The part of adding the data at the time of creation. Thanks

You might have not been giving the server enough memory for a transaction of that size. There is memory overhead for performing a transaction that bulk loading at creation time does not have.

Cheers,

Mike