large databases
Simon Strobl
I am trying to work with large databases. Most Prologs seem to be
unable to load databases whose size surpasses a certain limit. An
example: I had both Swi and Yap load a 3.1G database. Swi ran several
days until I stopped the experiment. Yap produced a segmentation fault
after a few minutes.
Are there any tricks for loading such databases?
Note that splitting a database up is not always a solution. Sometimes
one wants to be able to easily compare all facts with all other facts.
Simon
Jan Wielemaker
> Hello,
>
> I am trying to work with large databases. Most Prologs seem to be
> unable to load databases whose size surpasses a certain limit. An
> example: I had both Swi and Yap load a 3.1G database. Swi ran several
> days until I stopped the experiment. Yap produced a segmentation fault
> after a few minutes.
>
> Are there any tricks for loading such databases?
The Prolog system should of course be able to handle it, but most
importantly, make sure most of your database predicates are properly
indexed. Non-indexed calls to predicates with millions of clauses is no
fun. This may imply building additional indexes, but as long as you have
the memory to hold the extra tables that will pay off.
YAP does a lot more on dynamic indexing, so there is less work for you
to consider. SWI only does first argument indexing, but surely on 64-bit
hardware you can use a lot of memory.
Cheers --- Jan
Simon Strobl
> importantly, make sure most of your database predicates are properly
> indexed.
O.k. thanks for the tip. At the time being, however, my problem is not
the time Prolog uses for making use of facts. My problem is that
Prolog does not manage to load the facts. As I said, Yap crashes
(although enough RAM was available). Swi did not crash, but it did not
succeed to load the file in several days. I have now started a new Swi-
process. I will wait a week or two in order to see if Swi can load the
file.
Jan Wielemaker
SWI should load this *much* faster, so something is wrong. What OS are
you using and is this a 32-bit or 64-bit version? It is possible SWI
crashed too; sometimes it recovers poorly on fatal errors. You can
avoid recovery using pl --nosignals. Just to show:
?- tell('x.pl'), forall(between(1, 10000000, X), format('~q.~n', [a(X)])), told.
ct (~) 4_> ls -lh x.pl
-rw-r--r-- 1 jan users 114M 2008-05-21 19:23 x.pl
?- time([x]).
% x compiled 56.44 sec, 1,040,000,720 bytes
% 160,000,369 inferences, 56.44 CPU in 57.37 seconds (98% CPU, 2834875 Lips)
This is using 64-bit Linux version on AMD Athlon 5400+. That gives some
idea about the loading time to expect. Of course, these are very simple
facts.
--- Jan
Simon Strobl
> SWI should load this *much* faster, so something is wrong.
Oddly enough, I cannot reproduce the problem with Swi. It is working
now. Maybe my screen session crashed or there was a server problem.
Thanks for your tips, anyway.
Here is part of my Swi session:
Welcome to SWI-Prolog (Multi-threaded, 64 bits, Version 5.6.55)
Copyright (c) 1990-2008 University of Amsterdam.
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions.
Please visit http://www.swi-prolog.org for details.
For help, use ?- help(Topic). or ?- apropos(Word).
?- consult(data).
% data compiled 6743.67 sec, 11,872,550,352 bytes
true.
?-
Jan Wielemaker
Next time, try
?- qcompile(data).
That will load data and produce the file date.qlf. The next
?- [data].
will load data.qlf, which loads roughly 10 times faster (depending on
the structure of the data, whether loading is CPU bound or I/O bound,
etc.).
Enjoy --- Jan
Cameron Hughes
SWI-Prolog continues to be one of the most well rounded Prologs out
there! Jan
your always willingness to help, give advice, and support is very much
appreciated.
I hope this community realizes what a valuable resource you are!!!!
In case you haven't heard it recently, thanx for the committment and
all your contribution
to the Prolog community.
Cameron