FEATURE REQUEST qcompile_predicates/N

[Douglas Miles]

Hi,

I have a file with 5,570,822 prolog clauses.

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
Welcome to SWI-Prolog (Multi-threaded, 64 bits, Version 7.3.4)
Copyright (c) 1990-2015 University of Amsterdam, VU Amsterdam
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions.
Please visit http://www.swi-prolog.org for details.
For help, use ?- help(Topic). or ?- apropos(Word).
?- set_prolog_flag(verbose_load,true).
true.
?- ['el_assertions.pl'].
% el_assertions.pl compiled into el_assertions 281.57 sec, 5,570,822 clauses
true.

It takes 5 minutes  to load

So I do the obvious smart things and convert it to a QLF

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt." 

% 1,817,844,040 inferences, 277.090 CPU in 277.493 seconds (100% CPU, 6560478 Lips)

 whereas now it take only 6-7 seconds to load!

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
Welcome TO SWI-Prolog (Multi-threaded, 64 bits, Version 7.3.4)
Copyright (c) 1990-2015 University of Amsterdam, VU Amsterdam
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software,
AND you are welcome TO redistribute it under certain conditions.
Please visit http://www.swi-prolog.org FOR details.
FOR help, use ?- help(Topic). OR ?- apropos(Word).
?- time(['el_assertions.qlf']).
% 3,648 inferences, 6.205 CPU in 6.213 seconds (100% CPU, 588 Lips)
true.

YAY!

Here is the problem.   I'd like to change, add or remove one of the clauses.


I am stuck writing the file back out and compiling again... writing the clauses back out takes me 45 seconds

?- statistics(cputime,S),tell(fooff),ignore(((el_assertions:el_holds_pred_impl(F)),between(2,16,A),current_predicate(el_assertions:F/A),functor(P,F,A),forall(el_assertions:P,format('~q.~n',[P])),fail)),told,   statistics(cputime,E),Total is E - S.
Total = 45.97321134599997.

Now I will need to qcompile what I just wrote out (this will take  me another 277.090 secs to get it to QLF for the next time)

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt."

7 minutes to update a single clause seems a bit too much !

But I decide maybe there is a better Idea

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
Welcome to SWI-Prolog (Multi-threaded, 64 bits, Version 7.3.4)
Copyright (c) 1990-2015 University of Amsterdam, VU Amsterdam
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions.
Please visit http://www.swi-prolog.org for details.


For help, use ?- help(Topic). or ?- apropos(Word).


?- time(['el_assertions.qlf']).
% 3,648 inferences, 6.205 CPU in 6.213 seconds (100% CPU, 588 Lips)
true.

?- retract(el_holds(expansion,interArgIsa,[implies,[and,[admittedSentence,'?SENT'],[isa,'?INDEP-INS',':ARG3'],[operatorFormulas,':ARG1','?SENT'],[argN,'?DEP-INS',':ARG4','?SENT'],[argN,'?INDEP-INS',':ARG2','?SENT']],[isa,'?DEP-INS',':ARG5']],'BaseKB',_))
true.

And QSAVE my program isn't this a QLF+Runtime?

?- time(qsave_program(compatQLF0,[autoload(false),goal((statistics(cputime,Time),writeln(user_output,Time)))])).
% 22,514,037 inferences, 36.441 CPU in 36.511 seconds (100% CPU, 617815 Lips)
true.

?-
% halt

So I wonder if it takes around 6 seconds to load?

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# ./compatQLF0
7.112799221
?-

Indeed it does!   If infact.

root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# ls -l
total 2263188
-rwxr-xr-x 1 root root 563042175 Jul 25 18:07 compatQLF0
-rwxr-xr-x 1 root root 69078779 Jun 19 02:55 el_assertions.pl.rar
-rw-r--r-- 1 root root 560782717 Jul 25 18:53 el_assertions.qlf
-rwxr-xr-x 1 root root 1124578930 Jun 18 20:41 el_assertions.pl

compatQLF0 and el_assertions.qlf are about the same size file.

.......

So now here comes the feature request!

How can I qcompile  what was in memory and write to a QLF?

See, we can do this in 36.511 seconds !

By making something like qsave does and not adding the executable header.  (though still saving the operators and term_expansion states probably saved)

We need to be about to say 

?- qcompile_predicates('el_assertions.qlf',[foo/1,bar/2],[QSave/QCompile Options]).

The code is there we just need to expose it to prolog?

Also would this be good for the persistency module?  

(Oh sinking feeling after writing this email ...maybe the persistency module already does this!)

I am going to send the email anyways.

- Douglas

[Jan Wielemaker]

Hi Douglas,

I think you want something similar to the RDF database. Here, a graph is
represented by two files (which may or may not exist): the _snapshot_ which
uses the internal binary RDF format and the _journal_, which caries the
_changes_ expressed as Prolog terms. Loading the graph implies loading the
(possibly non-existing) snapshot and replay the (possibly non-existing)
changes. A change extends the change journal and there is a call to
write a new snapshot, destroying the journal.

Library persistency does something like that, except that there is only
a journal that may be compacted by writing the current state to a fresh
journal so you effectively get rid of events that cancel each other.

The snapshot could indeed using the qlf format. Using qlf for the journal
is a bit harder because the qlf format is stateful.

A disadvantage is that the qlf version depends on 32/64 bit and is version
dependant. It might be wise to consider an alternative, in particular
if we consider only facts. Possibly something based on
PL_record_external(),
which creates a binary representation of a Prolog term that is supposed to
be stable (it changed twice during SWI-Prolog's lifetime).

We'd need some timings. Loading .pl is expensive due to the parsing and
expansion process. Probably library persistency already wins as it skips
a lot of the compilation steps, but still needs the parser and VM compiler.
Basing library persistency on PL_record_external() would get rid of the
parser overhead, but still needs to intern each atom one-by-one and needs
to do the VM code generation.

We could design a new term sequence binary format based on
PL_record_external()
that avoids interning atoms one-by-one. Such a format could also be used to
link up Prolog processes using a pipe or socket to exchange Prolog terms
much
faster than read/write. If we would make the sequence restartable, i.e.,
flush the state, it could be used with library persistency without
significant changes to this library.

Cheers --- Jan

On 07/26/15 02:12, Douglas Miles wrote:
>
> Hi,
>
>
> I have a file with 5,570,822 prolog clauses.
>
> |
>
>
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> Welcometo SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam
> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,

> andyou are welcome to redistribute it under certain conditions.

> Pleasevisit http://www.swi-prolog.org for details.
> Forhelp,use?-help(Topic).or?-apropos(Word).
> ?-set_prolog_flag(verbose_load,true).
> true.
> ?-['el_assertions.pl'].
> %el_assertions.pl compiled intoel_assertions 281.57sec,5,570,822clauses
> true.
>
>
>
> |
>
>
>
> It takes 5 minutes to load
>
>
> So I do the obvious smart things and convert it to a QLF
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g
> "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt."
>
> % 1,817,844,040 inferences, 277.090 CPU in 277.493 seconds (100%
> CPU, 6560478 Lips)
>
>
>
> whereas now it take only 6-7 seconds to load!
>
> |
>
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> WelcomeTO SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam

> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,

> AND you are welcome TO redistribute it under certain conditions.

> Pleasevisit http://www.swi-prolog.org FOR details.

> FOR help,use?-help(Topic).OR ?-apropos(Word).
> ?-time(['el_assertions.qlf']).
> %3,648inferences,6.205CPU in6.213seconds (100%CPU,588Lips)
> true.
> |
>
>
>
>
> YAY!
>
>
> Here is the problem. I'd like to change, add or remove one of the clauses.
>
>
> I am stuck writing the file back out and compiling again... writing the
> clauses back out takes me 45 seconds
> |
>
>

> ?-statistics(cputime,S),tell(fooff),ignore(((el_assertions:el_holds_pred_impl(F)),between(2,16,A),current_predicate(el_assertions:F/A),functor(P,F,A),forall(el_assertions:P,format('~q.~n',[P])),fail)),told,

> statistics(cputime,E),TotalisE -S.
> Total=45.97321134599997.
>
> |
>
>
>
> Now I will need to qcompile what I just wrote out (this will take me
> another 277.090 secs to get it to QLF for the next time)
>
> |
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g
> "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt."
>
> |
>
>
>
>
> 7 minutes to update a single clause seems a bit too much !
>
>
>
> But I decide maybe there is a better Idea
>
>
>
>
>
>
>
> |
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> Welcometo SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam
> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,

> andyou are welcome to redistribute it under certain conditions.

> Pleasevisit http://www.swi-prolog.org for details.
>
>
> Forhelp,use?-help(Topic).or?-apropos(Word).
>
>

> ?-time(['el_assertions.qlf']).
> %3,648inferences,6.205CPU in6.213seconds (100%CPU,588Lips)
> true.
>
> ?-retract(el_holds(expansion,interArgIsa,[implies,[and,[admittedSentence,'?SENT'],[isa,'?INDEP-INS',':ARG3'],[operatorFormulas,':ARG1','?SENT'],[argN,'?DEP-INS',':ARG4','?SENT'],[argN,'?INDEP-INS',':ARG2','?SENT']],[isa,'?DEP-INS',':ARG5']],'BaseKB',_))

> ?-qcompile_predicates('el_assertions.qlf',[foo/1,bar/2],[QSave/QCompileOptions]).

> |
>
>
>
> The code is there we just need to expose it to prolog?
>
>
>
>
> Also would this be good for the persistency module?
>
> (Oh sinking feeling after writing this email ...maybe the persistency
> module already does this!)
>
> I am going to send the email anyways.
>
>
> - Douglas
>
>
>
>
>
>
>
>
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "SWI-Prolog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swi-prolog+...@googlegroups.com
> <mailto:swi-prolog+...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/swi-prolog.
> For more options, visit https://groups.google.com/d/optout.