Hi Douglas,
I think you want something similar to the RDF database. Here, a graph is
represented by two files (which may or may not exist): the _snapshot_ which
uses the internal binary RDF format and the _journal_, which caries the
_changes_ expressed as Prolog terms. Loading the graph implies loading the
(possibly non-existing) snapshot and replay the (possibly non-existing)
changes. A change extends the change journal and there is a call to
write a new snapshot, destroying the journal.
Library persistency does something like that, except that there is only
a journal that may be compacted by writing the current state to a fresh
journal so you effectively get rid of events that cancel each other.
The snapshot could indeed using the qlf format. Using qlf for the journal
is a bit harder because the qlf format is stateful.
A disadvantage is that the qlf version depends on 32/64 bit and is version
dependant. It might be wise to consider an alternative, in particular
if we consider only facts. Possibly something based on
PL_record_external(),
which creates a binary representation of a Prolog term that is supposed to
be stable (it changed twice during SWI-Prolog's lifetime).
We'd need some timings. Loading .pl is expensive due to the parsing and
expansion process. Probably library persistency already wins as it skips
a lot of the compilation steps, but still needs the parser and VM compiler.
Basing library persistency on PL_record_external() would get rid of the
parser overhead, but still needs to intern each atom one-by-one and needs
to do the VM code generation.
We could design a new term sequence binary format based on
PL_record_external()
that avoids interning atoms one-by-one. Such a format could also be used to
link up Prolog processes using a pipe or socket to exchange Prolog terms
much
faster than read/write. If we would make the sequence restartable, i.e.,
flush the state, it could be used with library persistency without
significant changes to this library.
Cheers --- Jan
On 07/26/15 02:12, Douglas Miles wrote:
>
> Hi,
>
>
> I have a file with 5,570,822 prolog clauses.
>
> |
>
>
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> Welcometo SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam
> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,
> andyou are welcome to redistribute it under certain conditions.
> Pleasevisit
http://www.swi-prolog.org for details.
> Forhelp,use?-help(Topic).or?-apropos(Word).
> ?-set_prolog_flag(verbose_load,true).
> true.
> ?-['
el_assertions.pl'].
> %
el_assertions.pl compiled intoel_assertions 281.57sec,5,570,822clauses
> true.
>
>
>
> |
>
>
>
> It takes 5 minutes to load
>
>
> So I do the obvious smart things and convert it to a QLF
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g
> "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt."
>
> % 1,817,844,040 inferences, 277.090 CPU in 277.493 seconds (100%
> CPU, 6560478 Lips)
>
>
>
> whereas now it take only 6-7 seconds to load!
>
> |
>
>
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> WelcomeTO SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam
> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,
> AND you are welcome TO redistribute it under certain conditions.
> Pleasevisit
http://www.swi-prolog.org FOR details.
> FOR help,use?-help(Topic).OR ?-apropos(Word).
> ?-time(['el_assertions.qlf']).
> %3,648inferences,6.205CPU in6.213seconds (100%CPU,588Lips)
> true.
> |
>
>
>
>
> YAY!
>
>
> Here is the problem. I'd like to change, add or remove one of the clauses.
>
>
> I am stuck writing the file back out and compiling again... writing the
> clauses back out takes me 45 seconds
> |
>
>
> ?-statistics(cputime,S),tell(fooff),ignore(((el_assertions:el_holds_pred_impl(F)),between(2,16,A),current_predicate(el_assertions:F/A),functor(P,F,A),forall(el_assertions:P,format('~q.~n',[P])),fail)),told,
> statistics(cputime,E),TotalisE -S.
> Total=45.97321134599997.
>
> |
>
>
>
> Now I will need to qcompile what I just wrote out (this will take me
> another 277.090 secs to get it to QLF for the next time)
>
> |
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl -g
> "time(load_files(['el_assertions'],[qcompile(auto),if_needed(true)])),halt."
>
> |
>
>
>
>
> 7 minutes to update a single clause seems a bit too much !
>
>
>
> But I decide maybe there is a better Idea
>
>
>
>
>
>
>
> |
> root@c3po:/opt/PrologMUD/pack/pldata_larkc/prolog/el_holds# swipl
> Welcometo SWI-Prolog(Multi-threaded,64bits,Version7.3.4)
> Copyright(c)1990-2015Universityof Amsterdam,VU Amsterdam
> SWI-Prologcomes withABSOLUTELY NO WARRANTY.Thisisfree software,
> andyou are welcome to redistribute it under certain conditions.
> ?-time(['el_assertions.qlf']).
> %3,648inferences,6.205CPU in6.213seconds (100%CPU,588Lips)
> true.
>
> ?-retract(el_holds(expansion,interArgIsa,[implies,[and,[admittedSentence,'?SENT'],[isa,'?INDEP-INS',':ARG3'],[operatorFormulas,':ARG1','?SENT'],[argN,'?DEP-INS',':ARG4','?SENT'],[argN,'?INDEP-INS',':ARG2','?SENT']],[isa,'?DEP-INS',':ARG5']],'BaseKB',_))
> ?-qcompile_predicates('el_assertions.qlf',[foo/1,bar/2],[QSave/QCompileOptions]).
> |
>
>
>
> The code is there we just need to expose it to prolog?
>
>
>
>
> Also would this be good for the persistency module?
>
> (Oh sinking feeling after writing this email ...maybe the persistency
> module already does this!)
>
> I am going to send the email anyways.
>
>
> - Douglas
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "SWI-Prolog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to
swi-prolog+...@googlegroups.com
> <mailto:
swi-prolog+...@googlegroups.com>.
> Visit this group at
http://groups.google.com/group/swi-prolog.
> For more options, visit
https://groups.google.com/d/optout.