Drools 6 Incremental Compilation of Large Rule Base (10,000 to 20,000 rules)

[Brian Dolbeare]

Question:  What is the best way to handle compilation of 10,000 to 20,000 rules in Drools 6?  One complication is that some of the rules might contain invalid DRL that won't compile but I won't know that in advance since the rules are stored in a database and can be inserted/updated by end users at any time.

Background:  My office has a custom web app for creating rules.  These rules are stored in Oracle in a proprietary XML format which we convert to DRL (legacy project converted to Prolog).  The application allows user to create/update/suspended/delete rules at any time.  There are currently 15,000 active rules and they are run against ~2 million transactions a day.  

In a previous project, I used Drools 5.4 and was able to use the code below to quickly compile rules in multiple threads.  Our team was worried about the potential that a rule might not be converted from the proprietary XML to DRL correctly so, we compiled each rule independently and made sure it had no errors.  We collected the KnowledgePackages of the rules that compiled successfully and after all rules were compiled, we created a new KnowledgeBase and added all of the KnowledgePackages we had collected.  The process was a bit slow (around 100 ms to 300 ms per rule), so we used a ExecutionService to do it in a multi-threaded manner when creating the initial KnowledgeBase.

// Code that compiled each rule

final KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();

kbuilder.add(ResourceFactory.newReaderResource(new StringReader(drlString)), ResourceType.DRL);

final KnowledgeBuilderErrors errors = kbuilder.getErrors();

if (errors.size() > 0) {

throw new CompileException("Could not compile rule: " + drlString);

}

return kbuilder.getKnowledgePackages();

// Code that created the KnowledgeBase from the compiled rules

final KnowledgeBase kb = KnowledgeBaseFactory.newKnowledgeBase();

kb.addKnowledgePackages(knowledgePackageCollection);

In Drools 6, it seems like the code above is no longer a good way to proceed since that API is only available through the legacy adapter (Knowledge API Legacy5 Adapter  ==> 'org.drools:knowledge-api').  Here are some things I tried:

The first approach I considered is to just add all of the rules to the KieFileSystem when the system loads and compile them all together which is fast (10s of seconds).  The only problem with this approach is if one of the rules added has a compilation error then the KieBase created is empty (i.e. no rules at all as per documentation).  I suppose we could inspect the compilation results for errors and then remove any files containing errors from the KieFileSystem; however, I'm not sure how reliable the Drools compiler is at identifying the files that actually have the problem.  Advice???

 for (Rule rule : ruleService.getAllRules()) {
 // Add the rule to the file system and try to compile it
 kieFS.write(fileName, rule.getDrl());
 
 final KieBuilder kieBuilder = kieServices.newKieBuilder(kieFS).buildAll();
 kieBuilder.getResults().getMessages().forEach(m -> LOG.info("{}", m));
 assertEquals(0, kieBuilder.getResults().getMessages(org.kie.api.builder.Message.Level.ERROR).size());
 }

Another approach I tried is to iterate through all of our rules and compile them one at a time discarding any rules that fail to compile; however, this approach is very slow for the initial compilation of all rules as you might expect (over 5 minutes).  This solution would work for incremental compilation but is too slow for building the intiial KieBase.  Unfortunately, there does not seem to be a way to compile each rule independently and then merge them together in the same KieBase (as I was able to do in Drools 5) or to persist each compiled rule independently (unless I'm missing something).  

 for (Rule rule : ruleService.getAllRules()) {
 // Add the rule to the file system and try to compile it
 kieFS.write(fileName, rule.getDrl());
 IncrementalResults results = ((InternalKieBuilder) kieBuilder).createFileSet(fileName).build();


 // Make sure we don't have any errors. If we do, then we need to remove this rule
 if (results.getAddedMessages().size() != 0)
 {
 LOG.error("Unable to compile rule {}.  DRL: {}", rule.getKey(), rule.getDrl());
 kieFS.delete(fileName);
 
 results = ((InternalKieBuilder) kieBuilder).incrementalBuild();
 assertEquals(0, results.getAddedMessages().size());
 assertEquals(0, results.getRemovedMessages().size());
 }
 }

By the way, Mark's response on the following thread was very useful for learning about how to accomplish incremental compilation (it would be great if the incremental option bubbled up to the public API at some point so we don't have to use the internal APIs):

https://groups.google.com/forum/#!searchin/drools-usage/database|sort:date/drools-usage/JdR-ccfsb94/-AGaWD0vBQAJ

https://github.com/droolsjbpm/drools/blob/master/drools-compiler/src/test/java/org/drools/compiler/integrationtests/IncrementalCompilationTest.java

PS, Thank you guys for all the great work you are doing with Drools - it's a great product!

[Ian Kundil]

What about this:

First compile file by file and  save compiled packages:

...

ieBuilder kBuilder = KieServices.Factory.get().newKieBuilder( kfs );
kBuilder.buildAll();

if (kBuilder.getResults().hasMessages(Message.Level.ERROR)) {
    List<Message> errors = kBuilder.getResults().getMessages(Message.Level.ERROR);
    LOGGER.error("Drools compilation errors:{}",errors);
   
    continue;
}

kfs.generateAndWritePomXML(rid);

KieContainer kContainer = kieServices.newKieContainer(rid);
KieBase kBase = kContainer.getKieBase();

Collection<KiePackage> compiledPackages = kBase.getKiePackages();

....

Second - put all compiled packages together:

KieModuleModel kieModuleModel = kieServices.newKieModuleModel();
kfs.writeKModuleXML(kieModuleModel.toXML());
kfs.generateAndWritePomXML(rid);

KieBuilder kBuilder = KieServices.Factory.get().newKieBuilder( kfs );
kBuilder.buildAll();

KieContainer kContainer = kieServices.newKieContainer(rid);

KnowledgeBaseImpl kBase = (KnowledgeBaseImpl)kContainer.getKieBase();

Collection<KnowledgePackage> allPacksTrans=Collections2.transform(allPackages, new Function<KiePackage, KnowledgePackage>() {
    @Override
    public KnowledgePackage apply(KiePackage kiePackage) {
        return (KnowledgePackage) kiePackage;
    }
});

kBase.addKnowledgePackages(allPacksTrans);