Pareto scoring now that custom scores are deprecated

[Jon Britton]

Hi,

Following on from this stackoverflow post, we have been using custom scores to implement pareto scoring, but custom scores are now deprecated. 

The docs mention that pareto scoring is not supported in OptaPlanner, but is there currently any plan to implement it?

Thanks,

Jon

[Geoffrey De Smet]

I think we forgot about Pareto scoring when we decided to deprecate custom scores implementations.

Sooner or later, we have to supply Pareto scoring out of the box, which doesn't imply we have to reinstance custom scoring.

Jon, would you be so kind to give us some insight into your pareto scoring usage?

Questions

1) What field of which type does your customer score have? All ints? How many?

2) Did you overwrite the BestSolutionRecaller? And how do you retrieve the set of pareto solutions?

3) How many pareto solutions do you retain?
What if you encounter billions (think out of memory error) that are pareto equal?

4) How do you assert they are all pareto solutions in your set are unique? (no duplicates)

5) What's your score calculation speed with all those.

Solution proposals

A) Reinstate custom scoring.

Meh - let's remember the set of reasons we had to get rid of it.

B) Support Pareto scoring out of the box.

We'd need a clear spec and a quickstarts to make sure that we're building it the right way.
See questions above.

Z) Status quo (for now)

With kind regards,
Geoffrey De Smet

--
You received this message because you are subscribed to the Google Groups "OptaPlanner development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to optaplanner-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/optaplanner-dev/3cde2f9d-8b7b-4f19-967d-f674581a072dn%40googlegroups.com.

[Jon Britton]

Hi Geoffrey, thanks for the response.

In response to your questions:

1) We have 3 integer score values, one of which is the hard score and the other two are the dimensions used for pareto scoring. One of these actually come from a double value which we multiply by a large number to convert to int.

2) Yes, we overwrote the BestSolutionRecaller. The pareto front is a an in-memory map of ParetoScore -> Solution_.

3) We retain all of them. We didn't think about memory implications, at least not yet - this is not a problem we are seeing in our specific case).

4) Only one solution is maintained per score (arguably a bug, arguably a feature). This is managed by the fact that the pareto front is a map of score to solution.

5) The score calculation, according the the OptaPlanner logs is around 230,000/second. Obviously this depends heavily on the problem we are solving and our scoring implementation.

Thanks,

Jon

[Geoffrey De Smet]

Thank you Jon for sharing some insight on pareto solving for your use case.

I favor proposal B as the way forward (sooner or later).
I expect that HardSoftParetoScore would have a hard int array and a soft in array, which would fit your requirements.

How do you extract your set of best solutions from the solver?
For proposal B, we'd have to figure out how to change the Solver and SolverManager API to accommodate that, by exception...

With kind regards,
Geoffrey De Smet

To view this discussion on the web visit https://groups.google.com/d/msgid/optaplanner-dev/6f800111-ca26-4b82-aa15-480044556445n%40googlegroups.com.

[jake forsey]

Hi,

I'm working with Jon on this project!

We would love Pareto scoring to be properly supported by Optaplanner and are happy to spend some time helping however we can.

Q: How do you extract your set of best solutions from the solver?
A: We don't have a neat solution for this, just something practical: we use the existing BestSolutionChangedEvents to work out when the pareto front has been updated, then we use reflection to get our customized BestSolutionRecaller from the Solver which contains the pareto front map.

We haven't looked at updating the Solver / SolverManager API.

How do you think we should progress? Would you be interested in a working (albeit ugly) fork with our current approach? 

Jake

[Geoffrey De Smet]

Thanks Jake, this is very useful to know.

At the moment, I am just gathering requirements for pareto optimization.
It affects the API of on another feature we want to develop:
  https://issues.redhat.com/browse/PLANNER-2793

Expectation wise, a pareto optimization API isn't in the short term roadmap
because it's a rare use case and it's too soon to write it into stone.


An intermediate step is probably to support a ParetoHardSoftScore with int[] hardScore and int[] softScore,
similar to BendableScore (pareto size determined at compile scope through a @PlanningScore argument).
so your workaround works again with the latest version.
Again, no promises roadmap wise though.

With kind regards,
Geoffrey De Smet

To view this discussion on the web visit https://groups.google.com/d/msgid/optaplanner-dev/d42d432d-b622-4eed-9154-03f091a97503n%40googlegroups.com.