Iterate trough groups of groupby result

[Esben Thomsen]

Hey, 

I am trying to implement the nurse rostering optaweb project for a university project. 

I am having trouble understanding how to work with the result of a groupby statement when using constraintstreams. 

One of the things I am trying to do is adding the groupby count together so f.x I have some normal shifts of 8 hours and abnormal 12 hour shift that overlap two of these, so I need to be counted in both the day shift and evening shift. So I can groupby shift type (day8, day 12, evening8 etc.) but how can I go trough the result and add the count of two groups together? 

When I search for a similar thing in java streams it always return as a map they can use, but here we get constraint streams so how can I make it work with that?

If all this is obvious please bear with me as I am learning java for this project. 

[Christopher Chianelli]

Is the count going directly to a penalize (for example, if your constraint is "minimize the total number of normal and abnormal shifts"), then you can model it as two separate constraints (one for normal, another for abnormal) like so:

```

Constraint minimizeNormalShifts(ConstraintFactory constraintFactory) {

    constraintFactory.from(Shift.class)

                                   .filter(Shift::isNormal)

                                   .join(Employee.class, Joiners.equal(Shift::getEmployee, Function.identity())

                                   .groupBy((shift, employee) -> employee, countBi())

                                   .penalize("Minimize Normal Shifts", HardSoftScore.ONE_SOFT, (employee, count) -> count);

}

Constraint minimizeAbnormalShifts(ConstraintFactory constraintFactory) {

    constraintFactory.from(Shift.class)

                                   .filter(Shift::isAbnormal)

                                   .join(Employee.class, Joiners.equal(Shift::getEmployee, Function.identity())

                                   .groupBy((shift, employee) -> employee, countBi())

                                   .penalize("Minimize Abnormal Shifts", HardSoftScore.ONE_SOFT, (employee, count) -> count);

}

```

If there a filter or calculation that depends on the combined count, then the above wouldn't work (example: "The sum of a nurse's normal and abnormal shifts can be no more than 10").

You can use "join" to join two seperate constraint stream as follow for those cases:

```

Constraint minimizeNormalShifts(ConstraintFactory constraintFactory) {

    constraintFactory.from(Shift.class)

                                   .filter(Shift::isNormal)

                                   .join(Employee.class, Joiners.equal(Shift::getEmployee, Function.identity())

                                   .groupBy((shift, employee) -> employee, countBi())

                                   .join(constraintFactory.from(Shift.class)

                                                                           .filter(Shift::isAbnormal)

                                                                           .join(Employee.class, Joiners.equal(Shift::getEmployee, Function.identity())

                                                                           .groupBy((shift, employee) -> employee, countBi())

                                    )

                                   .filter((countNormal, countAbnormal) -> countNormal + countAbnormal > 10)

                                   .penalize("More than 10 normal or abnormal shifts combined", HardSoftScore.ONE_HARD, (countNormal, countAbnormal) -> countNormal + countAbnormal - 10);

}

```

[Esben Thomsen]

Okay this seems like an expensive way to do calculations and maybe impossible in my case. I'm trying to group by day and then some other group key on a uniconstraint and then do some calculation on the result. For instance if in my project:

Constraint assignEveryShift(ConstraintFactory constraintFactory) {
return constraintFactory.fromUnfiltered(Shift.class)
.filter(shift -> shift.getEmployee() != null)
.groupBy(shift -> shift.getStartDateTime().toLocalDate(),
shift -> shift.getShiftTypeAndJob(),
count())

The getShiftAndJob() is because as I understand it is only possible to group by two keys, so it is a concatenation of two keys. 

So a key is day8_IntensivecareNurse, day8_normalNurse, day8_nurseAssistant, evening8_intensivecareNurse...night12evening_nurseAssistant

28 groups (days) with each 15 subgroups (shiftType and job) with a count 

it gives a triConstraint stream of "localDate day, String shiftTypeAndJob, Integer count"

the evening shift is both day12 and evening8 so to see how many is in the evening shift and how intensive care nurse are in it I need to do something like

this for each LocalDate day, (day12_IntensivecareNurse+evening8_IntensivecareNurse)/(day12_IntensivecareNurse+evening8_IntensivecareNurse+day12_normalNurse+evening8_normalNurse)

So is there a way to this without doing crazy amount of joins? Which I also would assume is quite expensive computationally. 

[Christopher Chianelli]

Joins aren't expensive; they are indexed (ConstraintStreams uses incremental calculation so it only recalculates what need to be recalculated). Although it might run on all N * M pairs on the first run, on each other run, we only check the impacted pairs (for instance, if you use the `equal` joiner on Employee, and Shift A's employee change to B, it only checks the Shifts who employees are B and ignore all others, changing it from needing to run on 10000 pairs to 100 (as an example). What isn't indexed are filters, which run on all possibly impacted tuples (so if Shift A changes in any way, then the tuples (*, Shift A), (Shift A, *) (where * is anything) are tested against the filters; the tuples (Shift B, Shift C) is not tested though (since it could not be impacted by Shift A changing).

Just a note: the join API only works on classes and UniStreams, so you cannot join a stream whose tuples have more than 1 element to another stream.

I notice you did

```

constraintFactory.fromUnfiltered(Shift.class)
                                .filter(shift -> shift.getEmployee() != null)

```

assuming Employee the only planning variable, that is exactly the same as

```

constraintFactory.from(Shift.class)

```

which is more efficient (note: in overconstrained planning (the employee planning variable is nullable)), then you still need the null check.

Is the constraint something like "the number of nurses in the hospital must be at least N each morning"/"the number of nurses in the hospital must be at least N each evening"? Then something like this should work:

```

Constraint atLeast10NursesInEvening(ConstraintFactory constraintFactory) {

    return constraintFactory.from(Shift.class)

                                               .filter(shift -> isShiftInEvening(shift)) // isShiftInEvening return true if it 12 hour day or 8 hours evening

                                               .groupBy(shift -> shift.getEmployee(), countDistinct())

                                               .filter(employeeCount -> employeeCount < 10)

                                               .penalize("Less than 10 nurses in the evening", HardSoftScore.ONE_HARD, employeeCount -> 10 - employeeCount);

}

```

[Esben Thomsen]

Okay thanks for the help, I think it will help me somewhat. In the constraint you created in the end wouldn't it take all the 12hour and 8hour shift for the planning period and not just for a specific day? So I would still have to do a time group by?

[Christopher Chianelli]

Good catch! It can be fixed by adding a key to the groupBy:

```

Constraint atLeast10NursesInEvening(ConstraintFactory constraintFactory) {

    return constraintFactory.from(Shift.class)

                                               .filter(shift -> isShiftInEvening(shift)) // isShiftInEvening return true if it 12 hour day or 8 hours evening

                                               .groupBy(shift -> shift.getStartDateTime().toLocalDate(), shift -> shift.getEmployee(), countDistinct())

                                               .filter(employeeCount -> employeeCount < 10)

                                               .penalize("Less than 10 nurses in the evening", HardSoftScore.ONE_HARD, employeeCount -> 10 - employeeCount);

}

```

[Christopher Chianelli]

A correction to the correction (countDistinct needs to count the employees, not the shifts):

```

Constraint atLeast10NursesInEvening(ConstraintFactory constraintFactory) {

    return constraintFactory.from(Shift.class)

                                               .filter(shift -> isShiftInEvening(shift)) // isShiftInEvening return true if it 12 hour day or 8 hours evening

                                               .groupBy(shift -> shift.getStartDateTime().toLocalDate(), countDistinct(Shift::getEmployee))

                                               .filter((date, employeeCount) -> employeeCount < 10)

                                               .penalize("Less than 10 nurses in the evening", HardSoftScore.ONE_HARD,(date, employeeCount) -> 10 - employeeCount);

}

```

[Esben Thomsen]

Okay so I still need to do the join because I still need to count if there is enough intensivecare nurses. I am having trouble with the join and I don't know if it is the thing you mentioned with the uniConstraint stream limitation earlier. The first groupby is fine in the editor (IntelliJ) but the second groupby there is an error saying "Bad return type in lambda expression: LocalDate cannot be converted to GroupKey_". Should I be able to join the streams? If so what am I doing wrong? Below is the code:

Constraint eveningShiftHasSkilledNurses(ConstraintFactory constraintFactory) {
return constraintFactory.from(Shift.class)
.filter(Shift::isEveningShift)
.filter(Shift::hasSpotSpecialSkills)

.groupBy(shift -> shift.getStartDateTime().toLocalDate(),
countDistinct(Shift::getEmployee))

.join(constraintFactory.from(Shift.class)
.filter(Shift::isEveningShift)
.filter(Shift::hasSpotSpecialSkills)
.groupBy(shift -> shift.getStartDateTime().toLocalDate(),
countDistinct(Shift::getEmployee)))