The charge limit can be reduced on the Multiplus but it will not be smooth just a PWM type control so peak current will still be very high generating a lot of electromagnetic noise.
The Smart shunt can not know when battery is full and only guess at best.
- Always send first Ongrid solar to house loads.
This is done by default due to the nature of the setup.
- Have the 12 panels + SBMS + Victron contribute all they can to the Loads up untill 50% of battery capacity.
I do not think that is possible from Victron software as the Victron can only work as charger or as inverter (there is no separate hardware) and if there is an AC input then it will just bypass that to the Essential loads. There may be some function to supplement the AC input if that is limited to some set current but not sure what are those set limits and if it will help in any way in your case.
- Remaining 50% left as UPS mode.
You can setup the SBMS120 EXT IO4 as type 3 and set the limit at 50% SOC then charging will be requested from the Victron only when battery SOC is below 50%
-In Winter or cloudy days Ongrid solar ideally could help charge battery if 12 panels are struggling
Essential load is powered by the 18 panels so is the rest of the house thus if the 12 panes are struggling the 18 panels will not do much better and likely not produce enough to both power all your AC loads and have extra to charge the battery.
-If Multi could talk to SBMS I will touch the sky ;-)
SBMS is master only and will not take any input from any device.
What could Multiplus communicate to SBMS120 that SBMS120 not already knows ?
The only thing Multiplus needs to know from SBMS120 is provided by the two wires EXT IO4 allows battery charging from grid and ECT IO3 allows battery discharge.
SBMS120 knows all about individual cell voltages and battery current. I can not imagine what you think Victron multiplus can provide to SBMS120 that will be useful.
When it is sunny there is more than enough energy from 12 panels to fully charge the battery and likely there will also be excess energy that can not be used unless you have some place to dump that excess energy like a thermal storage.
When it is cloudy most likely you will need to use grid energy to power your house as likely your consumption is higher than what panels produce and the small amount of battery stored energy.
You could have connected all PV panels to grid that way all PV panels energy can be exported to grid when there is excess. Then battery will have been charged only from Victron multiplus using PV energy or if that is not available then grid.