In the first case (1) it is obvious that the first two phalanxes will be under the third, because they are completely shaded. In the second case (2), the third phalanx has both light and shadow, as well as a hand. The light of the phalanx must pass into the light of the hand, and the shadow of the phalanx into the shadow of the hand. But the phalanx and the hand are two different groups that can completely overlap each other. How can I make the overlap of groups in this case?
I'm assuming you're using the "sandwich" in one level. so this example shows how to shade in one level.
THE DRAWING
In most cases I find it easier to use a few styles with lowered alpha that make up the shading. In the attached photo I use only one "Shadow style" but you can use two or three shadow styles to build up shading using different effects to composite transparent layers on top of each other. Any "shaded area" can be made from one or more straight strokes so shading this way can be done very quickly with the added benefit of very predictable auto-inbetweening. each finger in the above image has shadows within the group that makes up the finger. There are also shadow strokes and shapes at the very top of the images stacking-order to bring the fingers shadows in over the main mass of the hand.
THE EFFECTS
The idea here is to isolate any areas of the image that are in shadow, adjust color, and then composite it over the original image with shadows removed.
in the schematic window of the attached image, the RGBkeys are used to remove any areas where colors are unaffected by shadow strokes. in this example, there are 3 RGBkeys, one to remove areas of pure color for style 4, one to remove areas of pure color for style 5, one to remove areas of pure color for style 6...... unfortunately the RGB values will need to be punched in manually and then repeated for every color in your drawing (a color picker feature for RGBkey would be a great addition to the effect); when working with a rig (where parts of the fingers and hands are separated onto different columns), isolating the shadow is ultimately much easier.
after the RGBkeys, an Adjust Levels node is added which takes all the mixed colors of the shaded areas and outputs one color (in my case a dark purple) with a lowered alpha. then a blur node is added but you don't really need to add that one. Then a Matte In node is added so that the shadows are only visible within the silhouette of the hand.
perhaps this image with everything selected will give you a better idea of how everything goes together. This and a few other shading techniques would probably be better explained in a video showing how everything is set up............
There are a few different approaches depending on the situation. If the parts of the hand are placed on separate columns, this would involve a different effects setup than what I have in this picture; the node setup would be more complex than the one in the picture above but once set up and saved as a preset, it will be much easier to apply to future projects than what is in this picture.