What you're describing should work. How are you triggering the mapping? I'll attach a sample program (Toggle A.mssp) that toggles variable A every time the mod wheel (CC1) changes.
I'll also attach a program (Bitmask.mssp) that stores the state of toggles in one variable. Instead of useing the units column and 10s column like you tried this example uses to first bit and the second bit. MSS lets you do bitwise operations (& - And, | - Or, ^ - Xor) so that should simplify the equations. So to toggle bit 1 you could do something like:
Note that I'm only dividing my 3 because that is the max value of the variable. To check whether the first bit is set you could do:
In the sample program changing Mod or Pitch Bend will toggle a bit in A. When A changes, the state of each bit will be stored in the second and third variable.
Anyway let me know if you can get the simple case with just one toggle working.