If you subtract two instance vectors, then only the elements with *exactly* matching labels will be retained, as documented
here. In your case the LHS and RHS have different labels, so you will get an empty result.
But you make the LHS and RHS match on specific labels, or ignoring specific labels, as documented
here.
In your example, you can try this, assuming all the other labels match:
namedprocess_namegroup_memory_bytes{groupname="map[:process.jar]", memtype="virtual"} - ignoring(memtype) namedprocess_namegroup_memory_bytes{groupname="map[:process.jar]", memtype="residential"}
Note that I changed the label matching from not-equal to equal, to be sure there's only one value on the LHS which matches the value on the RHS, assuming all other labels are the same.
If you need to combine multiple values for different memtypes on either the LHS or the RHS then you will need to sum(...) over them.