While you can limit individual processes and other such methods, as you have found, it is not possible to limit per-database in MongoDB. The amount of physical RAM (resident memory) used by the process is actually managed by the OS and is based on the usage of the memory mapped data by the database.
If you wish to have multiple users with different data sets running on a host with no contention, then the sum of the size of the working data set for each user/db would have to be smaller than the total physical RAM available to MongoDB, otherwise you will see contention in the form of increased faults to disk (you will see some anyway as old data is paged out etc.). There is no way around that (beyond partitioning the instances using virtualization) at the moment, nor can I think of an obvious way to do it.
Adam