To develop a custom DRM you would build a custom security handler.
Another approach to build a custom security handler is described in
the following article:
In the article developer is using Java, however the same technique
works in C# and other supported languages.
For viewing documents online (i.e. in the browser) your developers may
also want to take a look at SilverDox (http://www.pdftron.com/ silverdox). SilverDox could be used for secure online document viewing
of your DRM protected documents on Windows and Mac without any
software installation beyond Silverlight. You can access some online
samples from the following page: http://www.pdftron.com/silverdox/samplecode.html