Jake,
If you are looking to put together a solution, then there are 2 ways to do what you want, using Plivo:
1. Plivo Framework:
www.plivo.com/framework/ - (Open source) Use an ATA hardware device which on one end connects to a Analog phone jack and on other side allows IP. Connect this IP side to your Plivo/FS Install Server via SIP and then you can play around with things.
2. Plivo Cloud:
www.plivo.com/ - (Commercial hosted Option) Use an ATA hardware device which on one end connects to a Analog phone jack and on other side allows IP. Connect this IP side to your endpoint on Plivo and you are free to play around then.
The differences in both options will be that first is open source and will allow you to even modify the media switching part incase you need to. I would love to do this if I was looking to hack around around with things. :)
Second option will allow you to focus only on your app, and leave the telephony plumbing to us. Useful when you want to focus on going to market and not worry about scaling issues etc..
Both options will require decent effort from your side. Since you mentioned you are looking to build your own vs pick up a packaged option, I am assuming you are up for it.
Cheers