I have seen it done this way:
Jenkins for glue throughout.
Smoke tests embedded in the CM code. For example, if you are using Chef and Test Kitchen, you get serverspec tests by default. That gets you the ability to do things like "this port should be listening" and "this service should be running", but not easily the user-experience tests (clicking, etc).
Next, people add such web tests - I like to use Capybara + rspec for simplicity, but Capybara + selenium works better but is more complicated.
Then, setup Jenkins to watch the CM repo, and kickoff a test kitchen full run on whatever cycle you want. Per-push, hourly, per-merge, whatever you want.
You'll also need some glue to get the test results out of rspec to appear in a sane way in Jenkins. Bizarrely, you may have to translate them into jUnit report format; I would love to hear better ways.
At that point, you have a system in which a push on the CM repo triggers a build, smoke test with clicky-clicky, results into a format Jenkins can ingest; jenkins will chart the results over time.
It's not something an ops person would be likely to feel OK setting up; may be able to contribute. But capybara, Jenkins, etc are more developer-space; you'd need a "goat" or a developer; pure-ops person will not be happy. And it is pretty brittle. Improvements?
--Clinton