Silent Mode only affects ringer and notification sounds, leaving media and alarm volumes unaffected. On recent Android versions, Ringer Volume also affects both ringer and notification sounds, but that's because of a (inexplicable) change in Android, where ringer and notification volumes are inseparably linked. Both Silent Mode and Ringer Volume *should* allow you to silence the ringer and notifications, but using Silent Mode allows you to return to the previous state with a simple Silent Mode Off action, whereas with Ringer Volume you would have to set the volume to a specific value.
I've never heard of a phone that didn't respond properly to Silent Mode, so I'm with Pak in thinking that what you heard while the phone was silenced was something other than a ringer or notification. But if that sound is indeed what you need to silence, you'll need to change the appropriate volume setting (or all of them).