The senior figure, who is close to Mr Cameron, made his remarks after being asked about turmoil in the party's ranks over Europe and plans to introduce same-sex marriage.
He said: "There's really no problem. The MPs just have to do it because the associations tell them to, and the associations are all mad, swivel-eyed loons."