Several peerage titles, specifically Earldoms and even Marquessates, omit the word “of” as part of the title, e.g. among Earldoms: Alexander of Hillsborough (1963); Alexander of Tunis (1952); Amherst (1826); Annesley [I] (1789); Attlee (1955); Baldwin of Bewdley (1937); Bathurst (1772); Beatty (1919); Beauchamp (1815); Beaulieu (1784); Belmore [I] (1797); Brassey (1911); Brooke (1746); Brownlow (1815); Bruce (1821); Buxton (1920); Cadogan (1718); Cadogan (1800); Cairns (1878); Camden (1786); Canning (1859); Castle Stewart [I] (1800); Castleton (1720); Cathcart (1814); Cave of Richmond (1928); Cawdor (1827); Clinton (1746); Compton (1812); Coningsby (1719); Conyngham [I] (1781); Conyngham [I] (1797); Cornwallis (1753); Cowley (1857); Cowper (1718); Curzon of Kedleston (1911); de Grey (1816); De La Warr (1761); de Montalt (1886); Digby (1790); Egerton (1897); Erne [I] (1789); Farquhar (1922); Fauconberg (1756); Fife [I] (1759); FitzHardinge (1841); FitzWalter (1730); FitzWilliam (1716, 1746); Gower (1746); Grandison [I] (1721); Grandison [I] (1767); Granville (1715); Granville (1833); Grey (1806); Grosvenor (1784); Haig (1919); Howe (1788); Howe (1821); Innes (1837); Jellicoe (1925); Jermyn (1826); Jowitt (1951); Ker (1722); Kitchener (1914); Landaff [I] (1797); Ligonier (1766); Ligonier [I] (1776); Lloyd-George of Dwyfor (1945); Loreburn (1911); Ludlow [I] (1760); Macartney [I] (1794); Manvers (1806); Mount Cashell [I] (1781); Mountbatten of Burma (1947); Nelson (1804); Nugent [I] (1776); O’Neill [I] (1800); Panmure [I] (1743); Peel (1929); Percy (1766); Rivers (1466, 1626, 1641); Roberts (1901); Russell (1861); Somers (1821); Sondes (1880); St Aldwyn (1915); St Maur (1863); St Vincent (1797); Stanhope (1718); Strange (1786); Sydney (1874); Talbot (1761); Talbot (1784); Temple (1749); Temple of Stowe (1822); Vane (1823); Verney [I] (1743); Wandesford [I] (1758); Wavell (1947); Whitworth (1815); Winterton [I] (1766). And among Marquessates: Camden (1812); Conyngham [I] (1816); Cornwallis (1792); Curzon of Kedleston (1921); Duoro (1814); Grey (1740); Townshend (1787); Wellesley [I] (1799). This usually occurs for a few reasons, i.e. (1) the peerage title is the family name, e.g. Amherst, Annesley, Cadogan, Cowley, Cowper etc; (2) in some cases it is omitted when the title was the surname (not of the party ennobled but only) of a family from whom the grantee derived descent, e.g. Beauchamp, Brooke, Brownlow, Ferrers, Innes, Manvers, Sondes etc; (3) Irish peerages sometimes leave out the locale connected with the title and are thus the anomalous cases of the other two, e.g. Belmore, Castle Stewart, Erne, Fife, Mount Cashell, Winterton etc. [S.S.]
I hope I did not miss any :)
S.S.
Not much different here from what SS already said, but for what it's worth, here is what Cokayne said in The Complete Peerage:
"When the family name is taken as the Peerage title it is more usual in the case of Earldoms, or even of Marquessates, to omit the word " of "; e.g. (among Marquesses) Conyngham, Comwallis, Townshend, &c; also (among Earls) Amherst, Annesley, Bathurst, Cadogan, Cairns, Cathcart, Cowley, Graham, Grey, Fitzwilliam, Howe, Nelson, Poulett, Russell, Spencer, Stanhope, Waldegrave, &c. In some cases "of" is omitted when the title was the surname (not of the party ennobled, but only) of a family from whom the grantee derived descent, e.g. (among Earls) Beauchamp, Brooke, Brownlow, Ferrers, Granville, Innes, Manvers, Sondes, Strange, Sydney, Vane, &c. Contrariwise, the word "of" is sometimes retained (more especially when the surname is of local origin), e.g., Earl of Berkeley, Earl of Coventry, Earl of Craven, &c. The Marquess Camden, Earl Cawdor, &c„ seem, as English titles, somewhat anomalous, tho', in the Irish peerage, names of places are not unfrequently so treated, e.g., Earl Belmore, Earl Castle Stewart, Earl Erne, Earl Fife, Earl Mountcashell, Earl Winterton, &c.”