The Athill family appears to have been distinguished in Antigua; George Athill (1806-1857), of Hildenborough, Tonbridge, Kent, farmer, Diana Athill's great-grandfather (m. 1845, Charlotte, née Tappenden- very probably, based on her age in census records [as also given at the UCL page below] and lack of competing records, she b. 1816, dau. of Thomas and Charlotte Tappenden, of Ashford, Kent) was (apparently a natural) s. of James Athill (1759-1822), Chief Justice of Antigua. George Athill was allegedly a 'free man of colour'; James Athill died a bachelor, but with three natural children- Vere Langford Oliver, author of 'The History of the Island of Antigua' (vol. I featuring a pedigree and considerable detail on the Athill family- see
https://archive.org/details/historyofislando01oliv/page/10), provides the text of his will, at the head of which is observed that the testator 'leaves legacies to certain mulattos' [this shown on page 11 of the volume linked above; it reads: 'Independent of other transactions between me and the co-partnership of John and Sam Athill* I have advanced £1100
c.at 5 per cent. I remit the interest and leave this sum to them in trust for their brother Geo.*' (* here Oliver notes beneath the text 'natural sons of testator')
Oliver traces two generations back from James Athill, thus: he s. of Dr James Athill (1727-1779), of Antigua, physician, Member of the Council, by his wife Anne (1737-1802), dau. of Samuel Redhead. Dr J. Athill s. of John Athill (-1732/3) of Guestwick, Norfolk, and his wife Elizabeth (c. 1690-1768 [d. aet. 78]).
George and Charlotte's son was Rev. Herbert Athill (1851-1919), rector of Digswell, Herts., as above.
The above based on the book '
Legacies of British Slave-ownership: Colonial Slavery and the Formation of Victorian Britain', by Catherine Hall et al, Cambridge University Press, 2014, some of the research from which featured here:
https://www.ucl.ac.uk/lbs/person/view/1 and
https://www.ucl.ac.uk/lbs/person/view/2146640977, which identifies George as brother of John and Samuel Athill, and all three as natural sons of James Athill, as above.