Romance versus reality - James, 4th or 5th Lord Duffus and Lady Mary Hay

[Shinjinee]

I was reading Anthony Glyn (a baronet)'s memoirs of his maternal
grandmother Elinor Glyn (yes, she of the tiger skin!). Lots of
aristocratic connections there - her lovers included Lord Alistair
Innes-Ker (yr brother of the duke of Roxburghe who married May Goelet
and who were grandparents of the present duke) and Lord Curzon (the
very superior person). What was interesting was his mention of the
family legends about the patrilineal ancestry of his grandmother and
great-aunt Lucy Duff-Gordon, nee Sutherland.

Glyn claimed that the family traces its line back to a Laird who was
forced to sell his lands in the 1700s. Nothing surprising there. That
the Laird was apparently son of a younger son of a Lord Duffus, but
the Scots Peerage has him dying unmarried. Isn't the Scots Peerage
generally fairly accurate on these points? I would assume that if he
was a son of a yr son of Lord Duffus that he was illegitimate or the
marriage was not quite up to the mark.

Glyn then mentioned the romantic story of James 4th or 5th Lord Duffus
(5th if we count his father Eric) who eloped with the newly married
Lady Mary Scott, wife of General John Scott of Balgonie. He left ten
natural children with his mistress/ lover who were unable to inherit
naturally. Which led to the Dunbar of Hemprigg family (patrilineally
Sutherland at that point) claiming the lordship in 1827 and a counter-
claim by the 4th/5th Lord's nephew Rev Eric Rudd.. a petition before
the House of Lords and a counter-petition by Rudd in 1838. The
lordship of Duffus is believed extinct since 1875 (the last Sutherland
Dunbar baronet dying unmarried), but there may be collateral
Sutherlands in the Duffus family.. which is where Elinor Glyn and Lucy
Duff-Gordon's father may come in. (He died in 1864).

After checking out this interesting family - Sutherland of Duffus -- I
thought that the name of John Scott of Balcomie sounded familiar. Aha!
Three daughters - two married to eldest sons of peers, and one to a
rising politician. Familiar? No less than Henrietta Scott, wife of the
4th Duke of Portland; Lucy Scott, first wife of Lord Doune, later 10th
Earl of Moray; and Joan Scott, Viscountess Canning. But their mother
wasn't Lady Mary Hay but Margaret Dundas, with impeccable Scottish
establishment connections.

So where is Lady Mary Hay? A little digging and I got the whole sad
story. Wife of James Hay, 15th Earl of Erroll by his first wife (who
remarried and had sons by his second wife). Married off in November
1770 to the rich General John Scott of Balcomie, who made his fortune
in India and via gambling, in lieu of a gambling debt owed by her
father - at the age of 16. The stuff of historical romances... And
unfortunately, she fell for a captain in the regiment of which her
husband was colonel - one Captain James Sutherland of Duffus of the
24th Foot XII. They ran off in December 1771, spent two-three nights
together, were caught in flagrante delicto by the chasing husband (the
seducer jumped out of the window nearly naked), and the silly lady was
abandoned.

The husband promptly divorced her, and apparently equally promptly
remarried, and left three daughters. He died 1775 and left an
interesting will requiring his daughters's husbands to take the name
Scott in addition to their own if they wanted to enjoy the estates.

Here is the rest of the story, extracted from here and there:

According to another source
http://newsarch.rootsweb.com/th/read/SCT-SUTHERLAND/2002-02/1013986496
[Lois Sutherland Wark writes in:]

I was just reading Sutherland: A Fighting Clan by Malcolm Sutherland
(Avon
Books, London, first pub. 1996), in which the author mentions Capt.
James
Sutherland, 5th Lord Duffus, "who can only be described as a scallywag
(see
footnote 140)." The author goes on to say "he died a bachelor and the
Duffus
title passed, not without dispute, to his cousin, Sir Benjamin
Sutherland
Dunbar of Hempriggs."

Here's footnote 140 (pp. 174-175), which suggests your ancestor was
quite the
ladies' man:

James, 5th Lord Duffus (1747-1827. Capt. 1770).

James, 5th Lord Duffus, born 8 June 1747 at Skibo, was the eldest son
of
Capt. Eric(k) Sutherland of Duffus, by his wife and cousin,
Elizabeth,
daughter of Sir James Dunbar (lst Bart.) (formerly Sutherland). The
title and
dignity of Lord Duffus were restored to James by Act of Parliament on
25 May
1826, his grandfather having been attainted after the 1715 Jacobite
uprising.
James served first in the 68th Foot, which was in Jersey in 1763 and
in
Antigua in 1765 to 1767. He appears in the AL as "Ja". His Ensigncy
was taken
over in April 1767 and he was placed on half-pay until Augtust 1768.
The 26th
and 36th Regiments of Foot served in America. McGregor says James
retired on
27 January 1772 and in AL 1772-1826 he is listed as being on English
half-pay
in the 106th Foot. At one time James was the lessee of Barrogill
Castle (now
the Castle of Mey in the possession of Queen Elizabeth, the Queen
Mother).
The lease expired in 1786. Between 1792 and 1799 James was the factor
of the
Earldom and Bishopric Estates in Orkney and a tenant of the Bu of
Burray. He
was regarded by the Orcadians as tyrannical and oppressive and he
forced
tenants of the Estate to build dykes for him at the Bu of Burray.

Although he eloped with Mary (nee Hay, daughter of James, Earl of
Errol),
wife of General John Scott of Balcomie, who divorced her in 1771,
James
abandoned her and died unmarried, at Harley Street, London on 30
January
1827. He was buried in Marlyebone Church on 7 February 1827.

General Scott was the Colonel of James's regiment. He was a
celebrated
gambler whose luck was so phenomenal that "as rich as Scott" became a
by-word. It is said that Scott won the hand of his first wife, Lady
Mary, in
settlement of a gambling debt owing to him by her father, the Earl of
Errol.
A graphic description of Lady Mary's elopement is given on the reverse
of a
letter dated Edinburgh 12 October 1771 from Gregory Grant to James
Grant of
Grant (SRO: GD248/50/2/50). "Captain Sutherland of Duffus has been at
his
cousin and very best friend, General Scott's house in Fife some days
(at the)
latter end of last month and, without any previous particular
suspicion,
eloped with Lady Mary Scott on Wednesday 2nd of this month at 3
o'clock in
the morning. They passed here about 10 o'clock that forenoon and slept
in the same bed for 3 or 4 hours at Durham that night. The General,
with some
gentlemen of the Law etc., set out from hence at 10 o'clock the same
night .
. . The poor unfortunate infatuated Lady with her wretched gallant
most
foolishly stopt on Friday's night within 10 miles of London, at
Barnet, where
the General came up with them before midnight . . . The General got
into
their bedroom so as to see the Captain in his shirt who, without a rag
on
him, jumped out of the window and has not been heard of since." The
General and his wife had been married only since November 1770. The
Process of Divorce is dated 18 December 1771.

In his Will James acknowledged 10 natural children (CSSS Newsletter,
December 1985), one of whom appears to have been Margaret, daughter of
Anne Dunnet of Orkney (CSSS Newsletter, December 1984). However,
according to Orkney tradition James may have had as many as 60
illegitimate children. It is said that the factor's stick and coat at
the farmhouse door was a sign to the farmer that he had better keep
out of his own house.

In his old age, James was reputedly haunted by the prospect that
half-brothers and sisters might marry each other in ignorance of
their
relationship (Private Correspondence: W.P.L. Thomson). Although
referred to
as Lieutenant James Sutherland, it is probably James, Lord Duffus who
received a civil pension of 100 pounds per annum (as did his father,
Erick
from 29 April 1769 until at least the second quarter of 1802 (Le Livre
Rouge
-- Pensions in England, Scotland and Ireland by P.G. McCallum (1810),
Dodsley's Annual Register (1801) and PRO: T36/10). It is likely that
Cadet
Eric of the Madras Infantry was one of the illegitimate sons.

[rest omitted]

Scallywag is a rather gentle word. And the difference between Anthony
Glyn's version and this one is really quite difference. I was
wondering why James Sutherland never married Lady Mary, and
legitimized their children. This alas explains why.

Some queries though:

1. What happened to Lady Mary? A Maria Bertram like fate (as in
Mansfield Park by Jane Austen) - sent off to live in a cottage with a
carping aunt? Or was she married off or sent abroad?

2. When did General John Scott remarry? [His wife Margaret was third
daughter of Robert Dundas, Lord President of the Court of Session, MP
for Edinburgh (by his 1st marriage). She died 1797]

3. What is his date of birth? I have his death date, and his parents'
name. he was second son of David Scott (1689-1766) of Scotstarvit by
his wife Lucy Gordon, daughter of Sir Robert Gordon of Gordonstoun,
Bt. They married 1716 and he had an older brother David Scott of
Scotstarvit. So presumably he was born circa 1720?

4. What are the dates of birth for Henrietta, Duchess of Portland and
Lucy, Lady Doune? I am assuming 1772, 1774 since Joan was born in
1776.

Thanks,
Shinjinee