Duff was the eldest illegitimate son of James Duff, 2nd Earl Fife (1729–1809), and Margaret Adam, of Keith. As his mother was of humble status, her three children with Fife were placed in the guardianship of William Rose, Fife's factor. Fife liked Duff and took good care of him, sending him to Keith Academy and at King's College, Aberdeen, where he graduated MA in 1771, and after he entered the army paid for Duff's promotions.^[2]

Duff entered the army as an ensign in the 1st Foot Guards on 18 April 1769. He was promoted lieutenant and captain on 26 April 1775, and made adjutant of his battalion in 1777, and on 30 April 1779 he was knighted as proxy for the celebrated diplomatist Sir James Harris, afterwards 1st Earl of Malmesbury (and a close friend of his father), at his installation as a knight of the Bath. James Duff was promoted captain and lieutenant-colonel, on 18 July 1780, colonel on 18 November 1790. His regiment was part of the expeditionary force sent to Flanders to campaign against the French as part of the First Coalition (the British contingent under the command of Prince Frederick, Duke of York). He fought at the Battle of Valenciennes commanding the guards' light infantry battalion in 1794. On 30 October he was promoted to major-general.^[1][2]

In 1797, Duff received the command of the Limerick district. While there he rendered important services during the Irish Rebellion of 1798, and managed to keep his district quiet in spite of the state of affairs elsewhere. His name is associated with the Gibbet Rath executions where men under his command massacred several hundred people, who had previously received amnesty. He was elected M.P. for Banffshire (1784–9), and colonel of the 50th Regiment of Foot from 1798 to death.^[1]

Duff was promoted lieutenant-general on 1 January 1801, and general on 25 October 1809, and at the time of his death, at Funtington, near Chichester, on 5 December 1839, he was the most senior general in the British army, and was one of the few officers who held a commission for over seventy years.^[1]

He had as aides-de-camp during his Limerick command two famous officers, William Napier and James Dawes Douglas. There are numerous allusions to him in the Life of Sir William Napier.^[1]

According to the Legacies of British Slave-Ownership at the University College London, Duff was awarded a compensation payment as a slave owner in the aftermath of the Slavery Abolition Act 1833 and the Slave Compensation Act 1837. The British Government took out a £15 million loan (worth £1.51 billion in 2024^[3]) with interest from Nathan Mayer Rothschild and Moses Montefiore which was subsequently paid off by the British taxpayers (ending in 2015). Duff was associated with "Jamaica Hanover 455 (Grange Sugar Estate)", he owned 202 slaves in Jamaica and received a £4,101 payment at the time (worth £414,058 in 2024^[3]).^[4] This is recorded by the Slave Compensation Commission and the records held at the National Archives at Kew, London.^[5][6]

On 12 August 1785, Duff married Basilia (d. 1849), daughter and heir of James Dawes, of Rockspring, Jamaica, through whom he gained control of a considerable fortune. They had one son and three daughters.^[2]

• Stephens, H. M. (May 2008) [2004]. "Duff, Sir James (1753–1839)". In Wood, Stephen (ed.). Oxford Dictionary of National Biography (online ed.). Oxford University Press. doi:10.1093/ref:odnb/8169.
•   This article incorporates text from a publication now in the public domain: Stephens, Henry Morse (1888). "Duff, James (1752-1839)". In Stephen, Leslie (ed.). Dictionary of National Biography. Vol. 16. London: Smith, Elder & Co. p. 129. ; Endnotes:
• Royal Military Calendar
• Gentlemen's Magazine March 1840
• Life of Sir William Napier.