The son of George FitzRoy, 2nd Baron Southampton, he succeeded his father in 1810. He was the grandson of Charles FitzRoy, 1st Baron Southampton and Anne Warren, the daughter and co-heir of Adml. Sir Peter Warren and a descendant of the Schuyler family, the Van Cortlandt family, and the Delancey family, all from British North America.

On 23 February 1826, he married Harriet Stanhope, granddaughter of William Stanhope, 2nd Earl of Harrington; they had no children. She died in 1860.

Southampton took his seat in the House of Lords on 10 June 1828.^[1] He was originally a Whig in politics but, having voted against the Reform Act 1832, he became a Conservative.^[2] He was Lord Lieutenant of Northamptonshire, from 1867 until his death. His seat was Whittlebury Lodge, Towcester, Northamptonshire.

In 1842, after being (ninth) Master of the Quorn Hunt from 1827 to 1831, he took over the Mastership of the Grafton Hunt from George FitzRoy, 4th Duke of Grafton, his father's cousin, and remained its Master until 1862.

On his retirement from the Mastership of the Quorn Hunt he sold that pack to Sir Harry Goodricke, 7th Bt. For the Grafton he bought hounds from the brewery magnate Harvey Combe, who had purchased them from the renowned sportsman George Osbaldeston; in 1847 more hounds were bought from the impoverished 4th Earl of Shannon. Lord Southampton hunted the Grafton pack entirely at his own cost, without subscription from any members of the Hunt.

William Webb depicted him as Master of the Grafton Hunt about 1845.

Whittlebury was central to the Grafton Hunt. It was on the FitzRoy estate of the Duke of Grafton, one of whose seats was Wakefield Lodge, nearby.

On 25 February 1862 he married Ismania Catherine Nugent, by whom he had five children:

• Ismay Mary Helen Augusta FitzRoy (13 April 1863 – 22 April 1952), married Rev. Lord Charles Edward FitzRoy; mother of Charles FitzRoy, 10th Duke of Grafton
• Frederica Louise FitzRoy (1864 – 9 April 1932), married to Percy Crutchley and mother of Victor Crutchley
• Blanche Georgiana FitzRoy (1865 – 28 November 1944), married Edward Douglas-Pennant, 3rd Baron Penrhyn
• Charles Henry FitzRoy, 4th Baron Southampton (1867–1958)
• Capt. Edward Algernon FitzRoy (1869–1943), who was Speaker of the House of Commons from 1928 until his death, and from whom Viscount Daventry descends.

It is an extraordinary fact that, although the 3rd Lord Southampton was born in 1804, his elder son and heir did not die until 1958.

• Burke's Peerage, Baronetage & Knightage,107th edition, 2003, vol. 3

• Burke's Peerage & Gentry, at www.burkespeerage.com

• Debrett's Peerage & Baronetage, 1995

• Cracroft's Peerage, at www.cracroftspeerage.co.uk

• Leigh Rayment's Peerage Page, at www.leighrayment.com

• www.thepeerage.com

• J.M.K. Elliott, Fifty Years’ Fox-Hunting with the Grafton and Other Packs of Hounds, 1900.

• Bernard Falk, The Royal FitzRoys. Dukes of Grafton Through Four Centuries, 1950.