The son of The 3rd Baron Rossmore (1792–1860) by his second marriage to Josephine Julia Helen Lloyd, the daughter of Henry Lloyd of County Tipperary and of Harriet Amelia Carden, Rossmore was born in 1851. He had two sisters and three brothers: Frances Kathleen (died 1925), Norah Josephine Harcourt (died 1934), Derrick Westenra, 5th Baron Rossmore (1853–1921), Richard Hamilton (1854–1880), and Peter Craven (1855–1932).^[1]

On 1 December 1860, when he was only nine, his father died and he succeeded to the titles of The 4th Baron Rossmore, of Monaghan in the County of Monaghan, in the Peerage of Ireland and The 3rd Baron Rossmore, of the County of Monaghan, in the Peerage of the United Kingdom, the second giving him a seat in the House of Lords as soon as he reached the age of twenty-one.^[1] He also inherited large estates in County Monaghan, and from his father's first wife, Anne Douglas-Hamilton (c. 1796–1844), a natural daughter of The Duke of Hamilton and Brandon (1756–1799), a further estate in the Aran Islands.

Lord Rossmore joined the 1st Life Guards, a cavalry regiment of the British Army. He died in a riding accident on 28 March 1874 and was buried in the family vault at Rossmore Park. His younger brother Derrick thus succeeded unexpectedly to the titles and estates. The house was abandoned in the 1940s, fell into ruin, and was demolished in 1975. ^{[citation needed]}