The family descends from Sir George Ramsay, who represented Kincardineshire in the Scottish Parliament in 1617. He received a charter of the barony of Dalhousie and also of the barony of Melrose on the resignation of John Ramsay, 1st Earl of Holderness. In 1618 he was raised to the Peerage of Scotland as Lord Ramsay of Melrose. However, as he did not like the title, he obtained a letter from James VI in 1619 to change it to Lord Ramsay of Dalhousie (with the precedence of 1618). He was succeeded by his eldest son, the second Lord. He sat as a member of the Scottish Parliament for Montrose in 1617 and 1621 and served as Sheriff Principal of Edinburghshire. In 1633 he was created Lord Ramsay of Keringtoun and Earl of Dalhousie, in the County of Midlothian, in the Peerage of Scotland. His grandson, the third Earl (who succeeded his father in 1674), fought at the Battle of Bothwell Brig in 1679. His eldest son, the fourth Earl, was killed in a duel with a Mr Hamilton. He was unmarried and was succeeded by his younger brother, the fifth Earl. He was a colonel in the Scots Guards and brigadier-general in the British Army and fought in the War of the Spanish Succession.

He was succeeded by his first cousin once removed, the sixth Earl. He was the son of Captain the Hon. John Ramsay, second son of the first Earl. Lord Dalhousie's eldest son of George Ramsay, Lord Ramsay (died 1739), married Jean, daughter of the Hon. Harry Maule of Kelly, younger son of George Maule, 2nd Earl of Panmure, brother of James Maule, 4th Earl of Panmure, who took part in the Jacobite Rebellion of 1715 and was attainted in 1716 with his titles forfeited (see Earl of Panmure). Dalhousie was succeeded by his grandson, the seventh Earl. He was the eldest son of Lord Ramsay. He died unmarried and was succeeded by his younger brother, the eighth Earl. He sat in the House of Lords as a Scottish representative peer from 1774 to 1787. His second son the Hon. William Ramsay succeeded to the Maule estates in 1784 and was created Baron Panmure in 1831. Lord Dalhousie was succeeded by his eldest son, the ninth Earl. He was a distinguished soldier and served as Governor General of British North America (and previously Governor of Nova Scotia where he founded Dalhousie University) and as Commander-in-Chief of India. In 1815 he was created Baron Dalhousie, of Dalhousie Castle in the County of Edinburgh, in the Peerage of the United Kingdom, which entitled him to an automatic seat in the House of Lords.

He was succeeded by his eldest surviving son, the tenth Earl. He was an influential Tory politician and also served as Governor-General of India from 1847 to 1856. In 1849 he was created Marquess of Dalhousie, of Dalhousie Castle in the County of Edinburgh and of the Punjab, in the Peerage of the United Kingdom. Lord Dalhousie assumed the additional surname of Broun of Colstoun upon succeeding to the Colstoun estates. He had no male issue and on his death in 1860 the marquessate and barony of 1815 became extinct. He was succeeded in the Scottish titles by his first cousin Fox Maule, 2nd Baron Panmure, the eleventh Earl (see Baron Panmure for earlier history of this branch of the family). He was a prominent Liberal politician and notably held office as Secretary of State for War. On succeeding to the earldom in 1860 he assumed the surname and arms of Ramsay of Dalhousie after that of Maule. Lord Dalhousie was childless and on his death in 1874 the barony of Panmure became extinct. He was succeeded in the Scottish titles by his first cousin, the twelfth Earl, who was the second son of the Hon. John Ramsay, fourth son of the eighth Earl. Lord Dalhousie was an admiral in the Royal Navy. In 1875, he was created Baron Ramsay, of Glenmark in the County of Forfar, in the Peerage of the United Kingdom, which entitled him to a seat in the House of Lords.

He was succeeded by his eldest son, the thirteenth Earl. He was a Liberal politician and served under William Ewart Gladstone as Secretary of State for Scotland in 1886. His eldest son, the fourteenth Earl, was a captain in the Scots Guards and an honorary colonel in the North Scottish Royal Garrison Artillery (TA). He was succeeded by his eldest son, the fifteenth Earl. He was a Deputy Lieutenant of Angus. He died unmarried and was succeeded by his younger brother, the sixteenth Earl. He was a Conservative Member of Parliament before succeeding in the earldom and later served as Governor-General of the Federation of Rhodesia and Nyasaland and was Chancellor of the University of Dundee. As of 2013^[update] the titles are held by his eldest son, the seventeenth Earl, who succeeded in 1999. Lord Dalhousie is Chief of Clan Ramsay.

Several other members of the family have also gained distinction. The Hon. George Ramsay (died 1705), younger son of the second Earl, was a lieutenant-general in the Army and served as commander-in-chief of the forces in Scotland in 1702. The aforementioned the Hon. William Ramsay, second son of the eighth Earl, was created Baron Panmure in 1831. The aforementioned the Hon. John Ramsay (1775–1842), fourth son of the eighth Earl, was a lieutenant-general in the service of the General Staff of India. He was the father of 1) William Ramsay (1804–1871), a major-general in the Bengal Army; 2) the twelfth Earl (see above); 3) James Ramsay (1808–1868), a major-general in the Bengal Army; and 4) the Hon. Sir Henry Ramsay (1816–1893), a general in the Bengal Army, whose grandson was the politician Archibald Maule Ramsay. The Hon. Charles Ramsay, fourth son of the twelfth Earl, represented Forfar in the House of Commons from 1894 to 1895. The Hon. Sir Patrick Ramsay (1879–1962), second son of the thirteenth Earl, was a diplomat and served as Envoy Extraordinary and Minister Plenipotentiary to Greece, Hungary and Denmark. The Hon. Sir Alexander Ramsay, third son of the thirteenth Earl, was an admiral in the Royal Navy and served as Fifth Sea Lord from 1938 to 1939. He was the husband of Princess Patricia of Connaught, youngest child of Prince Arthur, Duke of Connaught. Their son was Alexander Ramsay of Mar, husband of Flora Fraser, 21st Lady Saltoun.

The family seat is Brechin Castle near Brechin, Angus, which is currently put up for sale through the Savills real estate company. The other former seat, Dalhousie Castle, was sold in the late 20th century and is now a hotel. Until then it was said to be a property longer in one family than any other in Scotland.

The title of the earldom is pronounced "Dal-how-sy".

• George Ramsay, 1st Lord Ramsay of Dalhousie (died before 1629)
• William Ramsay, 2nd Lord Ramsay of Dalhousie (died 1672) (created Earl of Dalhousie in 1634)

• William Ramsay, 1st Earl of Dalhousie (died 1672)
• George Ramsay, 2nd Earl of Dalhousie (died 1674)
• William Ramsay, 3rd Earl of Dalhousie (died 1682)
• George Ramsay, 4th Earl of Dalhousie (died 1696)
• William Ramsay, 5th Earl of Dalhousie (died 1710)
• William Ramsay, 6th Earl of Dalhousie (c. 1660 – 1739)
• Charles Ramsay, 7th Earl of Dalhousie (died 1764)
• George Ramsay, 8th Earl of Dalhousie (died 1787)
• George Ramsay, 9th Earl of Dalhousie (1770–1838)
• James Andrew Broun-Ramsay, 10th Earl of Dalhousie (1812–1860) (created Marquess of Dalhousie in 1849)

• James Andrew Broun-Ramsay, 1st Marquess of Dalhousie (1812–1860)

• Fox Maule-Ramsay, 11th Earl of Dalhousie (1801–1874)
• George Ramsay, 12th Earl of Dalhousie (1806–1880)
• John William Ramsay, 13th Earl of Dalhousie (1847–1887)
• Arthur George Maule Ramsay, 14th Earl of Dalhousie (1878–1928)
• John Gilbert Ramsay, 15th Earl of Dalhousie (1904–1950)
• Simon Ramsay, 16th Earl of Dalhousie (1914–1999)
• James Hubert Ramsay, 17th Earl of Dalhousie (b. 1948)

The heir apparent is the present holder's son, Simon David Ramsay, Lord Ramsay (b. 1981).
The heir apparent's heir apparent is his son, the Hon. William Fox Ramsay (b. 2017).