The site was originally occupied by the king's castle in the Forest of Inglewood. The earliest written record is of the original castle is from 1272. This castle was a medieval square pele tower within a curtain wall. Sir William L'Engleys, descended from the family who held Little Asby, acquired Highhead Castle. He was Chief Forester of Inglewood from 1328, and was succeeded by his son Sir William whose daughter and heir, Isabel, married Nicholas Harrington.

The Richmond family, who bought and extended the house in the 16th century. Only its western wing remains, with its unmistakable straight-headed mullioned windows with round-arched lights under hood moulds, although attached to the south-west corner of this wing is the basement of a square tower which presents evidence of 14th-century work.

The house was later sold to the Baron Brougham and Vaux who made some alterations. One member of the family, Henry Richmond Brougham, had a new facade built in 1744–1748. It is eleven bays long, with a pedimented three-bay centre, and a walled front garden with coupled Ionic columns. When the Broughams no longer required the house, they rented it out for some time. By the mid-19th century, and until at least 1909, it was occupied by the Hills family, and was the childhood home of the soldier and astronomer Edmond Hills. The only other people known to have rented it were the Cavaghan family (co-founders of Cavaghan & Gray), who lived in the house for around eight years, from 1921 to 1929.

The castle was largely destroyed by fire in 1956, and is now little more than a shell. Both the medieval wing and the principal house are Grade II* listed.^[1][2] Adjacent is a fine stable quadrangle with heavy rustication, a steep pediment gable and cupola.

• Grade II* listed buildings in Eden District
• Listed buildings in Skelton, Cumbria

• Article on the Institute of Historic Building Conservation site

54°46′53″N 2°55′48″W / 54.7813°N 2.9300°W / 54.7813; -2.9300