The castle was originally associated with Clan Fraser, and was perhaps named for Oliver Fraser, who gave lands to Newbattle Abbey as recorded in its register. The lines of descent from Oliver and his nephew Adam are uncertain, but the Frasers continued to exert power from Oliver Castle with Sir Bernard Fraser and Sir Gilbert Fraser, who held in turn the hereditary office of Sheriff of Tweeddale. A descendant, Sir Simon Fraser of Oliver and Neidpath, Knight Banneret, fought in the Wars of Scottish Independence.^[1] After switching sides more than once he was eventually captured by the forces of King Edward I of England and sent to London where he was hanged, drawn and quartered in September 1306.

Oliver Castle was inherited by his daughter Joan who married Sir Patrick Fleming of Biggar.^[2] The estate remained with his successors until 1524 when John 2nd Lord Fleming was murdered by Thomas Tweedie. His son, Malcolm 3rd Lord Fleming, pledged Oliver Castle to Robert Dickson for one hundred pounds and eventually parted with the estate.^[3]

Oliver passed to the Tweedies soon afterward. The legendary lawlessness of the Scottish Marches revolved around bitter inter-family feuds and conflict over livestock. In an incident characteristic of the times, Thomas Porteus of nearby Hawkshaw was arraigned on 16 February 1489 for having lifted seventy-four lambs from the lands of Oliver Castle, belonging to William Tweedie and Lawrence Tweedie.^[4]

In the 17th or 18th century a new house was built by the Tweedies, which was itself replaced by the present Oliver House, which was begun in about 1780 by Thomas Tweedie of Oliver. Both houses are likely to have reused stone from the medieval tower house.^[5]

The site of the prehistoric hillfort is protected as a scheduled monument.^[6] It is on a low knoll, some 60 metres (200 ft) above the valley floor, and covered an area around 60 by 55 metres (197 by 180 ft). The two lines of defence are visible as little more than grassy banks. Inside, possible timber house sites have been noted as well as later stone foundations. The association of these remains with the medieval castle has not been confirmed by excavation, though tradition places the castle on the hillfort site.^[7]

The present Oliver House is located on lower ground, around 200 metres (660 ft) to the south-west of the hillfort, and remains occupied. It incorporates a heraldic panel that was brought from the earlier house, which stands some 50 metres (160 ft) to the north-west. The old house is a typical laird's residence, although much altered. It bears the date 1734, with the initials of James Tweedie of Oliver and his wife Margeret Ewart.^[8] An older date of 1649 is also recorded as being inscribed on the old house.^[9]

• List of places in the Scottish Borders
• List of places in Scotland

55°30′33″N 3°25′48″W / 55.50917°N 3.43000°W / 55.50917; -3.43000