The castle is located on a peninsula, south west of Wick. It is surrounded by sea cliffs and the landward approach was separated by two moats. Only the square tower remains, originally four storeys tall, however only three storeys remain. The ground floor contained the kitchen and storage areas; the first floor was the hall with a southeast entrance. The upper floors contained the personal quarters of the lord of the castle.

A narrow courtyard led to the centre of the peninsula with buildings on both sides, such as the barracks, the brewery, the chapel, etc. Outside the second moat was a defensive wall, which also formed the back wall of a number of buildings.

The castle was reached from the mainland via a drawbridge, which spanned a wide ditch cut into the rock, protected by a gatehouse and a defensive wall.

The history remains obscure. It was originally thought to have been constructed in the 12th century, by Harald Maddadson, Jarl of Caithness and Orkney. However, the surviving structure seems to date to the 14th century.^[2]

Reginald le Chen of Inverugie and Duffus, is known to have been in possession of the castle in the early 14th century.^[1] It passed by marriage of his daughter Mary to Nicholas Sutherland in 1345.

The castle later passed by the marriage of Christian, the daughter and heiress of Alexander Sutherland of Duffus, to William Oliphant in the 15th century.^[1] Andrew Oliphant of Berrideale sold the property to his uncle, Lord Oliphant in 1526. Laurence Oliphant, Lord Oliphant and his servants were besieged and attacked by the John Sinclair, Master of Caithness in July 1569.^[3]

It came to the Sinclair family in 1644, before passing to John Campbell, Lord Glenorchy, after the death of George Sinclair, Earl of Caithness, without issue. Lord Glenorchy, sold it to the Dunbars of Hempriggs in 1690. The Dunbars owned the castle until 1910.