It is believed that, in prehistoric times, a broch or dun, known as Dun David, or Dun Dhaibhidh stood here. However, no archaeological evidence has been found for this predecessor.^[2]

The castle was built in the 14th and 15th centuries, when the area was subject to feuds between the rival MacLeod and Macdonald clans. The defences were improved in the 16th century, and by the early 17th century the MacDonalds had finally gained the upper hand in the area. In 1618 the Privy Council and Sir Donald MacDonald of Sleat, "Donald Gorm Og", the 9th chief, signed a charter, requiring him to repair Duntulm.^[2] This was done, and a second tower was added. Around 1650, the castle's importance peaked, when further improvements were made, and a rectangular structure or house was built within the wall. Around 1732 the castle was abandoned, when Sir Alexander MacDonald built a new residence, Monkstadt House, 5 miles (8.0 km) to the south, robbing much of the castle's stone as building material.^[2]

The main structure of the castle is about 25 by 9 metres (82 by 30 ft), and stands on a basalt promontory above the sea. It is further defended by a ditch along the landward side. It comprises the vaults of a tower, which once stood to four storeys, surrounded by an irregular curtain wall. The later house measures around 10 by 5 metres (33 by 16 ft). The only entrance was via a narrow cleft in the sea cliff. The ruins of the castle are now in a very poor condition, with major falls of masonry as recently as 1990.^[2]

According to one local legend, the castle was abandoned after the infant son of the chieftain who dwelt there at the time, in the charge of a nursemaid, fell from a window and was dashed on the rocks below. As a punishment, the nursemaid was set adrift on the North Atlantic in a small boat.^[3]