The McSwynes probably came to Ireland in the 13th century,^[1] according to other sources only in the 14th century^[2] from Scottish Argyll to Ireland. They were foot soldiers and standard bearers of the O'Donnells. In later years, the McSwynes (McSuibhne (Irish)) fought alongside the O'Neills, suggesting they were mercenaries.^[1]

In the mid-15th century the McSwynes had their castle built; the land was theirs up to the Plantation of Ulster. With the conquest by the Scots, land and castle fell to the conquerors; King James VI and I bequeathed it to William Stuart and later to John Murray, who was made Earl of Annandale in 1625. When John Murray died in 1640, his son James inherited the castle and title. James died childless and so the castle passed to his cousin Robert Creighton who took the name 'Murray'.^[1]

After the Rebellion of 1641, in the reign of King Charles II, a dispute arose over the ownership of the castle and lands between the Creightons and Robert Murray of Scotland, another second cousin. After years of legal battles in England, Scotland and Ireland, the Creightons lost the estate. The castle was no longer inhabited and fell into disrepair.^[1] In 1844, stone from the ruins was used to build the parish church at Killybegs, dedicated to the Annunciation.^[2]

However, even the Scottish line of the Murray family could not definitively prove their ownership rights and so they could not sell the property until the 19th century. The tenants were more inclined towards the Irish Creightons than the Scottish Murrays and ended up paying their rent to David Creighton. However, the Murrays remained official owners of the estate until the Irish Land Commission took it over in the 20th century.^[1]