Of this 16th-century building there remain only the outer walls, including the entrance doorway, the great fireplace on the first floor and a small fireplace on the second floor (where there is also a garderobe closet), together with a short stretch of the original barmkin wall to the south of the tower, and some carved stone panels. In 1556 Saddell had been transferred to James Macdonald, who was busy annoying the English army in Ireland. In retaliation the Earl of Sussex mounted a raid on Kintyre in 1558, during which he burned and sacked the Castle, which he described as "a fayre pyle and a stronge".
The Castle seems to have been left as a ruin for the next hundred years, even after it was granted to Archibald Campbell, Earl of Argyll, in 1607. Then in 1650 the Earl, in turn, granted Saddell to William Ralston of that Ilk, a fugitive from religious persecution in the Lowlands, on condition that he made it habitable within two years. The Castle was given a new roof, and floors, and the walls and parapet were extensively repaired. The arrangement of the rooms is mostly of that same date.
William Ralston soon moved elsewhere, and by the end of the 17th century the estate had been granted to a junior branch of the Campbell family, who became known as the Campbells of Glensaddell. During the 18th century they tried to make the Castle more comfortable, by lining the bedroom walls with panelling for example; and they smartened up the sitting room with a new fireplace, alcoves and a moulded plaster ceiling.
They must have felt they were fighting a losing battle, however, because in about 1774 the Campbells built themselves a new and more convenient home, which they called Saddell House. The castle became a farm, and was lived in by estate employees. Stone from the Abbey was used for the farm buildings that cluster around the foot of the tower.
In 1890 the Castle once again became, for a few years, the chief residence of the estate, after Saddell House was damaged by fire. At that time it belonged to Colonel Macleod, who clearly had great fun restoring the castle. It was he who put up the heraldic shields in the dining room, which contain heraldic jokes and puns; and he made several other minor alterations, such as the ceiling in the top bathroom, and fireplaces in several of the bedrooms.
Once Saddell House was repaired the castle went back to being an estate farmhouse. In the 1930s it was given another new roof, but after the War there were no longer the funds to repair it. In 1939 the Saddell estate had been bought by Lt Col and Mrs Moreton, and it was they who in 1975 sold the Castle, with Shore Cottage (built in the 19th century) and Cul na Shee (built in the 1920s), to the Landmark Trust. In 1984 the Trust bought the remainder of the estate, and in 1990 Ferryman’s Cottage.