The National Trust's Archaeological Survey of Lundy (1989) states that:
The castle comprised a small Keep, measuring 51ft by 38ft, with 3ft thick walls, and with a small bailey on the landward side. This was enclosed within a curtain wall (although this may have been a 17th century addition) and a ditch, except towards the sea where the rock is almost perpendicular. The Keep was rectangular in design, constructed of local granite with the walls inclining inwards. The few windows are very small and these are in the south-facing wall. At some point the Keep's crenellations have been filled-in and the walls built up to the height of the earlier domed chimneys at the four corners.
Facing North Devon, the castle commands a fine view of the east coast of Lundy, the landing bay and the channel. Myrtle Langham writes, “at first it was used by a succession of keepers, with garrison, appointed by the King and until Sir John Borlase Warren built the Farmhouse (Old House) about 1775-80 it was, as far as we know, the main building on the island”.
Thomas Bushell, who held Lundy for the King during the Civil War, claimed that he had “built it from the ground”. At that time he added the East Parade or Bastion with batteries on the east and south sides. The Curtain walls were reinforced or rebuilt of coursed, random rubble. On the parade ground there was a complex of buildings below the keep. One of these, at the western corner, is known as The Old House and may well have been Governor Bushell’s private residence at a time when the Castle was garrisoned.
The Keep was used to house convicts by Thomas Benson, who leased the island from 1748-54, and at that time there were two houses on the parade in front of the Keep. By 1775 the Keep was ruinous and Sir John Borlase Warren deliberately dismantled the castle's defences to provide building stone for works elsewhere on the island but the eastern end of the ditch and rampart survive quite well. In 1824, Sir Aubrey de Vere Hunt's steward wrote that “one half of the oald castle fell first Winter I came to Lundy and the other part came down last Winter”. Other parts were described as still being in good order.
In the 1850s Mr Heaven repaired the keep and converted it into three cottages facing the central courtyard for his labourers. North and South Cottage had two tenements each and East Cottage facing the entrance had one. They all had metal roofs. After the Granite Company failed in 1868, islanders occupied the company’s abandoned buildings, but the Castle cottages remained in intermittent use until the end of the century and sometimes housed shipwreck victims.
In 1870 one cottage was inhabited by the herdsman Withycombe, his wife and their lodger, an old sailor-turned-mason called Sam Jarman. Another was the home of the carpenter Joseph Dark and his family and the four roomed cottage facing the entrance was used by fishermen from Sennen during the summer fishing season, one of whom was George Thomas, the builder of Hanmers. By 1928 the cottages were no longer habitable and in that year Martin Harman commissioned the architect Charles Winmill (Secretary of SPAB 1898, pupil of Leonard Stokes and follower of Philip Webb) to do a report on the Castle. Mr Winmill proposed clearing the shell, strengthening the walls and covering the whole with a flat concrete roof, to provide perhaps a cattle bower. The estimated cost was £1,430 and so the report was ignored.
When Landmark took on the island in 1969 the Castle was once again a ruin. The cottages had lost all but little bits of roof and though the stairs were there, the timbers were rotten. In July 1975, Landmark appointed the stone mason Mike Haycraft to consolidate the external walls. At that time there were two proposals: the castle could either be repaired and left as a maintained ruin or it could be fully restored as hostel accommodation.
Due to erosion large areas of the outside walls needed repacking with mortar. The lime had leached out, the mortar had turned to soil and was supporting vigorous samphire plants. Some of the wall had crumbled away and needed completely rebuilding. Mike used coarse grit sieved from the Landing Beach, Bideford grit and ‘Wallcrete’. Because it was too difficult to handle in the weather conditions, lime was not used. A Victorian chimney was removed but no speculative work was done at all. This work took three summers to complete.
In 1978 an archaeological survey was commissioned before work began on the interior. It was quickly carried out and a musket ball and a little Iron Age pottery were discovered. The team then moved on to the Parade Ground and began working on the floor of the house believed to have been erected by Thomas Bushell in the Civil War.
At this point it was discovered that the internal walls of the Castle were in such bad repair that as they fell in they would drag in the external ones. Moreover, the internal walls were designed to be plastered and if they had been repointed as part of the maintained ruin scheme, they would look wrong. It was therefore decided to plaster them and roof over the old cottages. As, by this time, there was hostel accommodation in the Barn, it was decided to make two comfortable cottages instead.
In 1979 work on the cottages began with sometimes two, sometimes three, islanders helping Mike. The original lintel over the courtyard entrance was found lying on site and identified from old photographs. Mike found the piece of dressed granite, now in the window of the South cottage loo, when working at Admiralty Lookout; the granite fireplace in the North cottage comes from the Quarry cottages. As there is a danger that slates would be whipped off by the Atlantic winds, the cottages are roofed with corrugated iron. Inside, sheet lead was placed between the floors for soundproofing and the cottages were furnished in 1981. In1988, the far end was divided off to create a new, smaller Castle Keep East, reverting to Mr Heaven's 1850s layout.
When David Thackray of the National Trust surveyed the Parade Ground, he discovered a furnace just beneath the east wall of the Castle. It is possible that this was used by Bushell for the King's mint in the Civil War. From 1983 - 1984 the Manpower Services Commission had a team on Lundy rebuilding the exterior walls of the Parade Ground.