They may have been commissioned in the 1680s by Sir John Rolle, who inherited in 1647 and was knighted at the Restoration, or perhaps by his grandson and heir Robert soon after he inherited in 1706. Both were architectural patrons - Robert was responsible for the colonnaded Queen Anne’s Walk in Barnstaple - and whichever undertook the task probably employed some gifted local mason/architect, who may well have been familiar with the work of Talman and perhaps Wren. Architectural details seem to indicate the earlier date, but it would still have been possible to find craftsmen to do work in the 'old-fashioned' manner 20 or 30 years later. Curiously, however, the armorial shields on the façade of the Library are not those of Sir John, nor of Robert nor either of their wives, but those of Robert’s brother John and his wife Isabella. It may have been this John Rolle who finished work on the Library, decorated the interior and at the same time built the Orangery.
So had the Library always been used as a library? Probably not. To begin with, the existence of a library as a separate room was not common before the age of the Grand Tour and Palladianism, say from 1720 onwards. But during the restoration it appeared that there had never been a cornice in the upper room, indicating that the walls had always been lined with bookcases, or at least had been so from the time that the interior plaster-work was done. Certainly by 1796 the Library contents appeared in property lists separately from those of the main house. By 1976 the bookcases had all disappeared except for a few fragments of inlaid veneer and these looked like late 18th-century work. Denys Rolle, who inherited in 1779 as the third and youngest son of Robert Rolle’s brother John, may have been responsible for fitting the building out as a library. He was an eccentric man, a naturalist who talked to the woodland animals and was a widely-read educationalist. He twice tried to establish a Utopian colony of Devon poor and homeless on 20,000 acres of land in East Florida and liked to work alongside them. When his settlers all deserted or returned home for the second time, Rolles turned instead to enslaved Africans for labour on his plantations, building up a large workforce. After Florida was ceded to Spain in 1783, Rolles was given plantations in Exuma, Bahamas in compensation but returned himself to Devon, perhaps finding solace in his library. The Rolles slaves were officially liberated in 1838 a generation later, the compensation from the British government adding to the Rolle family wealth. Many of the former slaves adopted the surname of Rolle. They took over the Rolle lands on Exuma as their own, running them communally. The lands are owned to this day by descendants of the former slaves, and cannot be sold.
Denys’s son John died childless and the estate passed to his nephew Mark, who took the name of Rolle and who rebuilt Stevenstone in 1868-70. The Library was partly rebuilt at the same time: the roof was renewed, the front was rendered and steps were built at the back to give access to the upper room. The Orangery was reroofed in glass and became a fernery and a new garden was laid out around them.
But in 1907 Mark Rolle also died without sons. In the years that followed the land was sold and most of the great house pulled down, with the remains being occupied by troops during the War. After the War the house was broken up and the lead from the roof sold for scrap, the stables were turned into cottages and more cottages were built on the land. In the late 1940s the Library itself was converted into a house, dividing the upper room and closing in the loggia. The fireplace from the dining room in the main house was put in a ground-floor sitting room. In 1978, with the Orangery, it was put up for sale and the two buildings were bought by the Landmark Trust, as a charity that specialises in the restoration of buildings of architectural or historic importance.