It has been home to three queens of England, served as a parliamentary garrison during the civil war and was a raucous hotel and bar, before it was ravaged by fire one particularly rowdy night in 1978.

Now the fortified remains of Astley Castle near Nuneaton in Warwickshire have been declared the best building of the year – winner of the 2013 Riba Stirling prize.

Architecture awards are not often given to structures where the roof has caved in and walls have crumbled, but nor are many ruins treated with the care and attention lavished by Witherford Watson Mann architects, whose striking £2.5m intervention has seen the building converted into a beguiling holiday home for the Landmark Trust.

Stitching a series of brick and timber walls into the ravaged shell, the architects have deftly intertwined a forcefully modern approach with a strong sense of the original ruin, crafting a layered sequence of spaces that celebrate the building's 800 years of history. Every view feels carefully choreographed, forming layered vistas through deep, timber-framed windows, past fractured stone mullions, through semi-open courtyards and beyond the overgrown moat to the rolling countryside.

The outdoor dining room is enclosed by broken gothic tracery on one side, partially roofed by a timber canopy and lit by a roaring open fire, conjuring the atmosphere of the building's Saxon origins through a distinctly modern lens.

The project beats the bookies' runaway favourite, a small stone chapel in Cuddesdon, Oxfordshire, by Niall McLaughlin, as well an affordable housing scheme in Harlow and the renovation of the 1960s Park Hill flats in Sheffield.

Second favourite, the Giant's Causeway visitor centre in Northern Ireland, also lost out, along with the University of Limerick's new medical school and student housing. But, unusually, the judges' choice has also been the public favourite: Astley Castle topped both the Guardian and BBC reader polls.

The judges praised Witherford Watson Mann for resolving the project "with beauty, intelligence and a rigour that runs through to the smallest of details" – which even extends to the placement of an individual stone lintel over the toilet roll recess in the downstairs loo.

"It has dealt with Astley's ruins with intelligence and practicality," they added, "while adding to them with a contemporary architecture that is rich, visually beautiful and tactile."

It is the first time the practice has graced the Stirling shortlist. Its previous buildings include the Amnesty International UK headquarters, the Whitechapel Art Gallery extension in London with Robbrecht en Daem, and the Arts Council in Manchester.

Eyebrows may be raised that the award has gone to a luxurious holiday home, which costs up to £2,500 a week for eight people (or £21 a night per person for the cheapest weekends), over the reinvention of an affordable housing type, or the renovation of a concrete council estate. But the architects' approach to Astley Castle points an intelligent way forward for breathing new life into redundant structures of all kinds in a way that is neither overly cautious, nor that shouts too loud.

Critic and historian Joseph Rykwert, recipient of this year's Riba Gold Medal, said of this project, "there is no comparable recovery of an ancient monument anywhere in this country, and very few elsewhere."