Residents of one of England’s most remote and picturesque valleys have expressed relief after a landowner was ordered to remove a gleaming structure that they believed was going to be turned into a luxury country retreat.

The landowner claimed the building, which appeared close to the summit of the Black Hill in Herefordshire, was a sheep shed but many locals feared it would be converted into a splendid hideaway home.

Dozens of letters of protest were written by local people, one claiming it looked like something that had landed from space, and a protest march was held.

Planners have now told the landowner – the artist Christopher Brooks, the brother of the racehorse trainer Charlie and brother-in-law of the former tabloid editor Rebekah Brooks – to remove the building.

Ruth Watkins, one of the leading critics, said: “I think it’s safe to say the whole community is delighted. It is a triumph for the people and very plucky of Herefordshire council, a small and impoverished unitary authority, to demand its removal.”

Another resident, who asked not to be named, said: “I think the council have had the fortitude to take the absolutely correct action. If they let this one by – in arguably the most spectacular landscape in the county – people would quite rightly be asking a lot of questions.”

Brooks lives with his wife, the American-born fashion director and writer Amanda Cutter Brooks, 70 miles away on an idyllic farm in the Cotswolds.

The land on slopes about half a mile from the summit of Black Hill, which was immortalised by the travel writer Bruce Chatwin in his novel On The Black Hill, was bought at auction four years ago and permission was granted for a lambing shed to be built.

However, people in the parish of Craswall were alarmed when the structure was completed, pointing out that it was larger than had been specified in the plans and visible from the valley floor, especially when sunlight glints off the large windows.

Last year Brooks applied for retrospective planning permission to cover the increased size and asked for the go-ahead to add a wood-burning stove. A letter accompanying the application says the stove is needed to revive lambs and keep a shepherd warm.

One objector described it as “a grand design that Kevin McCloud would be simpering over”, and claimed: “The spaceship abode would make a beautiful and very valuable piece of real estate. As it stands it is the single most glamorous sheep barn that has ever existed.”

The council turned down retrospective planning permission. It said: “The proposal is incongruous in character with harmful visual effects.”

It has served an enforcement notice requiring the owner to remove the building. The owner has until 13 May to lodge an appeal, A council spokesperson said last week it was not aware of any appeal having been made.

Brooks has not responded to a request for comment.