People who live in one of England’s most beautiful valleys are taking to the hills to protest against a new structure that its owners say is a barn but which critics fear is going to be turned into a country retreat.

The structure has been built about half a mile from the summit of the Black Hill in Herefordshire, on slopes owned by the artist Christopher Brooks, brother of the racehorse trainer Charlie Brooks and brother-in-law of the former tabloid editor Rebekah Brooks.

Christopher Brooks lives with his wife, Amanda Cutter Brooks, an American fashion director and writer, 70 miles away on a farm in the Cotswolds, and his representatives say the building in the parish of Craswall is simply a lambing and storage shed.

Dozens of people have written to Herefordshire council saying the metal structure is an eyesore, visible from the valley floor below and the hills above, and have expressed suspicions that it will be turned into a home. Critics have said it looks like the sort of super-modern structure that Kevin McCloud might feature in the Channel 4 show Grand Designs.

On Sunday, local people plan to hike up a rough, muddy track to protest about the building.

The structure viewed from beyond the boundary of Christopher Brooks’ land, near Black Hill in Herefordshire. Photograph: Adrian Sherratt/The Guardian

Ruth Watkins, one of the organisers of the walk – which will end up with cakes and tea in the village hall – believes the structure looks out of place in the landscape, a place of solid farmsteads tucked away up winding lanes and ruined monasteries.

“There’s a sense of anger,” she said. “We strive so hard to stick to the planning rules because the place and the architecture is so splendid and highly prized. We worry that this building could set a dangerous precedent. The protest will be peaceful but passionate.”

The land, Probert’s Farm, was bought at auction four years ago and Brooks applied in 2015 for planning permission for a lambing shed.

A planning officer concluded that it was “relatively modest” and would not be “unduly stark” in the landscape or visible from public vantage points. Permission was granted.

However, local people were alarmed when the structure was completed, pointing out that it was certainly visible, especially when sunlight glints off the large windows. They were also upset that it was larger than specified in the plans. There are currently no sheep on the land, only longhorn cows owned by a neighbour, and a few horses.

In September, Brooks applied for retrospective planning permission to cover the increased size, and he has asked for the go-ahead to add a wood-burning stove. A letter accompanying the application says the stove is needed to revive lambs and keep a shepherd warm.

The letter from a chartered surveyor in Hereford acting for Brooks says: “Our client farms 600 acres on the Cotswolds and acquired the land known as Probert’s Farm … to assist the whole agricultural endeavour to achieve organic status.

“The land was seen to be an ideal opportunity to acquire uncontaminated pasture for flock establishment as part of this operation. The barn’s primary use is for lambing, secure storage of tools and equipment in this location and an insulated and heated section for shepherd and lambs.”

The pub in the village of Craswall, where the structure is located. Photograph: Adrian Sherratt/The Guardian

Comments were invited by the council and they have flooded in not only from local people but from walkers and riders who love the spot. The area is also a favourite with fans of literature. The area was immortalised by the travel writer Bruce Chatwin in his novel On the Black Hill; Hay-on-Wye, famed for its book shops and literary festival, is only seven miles away.

In his letter of objection, Wyndham Powell, a farmer, wrote: “It does not resemble any lambing shed I have ever seen or been in and certainly not any other agricultural building in the area.”

Susan Quigley, another objector, said sheep were usually lambed at much lower levels in the area and added that infra-red lamps rather than wood-burning stoves were used to revive lambs.

Another objector described it as “a grand design that Kevin McCloud would be simpering over”, and claimed: “The spaceship abode would make a beautiful and very valuable piece of real estate. As it stands it is the single most glamorous sheep barn that has ever existed.”

Nicholas Keeble, a Herefordshire historic building and planning consultant, wrote: “Because of the decision to place the unit at such a high altitude, one is again forced to conclude that the site was chosen in order to take advantage of far-reaching views.”

Herefordshire Campaign to Protect Rural England claimed that the building gleamed “brightly and aggressively” in sunlight and had a “completely alien” appearance.

The target date for determination of the application is 20 December. Asked by the Guardian about the villagers’ concerns, Brooks said: “The barn as built is approved under planning legislation and the current application is being made through the proper channels.”