He has been called a diva and pretentious, and there are even those who don’t like his acting – but never before has Joaquin Phoenix been accused of causing mental health problems for British livestock farmers.

On Tuesday, the National Farmers’ Union (NFU) president, Minette Batters, changed all that, opening a new front against the US actor by claiming that he and other celebrity campaigners for veganism had played a part in demonising the UK’s meat producers and doing “enormous damage” to their wellbeing.

Days after the Joker actor’s Oscars speech attacking the meat industry, Batters said farmers fearing the imminent loss of their livelihoods and family holdings were in a state of stress and anxiety.

Asked at the union’s annual conference who she thought was driving the view that meat was bad and plants were good, she said: “A lot of people who seem to hit the red carpet at the Bafta awards.”

She added: “Celebrities have to be careful [because] there are real-life consequences for others … Joaquin Phoenix, he’s had a really challenging life, and you really feel for him and a lot of the things he was saying, but he has to remember there are people at the end of this, there are small family farms and they get hurt too.”

Her comments were immediately criticised by vegan and animal rights groups, who accused Batters of making claims without evidence and ignoring the ethical problems posed by meat production.

“Veganism is something of an easy target at the moment and I’m not sure that we are the cause of farmer’s problems,” said a spokesperson for the Vegan Society. “There are many causes of mental health issues and stress in farming and I haven’t seen evidence, a piece of research, showing that veganism is one of them.”

The NFU president, Minette Batters. Photograph: Fototek/PA

Phoenix, who has been a vegan since he was three, made a plea for tolerance and equality in his acceptance speech for the best actor award at the Oscars, saying no race, gender or species had rights over another.

“I think we’ve become very disconnected from the natural world,” he said. “We go into the natural world and we plunder it for its resources. We feel entitled to artificially inseminate a cow and steal her baby, even though her cries of anguish are unmistakeable. Then we take her milk that’s intended for her calf and we put it in our coffee and our cereal.”

Veganism continues to grow in popularity in the UK, with supermarkets clearing shelf space for plant-based ready meals, and meat-free dining in restaurants and pubs now commonplace.

Other celebrities who have spoken about the health benefits of plant-based diets are Benedict Cumberbatch, Ellie Goulding and Beyoncé, with their support credited with aiding a rise in veganism’s popularity.

Batters said she was not saying “veganism is wrong”, but argued that the debate around animal products had become so binary that meat was being put in the same category as tobacco.

“I remember the interview I did with Evan Davis on PM to talk about the government’s new food strategy and he said: ‘Is eating meat the new smoking?’ He compared us with the tobacco industry and you think, ‘Whoa, just think about all of this’.”

But the Vegan Society questioned whether its members were really so influential, pointing out that the total number of vegans in the UK was still only 600,000, and saying: “The fact is 99% of the population are still eating animal products. There might be a lot more meat reducers, but this is not an industry that has been threatened by veganism.”

Dawn Carr, the director of vegan corporate projects at People for the Ethical Treatment of Animals (Peta), said farmers were not the only ones feeling anxious.

“We can’t turn a blind eye to the visible fear and distress shown by animals raised for their flesh, milk and eggs,” she said. “They have no choice, but farmers do: instead of sending sentient animals to slaughter, they can sow oats or soya beans or grow vegetables, grains, nuts or fruits instead, depending on the quality of their land.”

Batters called for kindness to be shown to farmers and an understanding that they were “human” too. “It’s very polarised and it’s doing enormous damage to the mental health of livestock farmers,” she said.

“It’s just about instilling this philosophy and being kind and farmers need – we all need – to think that too … in this world of social media, we just need to take a step back sometimes.”