A poll of almost 38,000 UK students suggests rates of psychological distress and illness are on the rise in universities, with “alarmingly high” levels of anxiety, loneliness, substance misuse and thoughts of self-harm.

Researchers say the report, seen exclusively by the Guardian, is the largest mental health survey ever conducted among UK university students, and its findings constitute “an urgent call to action”.

The findings were based on responses from a self-selecting sample of students who took part in an online survey. Among the key findings likely to prompt concern among parents and those responsible for student welfare were those on self-harm. Half of the students (50.3%) who took part reported thoughts of self-harm – almost twice as high as reported rates in 2017 – while just under one in 10 (9.4%) thought of self-harm often or always.

More than four out of 10 (44.7%) admitted using alcohol or drugs to cope with their problems, while one in 10 (9.5%) said they did this often or always. One in three (33.9%) had experienced a serious psychological issue for which they felt they needed professional help – an increase of just under 1% in a year.

The survey also flagged up the vulnerability of students in their second and third years who are said to be at “significantly higher risk” than freshers.

While the findings echo previous research, the scale of the study, which polled 37,500 students at 140 universities across England, Wales and Scotland, make it hard for policymakers to ignore.

Students reported high levels of anxiety, with 42.8% often or always worried. Almost nine in 10 (87.7%) said they struggled with feelings of anxiety – an increase of 18.7 percentage points on 2017 figures – and a third (33%) reported suffering from loneliness often or all the time.

More than one in five (21.5%) said they had a current mental health diagnosis, most commonly depression (10.2%) and anxiety disorders (8.4%). The stigma surrounding mental illness persists however, with more than three-quarters (75.6%) concealing their symptoms from friends.

One of the most striking findings is that while student mental health concerns and efforts previously focused on first-years who have moved away from home for the first time, students later in their university careers are struggling too.

According to the survey findings, students moving into their second and third year report the highest rates of anxiety, loneliness and substance misuse. Persistent thoughts of self-harm are highest among second-years – 12.1% of those in the sample had thoughts of self-harm often or all the time, compared with 9.2% of first-years and 11.1% of third-years.

“Perhaps the fact that support initiatives trail off after the first year, or that academic pressure intensifies, or a combination of these and other factors, are the basis for second- and third-years finding life more difficult,” the report states.

The research was conducted by the Insight Network, a team of therapists and psychiatrists, in collaboration with Dig-In which provides a welcome box to 400,000 students at more than 140 universities.

Dr Stephen Pereira of the Insight Network said: “Counter to popular belief, it isn’t the initial transition from school to university that seems to be most associated with psychological difficulties. The research has shown that students who are navigating the transition into their second and third year report the highest rates of anxiety, loneliness, substance misuse, and thoughts of self-harm.

“More research is needed to understand the specific risk and protection factors at play during different years of university, in order for support services to effectively meet these year-specific demands.”

Many universities have already increased investment in counselling and support services for students. John de Pury, mental health policy lead at Universities UK which represents the UK higher education sector, said: “Universities cannot address these challenges alone and we are working closely with the NHS locally and nationally to make sure that students get the care they need.”

The universities minister, Chris Skidmore, added: “Student mental health is a top priority for me and from my regular visits to our universities across the country, I know this is an issue that they are taking seriously too.

“We know that university-wide approaches are vital in tackling this important issue, which is why the government is backing the university mental health charter led by Student Minds, which will encourage universities to improve the pastoral care they provide and will set a high standard for mental health support.”