Researchers are investigating a new screening tool for anal cancer, amid reports it is on the rise.

Anal cancer is a common cancer in gay men and doctors from Sydney's St Vincent's Hospital say it is becoming more common.

Like cervical cancer, it is caused by the human papillomavirus.

Professor Andrew Grulich, who is running a new study, says vaccination holds great promise for future prevention, but only for those who are not yet sexually active and therefore not yet infected.

"This means that adult men who have sex with men are likely to receive little benefit from vaccination," he said.

As part of the study, doctors will document the lives of 1,000 gay men every six months for three years.

Dr Grulich says the doctors will look at different strategies for anal cancer screening.

"This study will investigate whether a new screening test predicts the presence of pre-cancerous lesions and their progression to cancer," he said.

It will investigate the psychological harms or benefits from screening, cost-effectiveness and treatment outcomes.

Anal cancer is one of the most common cancers in gay men, but Dr Grulich says there is little research into treatment and screening.

"The findings of the study have the potential to be translated into improved cancer health services in the life of the study," he said.