A man has been rescued after falling into a sinkhole in a cave in the middle of the night and becoming stuck for more than eight hours.

The 23-year-old man was caving with outdoor adventure group Outward Bound near Buchan, in Victoria's east, when he fell 15 metres down the sinkhole about midnight.

He spent the morning stuck on a ledge at the Slocombes Cave Reserve before being freed about 8:30am.

CFA captain Geoff Bassett said the man had taken a small group away from the main tour, near the Jackson's Crossing Track, just after 10:00pm.

A rescuer descends into the cave to reach a the 23-year-old man. ( Supplied: Kim Woodward, CFA volunteer )

Outward Bound said the group had left "against instructions and without the knowledge of Outward Bound Australia staff".

The man was about 20 metres into the cave opening when he dropped his torch and, when going to retrieve it, lost his footing and fell.

"They're experienced guys," Mr Bassett said.

"It's just one of those things that could have happened to anyone."

'He managed to crack a smile'

The cave was in a mobile phone black spot, but fortunately the explorers were able to raise the alarm after midnight using a satellite phone.

"He was conscious and breathing but in a fair amount of pain, back pain," Mr Bassett said.

The man was finally removed from the cave and taken to hospital some eight hours later, after a rescue operation that was made complicated by the narrowness of the shaft.

"He managed to crack a bit of a smile when he came out," Mr Bassett said.

"He was certainly relieved to be out there."

In a statement, Outward Bound said an ambulance and the local CFA crew had been on the scene since about 2:00am.

"Assisted by Outward Bound Australia staff, a paramedic descended to the man for assessment and treatment," the statement said.

"The paramedic stayed with the man until Melbourne-based rescue services arrived at the scene."

The company said the man had spoken to his family since being rescued, and was flown to the Royal Melbourne Hospital in a serious but stable condition.

"The other three males directly involved in the incident are safe and have been reunited with their camp group where they will receive support and care."

The caves, which were formed by underground rivers cutting through limestone rock, are filled with stalactites and stalagmites and are a popular tourist attraction.