The Morrison government has granted a three-year $20m contract to maintain the Christmas Island detention centre, which currently houses just four people, to a company that mines phosphate in the remote Australian territory.

CI Resources, which won the contract, includes the former Western Australian government minister for state development Clive Brown on its board. It was one of two parties to tender for the contract, an Australian Border Force spokesman said.

Both border force and the company declined to say how many people were employed under the maintenance contract.

However, the Union of Christmas Island Workers secretary, Gordon Thomson, told Guardian Australia CI Resources employed about eight cleaners and six gardeners at the detention centre.

He said this was fewer than the approximately 25 cleaners and five to six gardeners employed when services giant Serco held the contract.

Serco continues to provide guards to the detention centre as part of a separate master contract with home affairs worth a total of $2.5bn.

Despite the lack of detainees, there are 96 Serco staff working at the centre, parliament heard on Monday.

Christmas Island’s only detainees are a family of two Tamil asylum seekers and their two Australian-born daughters, who are currently fighting moves by the home affairs minister, Peter Dutton, to deport them to Sri Lanka.

A border force spokesman said that under its contract, CI Resources is responsible for “repairs, maintenance, cleaning and waste removal” as well as transporting and booking rooms for detention centre staff.

He said CI Resources “has held contracts for similar services on Christmas Island”.

However, a company spokesman said it last worked at the centre about a decade ago.

The maintenance contract has the potential to boost ailing profits at CI Resources, which controls a phosphate mine on Christmas Island that has been hit by poor demand for the fertiliser in its key markets of Malaysia and Indonesia.

Sales of phosphate dropped from 643,000 tonnes to 486,000 tonnes in the 2018-19 financial year, the company said in its annual report, and profit fell from $21m to $8.6m.

Thomson said his union was currently locked in battle with CI Resources over a round of redundancies at the mine.

“They’ve been telling us for some time they’ll be reducing employment because they’ve got reduced demand,” he said.

“They say the market’s gone to shit and the price has gone to shit and everything has gone to shit.”

CI Resources, which claims to be directly or indirectly responsible for half the economic activity on Christmas Island, is closely linked to Malaysia’s Tee family, whose Prosper Group is a large palm oil producer. Palm oil is a controversial commodity because it is linked to the large-scale destruction of forests.

In the company’s annual report, the managing director, Lai Ah Hong, said the detention centre contract was “a wonderful result for [maintenance subsidiary] CIMS and the Christmas Island community!”

“This contract will provide continued employment opportunities for locals and contractors over its term while helping deliver on our diversification strategy,” he said.

Tender documents show the contract runs for three years from 1 July this year.

“The contract provides facilities management services, staff accommodation management and concierge services,” the border force spokesman said.

“Facilities management includes services such as repairs, maintenance, cleaning and waste removal.

“Staff accommodation management and concierge include services such as room bookings, check in, maintenance reporting and transport.”

However, Thomson said CI Resources was “still settling their services”.

“They haven’t got a real handle on the job yet, so we’ve objected,” he said.

He said the company wanted to employ people working at the centre as casuals but the union wanted them put on as permanent employees.