The family of Carl Sargeant have claimed they are being excluded from an inquiry into the circumstances surrounding his sacking as a Welsh minister four days before he took his own life.

Relatives of Sargeant, who lost his job amid allegations about his behaviour to women, are threatening to launch a judicial review if their barrister is not allowed to cross-examine witnesses.

They also claim the hearings have been deliberately arranged to take place at a time when the family barrister, Leslie Thomas QC, will not be able to attend because he is heavily involved in the Grenfell Tower inquiry.

Neil Hudgell, the family’s solicitor, called the situation “deeply unsatisfactory”. He said: “The grieving Sargeant family are losing patience and faith in the inquiry and are hurt and upset that everything they have asked for has been ignored.

“We currently have an inquiry process where there will be no effective involvement from the family. How can that be fair?”

Sargeant killed himself after being sacked as communities and children secretary by the Welsh first minister, Carwyn Jones. Sargeant’s family and friends claimed he was let down by Welsh Labour and after intense pressure, Jones announced that an inquiry would take place into how the sacking was handled.

Hudgell said the family was very concerned about decisions made by the permanent secretary, Shan Morgan, who leads the Welsh civil service, on how the inquiry will operate.

He said: “The permanent secretary, acting on behalf of the first minister, has refused to allow the family to have their own legal representation at the inquiry, meaning they will be unable to have a barrister cross-examine any of the witnesses.”

Hudgell said the family had taken “at face value” the assurances by the inquiry chair, Paul Bowen QC, that the process would be fair and independent.

But he added: “I have written to Paul Bowen and the permanent secretary to advise them that we believe the decision-making in relation to the protocol to be unreasonable and we will challenge it by way of a judicial review in the high court if we have to.”

Last month Jack Sargeant, who won his father’s Welsh assembly seat in a byelection, told the Guardian the family was upset at the time it was taking for an inquiry into the circumstances surrounding his father’s death to begin in earnest, and said his family had been unable to start the grieving process because of the delay.

The Welsh government insisted the family would be able to pose questions via an investigation team – which it said was a common feature of inquiries undertaken at the request of a government.

A spokesperson added: “No concerns about timing have been raised previously; indeed earlier representations about timing indicated a wish for the investigation to proceed as quickly as possible, which is now happening. We will consider the detailed legal arguments carefully but consider the proposed proceedings to be misconceived.” The hearings are expected to take place in the autumn.

On Friday at a pre-inquest review it emerged that Sargeant had depression for two years before he was found dead and had therapeutic levels of antidepressants in his system. The first minister will be called to give evidence at the inquest.

In the UK the Samaritans can be contacted on 116 123. In the US, the National Suicide Prevention Lifeline is 1-800-273-8255. In Australia, the crisis support service Lifeline is on 13 11 14. Other international suicide helplines can be found at www.befrienders.org.