$\begingroup$

I was lately curious about an iterative approach that would solve maths equations containing composed functions with contraints.

For example, if I have the following equation:

$$ f(g(h(w))) = 0 \text{, with } w = \begin{pmatrix} a_{11} & 0 & \ldots & a_{1n}\\ 0 & a_{22} & \ldots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 &\ldots & a_{nn} \end{pmatrix} $$

Along additional constraints on the $3$ functions like $ f < g $; $ h > 2 * g $; and $ f, g,h $ not constant

The goal is to find the $3$ functions expressions given a specific matrix $w$ and the constraints.

What approach would be the most convenient to find a solution or solutions to this problem ? I was personally thinking about using reinforcement learning (machine learning) where each time a solutions is chosen, a positive or negative reward will be attributed to the solution generator.

Thank you