I know little about computer vision and I'm trying to find a way to keep an image at the same perspective of its projection (obtained by a camera). The ideia is to get an image (a) and, using the image of its projection (b), do an affine transform to obtain (c). I'm using opencv's ORB algorithm to do a feature matching and then evaluate the transformation matrix. The problem is that it doesn't work with symmetric figures (like a square).

How can I solve for the general case? Any other suggestions that don't use algorithms like ORB are welcome.

Example of the described process for a square

Ps 1: The projection image (b) may have noise, as shown in the figure.

Ps 2: I'm using only images with one channel (grayscale).