I have a similar problem with a project of a different language.

As far as I was able to analyze it, this is due to how the dates of the files are checked.

The unzipped files from the cache all have their proper timestamps, but the checked out files from the git repository all have a current creation timestamp and therefore appear to be newer than their cached artifacts to the compiler.

I have not found a way around this. At least for my project the cache worked for the artifcats of dependencies, which solved most of my compile time issues.

For my rust project I have not yet tried to set up caching.