The first issue with your proposed setup is that your Mac can't communicate with any HomeKit devices. It's unclear why this is the case, and you would have thought that one Apple platform should be able to talk to the other. But, unfortunately, they can't.

Luckily, the Trådfri hub does support remote control from your Mac (or any device, really). You can talk to the gateway using CoAP (or find a library which does so, like pytradfri for Python).

You'll then need to connect that to a script which can determine the screen hue. A library like pymaScreen sounds like it fits the bill, and the task then is simply to connect the two.