I managed to fix this on my end.

In my case it turned out that my setup seems still to be affected by a slight drift.

I have especially stand corrected, that my setup does only show this on Skyrim. This was incorrect from my end. I tried some other software (Holoball to be precise), and it apparently also showed the drift. The difference between the other titles and Skyrim seems to be, that when recentering via the OPTIONS button, Skyrim does not recenter the HUD.

So to first sum up my symptoms were:

- HUD is moving around me, drifting more and more to the right

- Weapons followed the HUD, i.e. at some point the weapons stuck almost at a 90° angle, compared to my physical hands.

- Turning the headset off and on (while running the game), fixed it temporarily, as this resets HUD positioning (and also the weapons/hands). However this is just a temporary fix, as after a few minutes we are back with the problem.

To get this sorted, I did a bit of elimation of different theories.

FIrst thing, I checked it is camera-tracking related and not caused by the Gyros in the HMD. I did so, by simply using Cinema mode in middle or high size. I quit the game, stayed in the PS4 dashboard, and center initially. Then I turned my head side to side for around a minute or so. Afterwards I double checked, that the screen was still centered - which it was, so no Gyro problem.

With that knowledge, what I changed next is the following list of things:

- raised camera position to be more on head level with me (I play standing, but have the camera usually on top of my monitor on which I play non-VR)

- double checked distance to camera. Turned out, I was at the outer rim of what Sony recommends (1,5m-2m / 5-7ft). I move my central position to 1,5m

- and finally, the crazy(?) fix - I hung a dark grey blanket over the white sideboard that is to my right when playing

The last step is what actually fixed it. My best guess is, that although the furniture is not glossy, it still causes some kind of reflection.

I just finished a 90 minute session with the game, without any drift whatsoever.

To be clear - what I fixed here was a general problem with my PSVR setup, not a problem specific to Skyrim. However, the lack of support for resetting via the OPTIONS button is making this very common PSVR issue more apparent.

Hope this helps others.