Far-field optical imaging techniques allow the determination of the position of point-like emitters and scatterers1,2,3. Although the optical wavelength sets a fundamental limit to the image resolution of unknown objects, the position of an individual emitter can in principle be estimated from the image with arbitrary precision. This is used, for example, in the determination of the position of stars4 or in optical super-resolution microscopy5. Furthermore, precise position determination is an experimental prerequisite for the manipulation and measurement of individual quantum systems, such as atoms, ions and solid-state-based quantum emitters6,7,8. Here we demonstrate that spin–orbit coupling of light in the emission of elliptically polarized emitters can lead to systematic, wavelength-scale errors in the estimation of the emitter’s position. Imaging a single trapped atom as well as a single sub-wavelength-diameter gold nanoparticle, we demonstrate a shift between the emitters’ measured and actual positions, which is comparable to the optical wavelength. For certain settings, the expected shift can become arbitrarily large. Beyond optical imaging techniques, our findings could be relevant for the localization of objects using any type of wave that carries orbital angular momentum relative to the emitter’s position with a component orthogonal to the direction of observation.