Deterministic quantum interactions between single photons and single quantum emitters are a vital building block towards the distribution of quantum information between remote systems1,2,3,4. Deterministic photon–atom state transfer has previously been demonstrated with protocols that include active feedback or synchronized control pulses5,6,7,8,9,10. Here we demonstrate a passive swap operation between the states of a single photon and a single atom. The underlying mechanism is single-photon Raman interaction11,12,13,14,15—an interference-based scheme that leads to deterministic interaction between two photonic modes and the two ground states of a Λ-system. Using a nanofibre-coupled microsphere resonator coupled to single Rb atoms, we swap a photonic qubit into the atom and back, demonstrating fidelities exceeding the classical threshold of 2/3 in both directions. In this simultaneous write and read process, the returning photon, which carries the readout of the atomic qubit, also heralds the successful arrival of the write photon. Requiring no control fields, this single-step gate takes place automatically at the timescale of the atom’s cavity-enhanced spontaneous emission. Applicable to any waveguide-coupled Λ-system, this mechanism, which can also be harnessed to construct universal gates16,17, provides a versatile building block for the modular scaling up of quantum information systems.