We present a hybrid approach for simulating hypervelocity impacts onto asteroids. The overall system response is separated into two stages based on their different characteristic timescales. First, the short-timescale fragmentation phase is simulated using a modified version of the Tonge–Ramesh material model implemented in a Material Point Method framework. Then, a consistent hand-off to an N-body gravity code is formulated to execute the long-timescale gravitational reaccumulation calculation. We demonstrate this hybrid approach by considering the 5 km/s head-on impact of a 1.21 km diameter basalt impactor on a 25 km diameter target asteroid. The impact event resulted in the fragmentation, but not complete disruption, of the entire target. A granular core is observed at the end of the fragmentation simulations, which acts as a gravity well over which reaccumulation occurs in the N-body simulations. Our results suggest that disruption thresholds for rocky asteroids are higher when energy-dissipating mechanisms such as granular flow and pore collapse are included.