Building materials from the atom up is the pinnacle of materials fabrication. Until recently the only platform that offered single‐atom manipulation was scanning tunneling microscopy. Here controlled manipulation and assembly of a few atom structures are demonstrated by bringing together single atoms using a scanning transmission electron microscope. An atomically focused electron beam is used to introduce Si substitutional defects and defect clusters in graphene with spatial control of a few nanometers and enable controlled motion of Si atoms. The Si substitutional defects are then further manipulated to form dimers, trimers, and more complex structures. The dynamics of a beam‐induced atomic‐scale chemical process is captured in a time‐series of images at atomic resolution. These studies suggest that control of the e‐beam‐induced local processes offers the next step toward atom‐by‐atom nanofabrication, providing an enabling tool for the study of atomic‐scale chemistry in 2D materials and fabrication of predefined structures and defects with atomic specificity.