Agriculture is a significant contributor to phosphorus (P) enrichment in aquatic ecosystems. No‐till (NT) farming has been proposed as an alternative approach to conventional tillage (CT) in reducing soil P export, but published data have shown contrasting impacts, likely due to the interacting effects of different physical (climate region, rainfall variability, transport pathway, slope gradient) and management variables (NT duration, crop species). We conducted a meta‐analysis to understand the extent to which each of these variables controls the concentration and load of different P fractions (dissolved P, particulate P) in agricultural runoff and leaching. In comparison with CT, particulate P loss was significantly lower with NT adoption (45 and 55% reduction in concentration and load, respectively), but an increase in dissolved P loss was observed. The extent of the reduction or increase, however, varied with different physical and management variables. In comparison with CT, for example, NT was not effective in reducing particulate P concentration during wet years and particulate P load on steep slopes (4–9%). Total P concentration was also similar with CT at sites under prolonged NT duration (∼10 yr) and at NT fields planted with soybean [Glycine max (L.) Merr.]. Our results underscore the need to consider the covarying physical and management factors when assessing the potential of NT farming in controlling P loss in the environment. The limited impact of NT on dissolved P loss remains a serious impediment toward harnessing the water quality benefits of this management practice.