Genetic and environmental factors both make substantial contributions to the heterogeneity in individuals’ levels of cognitive ability. Many studies have examined the relationship between educational attainment and cognitive performance and its rate of change. Yet there remains a gap in knowledge regarding whether the effect of genetic predictors on individual differences in cognition becomes more or less prominent over the life course. In this analysis of over 5000 older adults from the Health and Retirement Study (HRS) in the U.S., we measured the change in performance on global cognition, episodic memory, attention & concentration, and mental status over 14 years. Growth curve models are used to evaluate the association between a polygenic risk score for education (education PGS) and cognitive change. Using the most recent education PGS, we find that individuals with higher scores perform better across all measures of cognition in later life. Education PGS is associated with a faster decline in episodic memory in old age. The relationships are robust even after controlling for phenotypic educational attainment, and are unlikely to be driven by mortality bias. Future research should consider genetic effects when examining non-genetic factors in cognitive decline. Our findings represent a need to understand the mechanisms between genetic endowment of educational attainment and cognitive decline from a biological angle.