Paul Jennings was born a slave on James Madison's plantation. When Madison was elected President in 1809, Jennings accompanied the Madisons to the White House, and was present for the burning of the public buildings of Washington in 1814. He returned with the Madisons to Virginia after 1817. In the 1820s, he married a fellow slave named Fanny and they had five children.

When Dolley Madison returned to Washington in 1837, following the death of James Madison, she took Jennings with her but not his family. His wife died in 1844, the same year that Mrs. Madison sold her Virginia home. Jennings' children were sold to other slave owners in Virginia. Although her will had stated that Jennings would be freed, Mrs. Madison sold him in 1846 to Pollard Webb. Six months later, in early 1847, he was purchased by Senator Daniel Webster who then freed him in exchange for work.

Jennings became a prominent member of the free black community of Washington, DC. In 1848 he helped to arrange an escape attempt for slaves in the district, whose path took them across the Mall. In 1851, with a glowing recommendation from Webster, Jennings took a job at the pension office of the Department of the Interior. It was there he met a coworker from Massachusetts who encouraged him to record some of his life experiences. His memoir, A Colored Man's Reminiscences of James Madison, was published in 1863; it contains the only first-hand account of Madison's death.

Jennings briefly returned to Virginia in 1850 where he was able to find and secure freedom for his children. In 1849, he had remarried to a free mulatto woman from Alexandria, and he settled the children from his first marriage next door to his new home on L Street NW. When Jennings died in 1874, he was able to leave his children a house and valuable property.