Ambassador Charles A. James was born in 1922 in Washington, D.C. and grew up in Philadelphia, Pennsylvania where he attended public schools. After high school, James enrolled at Westchester State Teachers College in Pennsylvania (now Westchester University) where he studied for one year before enlisting in the U.S. Navy during World War II where he served for three years. James received his B.A. from Middlebury College in Middlebury, Vermont in 1949 and an LL.B. from Yale Law School in New Haven, Connecticut in 1952. In 1977, Middlebury College conferred an Honorary Doctor of Laws degree on Ambassador James for his lifetime of public service.

James practiced law in Sacramento and Stockton, California for ten years. In Stockton, he served as area President of the National Association for the Advancement of Colored People (NAACP) covering northern California and Nevada. In 1961, James was appointed Assistant Attorney General for the State of California by Attorney General Stanley Mosk. James played a significant role in the enactment of truth in lending legislation in California and negotiated changes in the packaging practices of 21 major cosmetic firms. He also served as chairman of the staff to the Committee for Youth and Children in the California State Assembly.

In 1964, James became the Deputy Director and later, Director of the Peace Corps, first in Ghana and then Uganda. He later served in an administrative position for the U.S. Agency for International Development (USAID) in Thailand and Vietnam. In 1974 he joined the U.S. Foreign Service and from 1974 to 1976, he worked as the Deputy Assistant Secretary of State for African Affairs in the U.S. State Department in Washington, D.C. During his tenure as Deputy Assistant Secretary, James headed the U.S. delegation to the African Economic Commission.

On September 16, 1976, President Gerald Ford nominated James as U.S. Ambassador to the Republic of Niger. After confirmation by the U.S. Senate, Ambassador Nelson traveled to Niamey, the capital, to