Born to James Stevens Sr., a wholesale buyer and seller of produce, and Helen Paula Turner, a homemaker, James Wilmer Stevens Jr. graduated from Loyola Blakefield in 1953. Despite his size, chronic asthma kept him from playing sports. He studied one year at the University of Maryland then began working full time for the family wholesale company, Stevens Brothers. From a warehouse near present-day M&T Bank Stadium, they bought fresh fruits and vegetables from farmers and sold the produce to stores across Baltimore. Mr. Stevens would even travel with his father to avocado farms in New Mexico and tomato farms in California.