Born in West Virginia, Willey served in the Navy during World War II. He was a ferociously tough fullback at Marshall and the Eagles made him their 13th-round pick in the 1950 draft. With two fullbacks ahead of him, Willey was moved to the other side of the ball, where coaches felt his aggression might be better channeled.