In a large saucepan, heat 4 cups milk until bubbles form around sides of pan. Meanwhile, in a large bowl, whisk eggs and 1 cup sugar until blended. Slowly stir in hot milk; return all to saucepan.

Cook over medium-low heat until slightly thickened and a thermometer reads at least 160°, 6-8 minutes, stirring constantly (do not allow to boil). Immediately transfer to a large bowl.

Stir in rum, brandy, nutmeg and remaining milk. Refrigerate, covered, several hours or until cold.