It would note the people identified in a photo, and how frequently the people are included in your pictures. Then, it would assess information from comments on the photos, captions, or tags (#family, #mom, #kids)–anything that indicates whether someone is a husband, daughter, cousin, etc.–to predict what your family/household actually looks like.

According to the patent application, Facebook’s prediction models would also analyze “messaging history, past tagging history, [and] web browsing history” to see if multiple people share IP addresses (a unique identifier for every internet network).

In one specific example, the model looked at an image a user posted with two females tagged, “#my_boss_at_home,” and another image with a young girl marked “my angel.” Facebook predicted that there were three people in the household, including the male user, and two females, “who are likely the male user’s wife and daughter.”