Paternal and maternal epigenomes undergo marked changes after fertilization1. Recent epigenomic studies have revealed the unusual chromatin landscapes that are present in oocytes, sperm and early preimplantation embryos, including atypical patterns of histone modifications2,3,4 and differences in chromosome organization and accessibility, both in gametes5,6,7,8 and after fertilization5,8,9,10. However, these studies have led to very different conclusions: the global absence of local topological-associated domains (TADs) in gametes and their appearance in the embryo8,9 versus the pre-existence of TADs and loops in the zygote5,11. The questions of whether parental structures can be inherited in the newly formed embryo and how these structures might relate to allele-specific gene regulation remain open. Here we map genomic interactions for each parental genome (including the X chromosome), using an optimized single-cell high-throughput chromosome conformation capture (HiC) protocol12,13, during preimplantation in the mouse. We integrate chromosome organization with allelic expression states and chromatin marks, and reveal that higher-order chromatin structure after fertilization coincides with an allele-specific enrichment of methylation of histone H3 at lysine 27. These early parental-specific domains correlate with gene repression and participate in parentally biased gene expression—including in recently described, transiently imprinted loci14. We also find TADs that arise in a non-parental-specific manner during a second wave of genome assembly. These de novo domains are associated with active chromatin. Finally, we obtain insights into the relationship between TADs and gene expression by investigating structural changes to the paternal X chromosome before and during X chromosome inactivation in preimplantation female embryos15. We find that TADs are lost as genes become silenced on the paternal X chromosome but linger in regions that escape X chromosome inactivation. These findings demonstrate the complex dynamics of three-dimensional genome organization and gene expression during early development.