Recent work in imaging genetics suggests high levels of genetic overlap within cortical regions for cortical thickness (CT) and surface area (SA). We model this multivariate system of genetic relationships by applying Genomic Structural Equation Modeling (Genomic SEM) and parsimoniously define five genomic brain factors underlying both CT and SA along with a general factor capturing genetic overlap across all brain regions. We validate these factors by demonstrating the generalizability of the model to a semi-independent sample and show that the factors align with biologically and functionally relevant parcellations of the cortex. We apply Stratified Genomic SEM to identify specific categories of genes (e.g., neuronal cell types) that are disproportionately associated with pleiotropy across specific subclusters of brain regions, as indexed by the genomic factors. Finally, we examine genetic associations with psychiatric and cognitive correlates, finding that broad aspects of cognitive function are associated with a general factor for SA and that psychiatric associations are null. These analyses provide key insights into the multivariate genomic architecture of two critical features of the cerebral cortex.