Spatial normalization or deformation to a standard brain template is routinely used as a key module in various pipelines for the processing of magnetic resonance imaging (MRI) data. Brain templates are often constructed using MRI data from a limited number of subjects. Individual brains show significant variabilities in their morphology; thus, sample sizes and population differences are two key factors that influence brain template construction. To address these influences, we employed two independent groups from the Human Connectome Project (HCP) and the Chinese Human Connectome Project (CHCP) to quantify the impacts of sample sizes and population on brain template construction. We first assessed the effect of sample size on the construction of volumetric brain templates using data subsets from the HCP and CHCP datasets. We applied a voxel-wise index of the deformation variability and a logarithmically transformed Jacobian determinant to quantify the variability associated with the template construction and modeled the brain template variability as a power function of the sample size. At the system level, the frontoparietal control network and dorsal attention network demonstrated higher deformation variability and logged Jacobian determinants, whereas other primary networks showed lower variability. To investigate the population differences, we constructed Caucasian and Chinese standard brain atlases (namely, US200 and CN200). The two demographically matched templates, particularly the language-related areas, exhibited dramatic bilaterally in supramarginal gyri and inferior frontal gyri differences in their deformation variability and logged Jacobian determinant. Using independent data from the HCP and CHCP, we examined the segmentation and registration accuracy and observed significant reduction in the performance of the brain segmentation and registration when the population-mismatched templates were used in the spatial normalization. Our findings provide evidence to support the use of population-matched templates in human brain mapping studies. The US200 and CN200 templates have been released on the Neuroimage Informatics Tools and Resources Clearinghouse (NITRC) website (https://www.nitrc.org/projects/us200_cn200/).