Studies have demonstrated that at least 20% of individuals infected with SARS-CoV-2 remain asymptomatic. Although most global efforts have focused on severe illness in COVID-19, examining asymptomatic infection provides a unique opportunity to consider early immunological features that promote rapid viral clearance. Here, postulating that variation in the human leukocyte antigen (HLA) loci may underly processes mediating asymptomatic infection, we enrolled 29,947 individuals, for whom high-resolution HLA genotyping data were available, in a smartphone-based study designed to track COVID-19 symptoms and outcomes. Our discovery cohort (n = 1,428) comprised unvaccinated individuals who reported a positive test result for SARS-CoV-2. We tested for association of five HLA loci with disease course and identified a strong association between HLA-B*15:01 and asymptomatic infection, observed in two independent cohorts. Suggesting that this genetic association is due to pre-existing T cell immunity, we show that T cells from pre-pandemic samples from individuals carrying HLA-B*15:01 were reactive to the immunodominant SARS-CoV-2 S-derived peptide NQKLIANQF. The majority of the reactive T cells displayed a memory phenotype, were highly polyfunctional and were cross-reactive to a peptide derived from seasonal coronaviruses. The crystal structure of HLA-B*15:01-peptide complexes demonstrates that the peptides NQKLIANQF and NQKLIANAF (from OC43-CoV and HKU1-CoV) share a similar ability to be stabilized and presented by HLA-B*15:01. Finally, we show that the structural similarity of the peptides underpins T cell cross-reactivity of high-affinity public T cell receptors, providing the molecular basis for HLA-B*15:01-mediated pre-existing immunity.