We present an approach for estimating the fraction of text in a large corpuswhich is likely to be substantially modified or produced by a large languagemodel (LLM). Our maximum likelihood model leverages expert-written andAI-generated reference texts to accurately and efficiently examine real-worldLLM-use at the corpus level. We apply this approach to a case study ofscientific peer review in AI conferences that took place after the release ofChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggestthat between 6.5% and 16.9% of text submitted as peer reviews to theseconferences could have been substantially modified by LLMs, i.e. beyondspell-checking or minor writing updates. The circumstances in which generatedtext occurs offer insight into user behavior: the estimated fraction ofLLM-generated text is higher in reviews which report lower confidence, weresubmitted close to the deadline, and from reviewers who are less likely torespond to author rebuttals. We also observe corpus-level trends in generatedtext which may be too subtle to detect at the individual level, and discuss theimplications of such trends on peer review. We call for futureinterdisciplinary work to examine how LLM use is changing our information andknowledge practices.