Large Language Models (LLMs) and LLM-based agents show great promise inaccelerating scientific research. Existing benchmarks for measuring thispotential and guiding future development continue to evolve from pure recalland rote knowledge tasks, towards more practical work such as literature reviewand experimental planning. Bioinformatics is a domain where fully autonomousAI-driven discovery may be near, but no extensive benchmarks for measuringprogress have been introduced to date. We therefore present the BioinformaticsBenchmark (BixBench), a dataset comprising over 50 real-world scenarios ofpractical biological data analysis with nearly 300 associated open-answerquestions designed to measure the ability of LLM-based agents to explorebiological datasets, perform long, multi-step analytical trajectories, andinterpret the nuanced results of those analyses. We evaluate the performance oftwo frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent frameworkwe open source. We find that even the latest frontier models only achieve 17%accuracy in the open-answer regime, and no better than random in amultiple-choice setting. By exposing the current limitations of frontiermodels, we hope BixBench can spur the development of agents capable ofconducting rigorous bioinformatic analysis and accelerate scientific discovery.