Large Language Models (LLMs) and LLM-based agents show great promise in<br>accelerating scientific research. Existing benchmarks for measuring this<br>potential and guiding future development continue to evolve from pure recall<br>and rote knowledge tasks, towards more practical work such as literature review<br>and experimental planning. Bioinformatics is a domain where fully autonomous<br>AI-driven discovery may be near, but no extensive benchmarks for measuring<br>progress have been introduced to date. We therefore present the Bioinformatics<br>Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of<br>practical biological data analysis with nearly 300 associated open-answer<br>questions designed to measure the ability of LLM-based agents to explore<br>biological datasets, perform long, multi-step analytical trajectories, and<br>interpret the nuanced results of those analyses. We evaluate the performance of<br>two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework<br>we open source. We find that even the latest frontier models only achieve 17%<br>accuracy in the open-answer regime, and no better than random in a<br>multiple-choice setting. By exposing the current limitations of frontier<br>models, we hope BixBench can spur the development of agents capable of<br>conducting rigorous bioinformatic analysis and accelerate scientific discovery.