Autonomous systems are soon to be ubiquitous, from manufacturing autonomy to
agricultural field robots, and from health care assistants to the entertainment
industry. The majority of these systems are developed with modular
sub-components for decision-making, planning, and control that may be
hand-engineered or learning-based. While these existing approaches have been
shown to perform well under the situations they were specifically designed for,
they can perform especially poorly in rare, out-of-distribution scenarios that
will undoubtedly arise at test-time. The rise of foundation models trained on
multiple tasks with impressively large datasets from a variety of fields has
led researchers to believe that these models may provide common sense reasoning
that existing planners are missing. Researchers posit that this common sense
reasoning will bridge the gap between algorithm development and deployment to
out-of-distribution tasks, like how humans adapt to unexpected scenarios. Large
language models have already penetrated the robotics and autonomous systems
domains as researchers are scrambling to showcase their potential use cases in
deployment. While this application direction is very promising empirically,
foundation models are known to hallucinate and generate decisions that may
sound reasonable, but are in fact poor. We argue there is a need to step back
and simultaneously design systems that can quantify the certainty of a model's
decision, and detect when it may be hallucinating. In this work, we discuss the
current use cases of foundation models for decision-making tasks, provide a
general definition for hallucinations with examples, discuss existing
approaches to hallucination detection and mitigation with a focus on decision
problems, and explore areas for further research in this exciting field.