Although models developed using machine learning are increasingly prevalent in science, their opacity can limit their scientific utility. Explainable AI (XAI) aims to diminish this impact by rendering opaque models transparent. But, XAI is more than just the solution to a problem--it also plays an invaluable exploratory role. In this talk, I will introduce a series of XAI techniques and in each case demonstrate their potential usefulness for scientific exploration. In particular, I argue that these tools can be used to (1) better understand what an ML model is a model of, (2) engage in causal inference over high-dimensional nonlinear systems, and (3) generate "algorithmic-level" hypotheses in cognitive science.