Could Large Language Models Be Useful in Theory Development?

Abstract

This paper presents an analysis of OpenAI's O3 language model to assess its capacity for reasoning about the non-mainstream framework of Topological Geometrodynamics (TGD). Through a series of prompts in a single, continuous session, we evaluated the model's ability to explain core TGD concepts. Our findings reveal several systematic pitfalls in the model's reasoning, including a strong bias towards mainstream physics, hallucination of mathematical formulas and citations, and a fragmented understanding of foundational principles. Despite these significant failures, the exercise proved beneficial. The process of correcting the model's errors forced us to study the earlier views in more detail and allowed us to fill in the details in some earlier views. We conclude that while current large language models are not reliable authorities for specialized technical claims, they can serve as valuable, interactive tools for getting a bird's eye view of theoretical structures involving a large number of evolving ideas, provided their outputs are subject to deep expert scrutiny.