A Cross-Entropy Audit (CEA) for Consciousness Tests Moved from Brain to Large Language Model

Abstract

Brain tests of consciousness are often applied to large language models (LLMs). Here consciousness means access: whether information, such as a heard sound, becomes available for report and use. A brain recording is in microvolts and an LLM's internal activations are arbitrary numbers, so the two share no common measurement unit. A brain test of this kind compares competing statistical models of how responses vary over repeated trials. The test's decision names the best-fitting model. The decision is wrong when the test names two states for responses that vary smoothly, or the reverse. No method we reviewed measures how often a transferred test is wrong. We call such a method a transfer audit and propose one, the cross-entropy audit (CEA). First, our audit holds back part of the trials and fits each model to the remaining trials. It then scores how well each fitted model predicts the held-back responses. Comparing two models gives a score difference in nats per trial. That difference does not depend on whether the recording is in microvolts or in activations, so brain and machine are scored on one scale. Second, the audit writes down five choices that such tests make for brain data: the competing models, their fitting, what counts as one independent sample, what the uncertainty interval describes, and the decision rule. Each may need to change on a machine. Third, the audit runs the test on simulated data built to follow one model, so the right decision is known, and counts wrong decisions. In our example, the two models of a two-state test, carried over unchanged to an LLM, found two states in 989 of 12,000 smoothly varying simulated datasets. The audited version of the test found two states in none of them. It detected 599 of 600 two-state datasets. The audited version's uncertainty interval, however, was too narrow at six of the twelve smoothly varying settings and is being repaired.