Reliability of self-reported real-world psychedelic use – A test-retest study in a population sample

Objective: In 2024, the United States had the highest prevalence of psychedelic use recorded, but reliability of self-report data in surveys has not been rigorously assessed. Our objective was to quantify the reliability of self-report across 17 psychedelic and dissociative substances in an adult population sampled from the National Survey Investigating Hallucinogenic Trends (NSIHT).

Methods: In 2024, a test-retest sub-study was conducted on NSIHT. Overall agreement, average positive/negative agreement, and Gwet’s AC1 were calculated for use in the last 12 months. Reason and recency of use reliability was assessed through separate mixed effects logistic regressions models adjusting for all substances used and other reasons for use.

Results: There were 690 of 2,023 respondents (34.1%) who returned to the retest sample; there were minimal differences in demographics or none in health metrics of respondents and non-respondents. Across 17 substances, use showed “substantial” to “excellent reliability” indicated by Gwet’s AC1 (all ≥0.73, average 0.87), overall agreement (all ≥85.1%, average 90.4%). Overall agreement was driven by reliable report of “no” use of substances. Reliability for “yes” use decreased alongside decreasing prevalence of use. After adjustment, the probability of a reliable answer for all reported reasons and recency of use were 0.96 to 0.98.

Conclusions: This is the first study to demonstrate substantial to excellent reliability based on Gwet’s AC1 across 17 self-reported psychedelic and dissociative substances for use and behaviors. Methodological rigor and transparency allow for generation of trustworthy, reliable estimates and for the field to accurately interpret and compare findings across studies.