Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
Yunshu Du, Garrett Warnell, Assefaw Gebremedhin, Peter Stone, Matthew E. Taylor
arXiv Preprint Archive September 29, 2020 preprint
Study at a glance
AI-extracted from the abstract| Characteristics | Case study Case report |
|---|---|
| Population | Six Atari 2600 games |
| Intervention | Lucid Dreaming for Experience Replay (LiDER) |
| Topics | Dreaming Lucid dreaming |
| Keywords | Cs.ai Stat.ml Machine-learning cs.lg Artificial intelligence Reinforcement-learning rl Deep rl Drl Artificial-dreams ai dreams Machine dreams Synthetic dreams |
| Key findings | LiDER consistently improves performance over the baseline in six Atari 2600 games. |
Abstract
Experience replay (ER) improves the data efficiency of off-policy reinforcement learning (RL) algorithms by allowing an agent to store and reuse its past experiences in a replay buffer. While many techniques have been proposed to enhance ER by biasing how experiences are sampled from the buffer, thus far they have not considered strategies for refreshing experiences inside the buffer. In this work, we introduce Lucid Dreaming for Experience Replay (LiDER), a conceptually new framework that allows replay experiences to be refreshed by leveraging the agent's current policy. LiDER consists of three steps: First, LiDER moves an agent back to a past state. Second, from that state, LiDER then lets the agent execute a sequence of actions by following its current policy -- as if the agent were "dreaming" about the past and can try out different behaviors to encounter new experiences in the dream. Third, LiDER stores and reuses the new experience if it turned out better than what the agent previously experienced, i.e., to refresh its memories. LiDER is designed to be easily incorporated into off-policy, multi-worker RL algorithms that use ER; we present in this work a case study of applying LiDER to an actor-critic based algorithm. Results show LiDER consistently improves performance over the baseline in six Atari 2600 games. Our open-source implementation of LiDER and the data used to generate all plots in this work are available at github.com/duyunshu/lucid-dreaming-for-exp-replay.
Comparable studies
Other case reports and case series on lucid dreaming, most cited first.
| Study | Year | Design | Participants |
|---|---|---|---|
| Lucid Dreaming as a Learnable Skill: A Case Study The author | 1980 | Case study | n = 1 |
| Lucid Dreaming as a Treatment for Recurrent Nightmares People with recurrent nightmares | 2010 | Case study | n = 5 |
| Lucid dreaming: Correspondence between dreamed and actual events in one subject during rem sleep A lucid dreamer | 1984 | Case study | n = 1 |
| Lucid dreaming treatment for nightmares: A series of cases. Participants with nightmares | 2003 | Case series | n = 8 |
| Nightmares in the Context of PTSD Treated with Psychoeducation Regarding Lucid Dreaming 39-year-old male with a history of suicide attempts and psychiatric admissions | 2010 | Case study | n = 1 |