Source Google Research · Published · Open the original ↗
Lab notes
The sleep framework: A new way for LLMs to retain knowledge in continual learning
Education Innovation
We introduce a sleep phase in continual learning, enabling large language models to continuously integrate new facts and enhance reasoning while mitigating catastrophic forgetting or static knowledge cutoffs.
Modern LLMs achieve impressive performance via pre-training and in-context learning (ICL). However, once deployed, they remain largely static, operating with a fixed knowledge cutoff date that leaves them unable to acquire long-term capabilities or adapt to new information.
While updating models with new data seems like the obvious solution, continual parameter updates often trigger catastrophic forgetting , a fundamental issue where learning new tasks severely degrades performance on previously learned ones. Alternatively, relying strictly on ICL limits knowledge retention to the immediate context window: once the context session ends, the acquired short-term information is forgotten.
The opening of the post, quoted unchanged from Google Research. The full text continues at the source.