Recently, I was reading a research article from a16z, and there was an analogy that I found quite interesting—LLMs actually live in an eternal present, just like the amnesiac protagonist in the movie "Memento." Once trained, they are frozen; new information can't be integrated, and they can only rely on external tools like chat logs and retrieval systems for emergency responses. But is that really enough?

More and more researchers believe it's not. Contextual learning is indeed useful, but fundamentally, it’s retrieval, not learning. Imagine an infinitely large filing cabinet where you can find anything, but it has never been forced to understand, compress, or truly internalize new knowledge. For problems that require genuine discovery—such as entirely new mathematical proofs, adversarial scenarios, or knowledge that is too implicit or inexpressible in language—relying solely on retrieval is definitely insufficient.

This is why continuous learning is becoming an increasingly important research direction. The core question is simple: **Where does compression happen?** Current systems outsource compression to prompt engineering, RAG pipelines, and agent shells. But the mechanism that makes LLMs powerful during training—lossy compression and parameter-level learning—is turned off at deployment.

The research community roughly divides into three paths. One is situational learning, where teams focus on optimizing retrieval pipelines, context management, and multi-agent architectures. This is the most mature, with infrastructure validated, but the ceiling is the context length limit. The other end is weight-level learning, which involves actual parameter updates—sparse memory layers, reinforcement learning loops, training during inference. In the middle is the modular approach, which achieves specialization through pluggable knowledge modules without altering core weights.

There are many directions within weight-level research. Some involve regularization methods (like EWC), some involve training during inference (performing gradient descent during reasoning), some involve meta-learning (training models to learn how to learn), and others include self-distillation and recursive self-improvement. These directions are converging, and the next generation of systems will likely blend multiple strategies.

But here’s a key issue: naive weight updates in production environments can cause a host of problems. Catastrophic forgetting, temporal decoupling, logical integration failures, and the fundamental difficulty of operations like forgetting. Even more problematic are safety and governance concerns—once training and deployment boundaries are opened, alignment may collapse, data poisoning attack surfaces are exposed, auditability disappears, and privacy risks increase. These are open problems, but they are also part of the research agenda.

Interestingly, the entrepreneurial ecosystem is already moving at these levels. On the situational side, companies like Letta and mem0 are managing context strategies; on the parameter side, teams are experimenting with partial compression, RL feedback loops, data center methods, and even radical redesigns of architecture. No single approach has yet emerged as the winner, and given the diversity of use cases, perhaps there shouldn’t be only one.

From a certain perspective, we are at a turning point. Retrieval systems are indeed powerful, but retrieval is never equivalent to learning. A truly capable model that can continue compressing experiences and internalizing new knowledge after deployment will generate compound value in ways current systems cannot. This implies advances in sparse architectures, meta-learning, and self-improvement cycles, and may also mean we need to redefine what a “model” is—no longer just a fixed set of weights, but an evolving system.

The future of continual learning lies here. A filing cabinet, no matter how large, is still just a filing cabinet. The breakthrough will come from enabling models to do the training that makes them powerful after deployment: compression, abstraction, and genuine learning. Otherwise, we risk being trapped in our own eternal present.