We all tend to anthropomorphize AI, and the AI industry particularly likes to borrow terminology from neuroscience and human cognition. “Neural networks,” for example, are named after, but only loosely similar to, biological neurons.
One term that feels especially familiar is memory. We increasingly hear about AI systems that “remember” us, recall previous conversations, or can be directed to forget things we have told them. But the familiarity of this term is misleading. It makes the concept seem intuitive while obscuring what how AI systems actually work – what they retain, how that information is used later, and what it really means for a system to “remember.” For companies building or deploying these systems, those details are not academic: they determine what personal data is collected, where it lives, who can manipulate it, and whether it can truly be deleted. For users, understanding the mechanics is a safeguard: recognizing that an assistant that ‘remembers’ you is just rereading prior content, not advancing a relationship.
What is memory, and why does AI need it?
One initial misconception is that the system learns from each user’s engagement directly, changing the underlying model. This is almost never the case. Underneath the conversational interface is an already trained language model, an elaborate “number soup” of billions of parameters that do not change every time you tell the assistant something. Chatting with an AI model is not the same as training it. Even within a single conversation, the application typically supplies the relevant history, or a summarized version of it, alongside each new message. What feels like remembering is the model being handed its notes again and again. Without that loop, every prompt would start from scratch – a limitation Andrej Karpathy has compared to anterograde amnesia.
It helps to distinguish three things often lumped together as “memory”: what a model learned during training (encoded in its parameters); what it has access to during the current interaction (its context); and what the application retains for future use. Memory mechanisms primarily concern the latter two. That distinction matters legally as well as technically. Information the application saves typically sits in identifiable, queryable storage – which may look like a conventional database of personal information, subject to access, correction, deletion, retention, and security obligations, and is not as much like the opaque contents of a trained model, or a shift of its model weights.
Short-Term and Long-Term Memory
AI memory is commonly divided into short-term and long-term memory, a distinction less about mimicking the brain than about how long information needs to remain available. Short-term memory concerns the current conversation or task – recent messages, tool outputs, or steps an agent has completed – processed through the model’s context window. Larger context windows do not guarantee better use of context. Models can overlook information buried in the middle of long inputs and degrade as context grows; Anthropic describes a finite “attention budget”. Fitting an entire book into a context window does not mean the model will ace an exam on page 145.
Long-term memory extends beyond the current conversation, allowing a system to retain information about users or completed work and retrieve it later. A system might maintain a structured user profile, store conversation summaries, or search saved interactions using embeddings, numerical representations that help identify semantically related content. Ask for dinner recommendations, and the system might retrieve an earlier note that you are allergic to gluten, even though your request never mentioned it.
So, is memory just a glorified RAG system?
Is this just retrieval-augmented generation (RAG) with better marketing? Not entirely wrong: memory retrieves and supplies context much like RAG, just drawn from past conversations and user preferences rather than reference documents. But a useful memory system must also identify what to save, how new information relates to existing records, and what to do when information becomes outdated or contradictory. If you move from Delhi to Chicago, the system should stop recommending Delhi restaurants (even if you preferred the food there, the commute might be a problem) while perhaps remembering that you used to live there.
This is where things start getting interesting. A memory system can preserve mistakes just as effectively as useful information. Ask for a bakery recommendation for a friend who loves a good sponge cake, and an overly enthusiastic memory system might conclude that you are no longer allergic to gluten (one can wish). Once saved, that assumption quietly shapes future recommendations. In legal terms around data management, this is an accuracy problem. The GDPR requires personal information to be accurate and kept up to date (Article 5(1)(d)) and provides a right to rectification (Article 16); the CCPA provides a right to correct. And because the CCPA expressly treats inferences used to profile a consumer as personal information, what a system concludes about a user can be as regulated as what the user actually said.
The Cost of Remembering
The same persistence that makes memory useful also introduces new risks that map fairly directly onto existing security, privacy, and AI-specific legal obligations.
Memory creates a new attack surface
If a system can store information that influences future behavior, an attacker has an incentive to manipulate what gets stored. A malicious instruction embedded in a document or web page may be treated as authoritative and saved to memory, and will then influence interactions long after the original document is gone. This is prompt injection with the added complication of persistence.
OWASP has identified memory and context poisoning as a security concern for agentic AI, and the risk is not hypothetical: In early 2026, Microsoft’s security researchers reported that some businesses were hiding instructions in “Summarize with AI” buttons on their websites, aiming to plant a lasting preference for the company in the user’s AI assistant. Over two months, they identified more than 50 such attempts from 31 companies across industries, including health and finance.
Persistent memory therefore falls squarely within existing security obligations, such as the GDPR’s requirement of security appropriate to the risk (Article 32), and the FTC’s long-standing position that inadequate data security can be an unfair practice. For high-risk systems, Article 15 of the EU AI Act requires resilience against attempts to alter a system’s outputs or performance by exploiting its vulnerabilities, expressly naming data poisoning and confidentiality attacks. For developers and operators, threat models and red-teaming should treat the memory store as a target in its own right.
Memory raises uncomfortable privacy questions
Over time, an assistant can accumulate a surprisingly detailed profile of its user from individually innocuous details. Some are sensitive by any legal measure: that gluten allergy is health information, and a system that infers a user’s health, religion, or sexual orientation from casual conversation may be creating special category data under GDPR Article 9, or sensitive personal information under the CCPA, without anyone deciding to collect it.
Information shared in a personal conversation might also later surface in an unrelated professional one, in tension with purpose limitation (GDPR Article 5(1)(b)) and the CCPA’s requirement that processing be reasonably necessary and proportionate to the purposes of collection. Accurate recall does not make it appropriate to bring something up. (Humans have not mastered this distinction either.)
Deletion can also be more complicated than it appears. OpenAI’s documentation on ChatGPT memory explains that deleting a conversation does not necessarily remove information retained separately through the memory functions. That is exactly the gap regulators look for: deletion rights under GDPR Article 17 and the CCPA reach personal information wherever it is retained. The FTC and DOJ’s 2023 action centered on allegations that some children’s recordings were kept indefinitely and deletion requests were not fully honored; the amended COPPA Rule now expressly prohibits indefinite retention of children’s data.
Default settings are another important consideration. Many popular chatbots have memory enabled by default, meaning users must actively opt out rather than choose to enable it. Choosing to be remembered differs meaningfully from having to opt out – and privacy-by-default obligations (GDPR Article 25) and state opt-in requirements for sensitive data make that a legal design choice, not just a product one.
Memory can change the relationship between users and AI
An assistant that remembers the miserable Tuesday you mentioned three months ago may start to feel like something that knows you. One concern is sycophancy – the tendency of AI systems to affirm users at the expense of accuracy. In research by MIT and Penn State, condensed user profiles produced the largest increase in sycophantic behavior among the forms of user context tested. A system that knows you better gets better at telling you what you want to hear.
This is particularly concerning for companion-style chatbots, where memory reinforces the impression that the system understands or cares about the user (which it does not). Lawmakers have taken notice and responded. New York’s 2025 law defines an “AI companion” in part by its retention of information on prior interactions and user preferences intended to personalize the interaction and facilitate ongoing engagement; memory is effectively part of the statutory trigger. California’s SB 243, effective January 1, 2026, imposes disclosure, crisis-protocol, and minor-specific safeguards, backed by a private right of action. The FTC’s September 2025 6(b) inquiry into seven companion chatbot providers asks how personal information from conversations is used or shared. And the EU AI Act prohibits AI systems that exploit vulnerabilities due to age, disability, or social or economic situation to materially distort behavior in ways that cause significant harm (Article 5(1)(b)). Remembering user preferences can certainly make an assistant more useful, so long as its memory is accurate, but remembering their vulnerabilities, and using those to strengthen user attachment to the system, is a rather different proposition – and, increasingly, a regulated one.
Deciding What is Worth Remembering
What does an AI system actually need to remember? The answer drives a design choice that should depend on what the system is intended to do, an approach familiar to privacy professionals. A coding assistant has a clear reason to remember a project’s conventions, but no reason to profile its user’s personal life. It can remember how you like your functions named without keeping a diary. Factors include:
Purpose: Information should be retained because it serves a legitimate function, not just because the system can retain it. This aligns with purpose limitation, data minimization, and storage limitation, and means clearly defining how long memories remain useful and whether they should carry over between tasks.
Control: Users should be able to see what a system remembers, correct what is wrong, and remove what they no longer want retained (e.g. access, correction, and deletion rights). The architecture must back this up: deletion should include stored memories, summaries, and embeddings to the extent possible; information from one context should not automatically become available everywhere; and access restrictions should be enforced by the application.
Provenance and reliability: A system should distinguish retained data by category: what a user provided, what the model inferred, and what came from external sources. An inference should not quietly become fact, and instructions from an external document should not acquire authority because they found their way into memory. Provenance is a security control, as well as the practical foundation for honoring correction requests.
Behavior: Evaluating memory should not stop at accuracy and curation though. Does personalization make the assistant merely more agreeable, or does it expressly encourage dependence? For products likely to reach children or other vulnerable users, avoiding companion-style dynamics should be a deliberate, documented design decision. This is the kind of record companion chatbot laws and regulators may increasingly expect.
AI “memory” is not inherently problematic. But first, it’s not actual memory in the getting to know you sense. It is simply stored content that is re-accessed when and how appropriate to achieve a new task. We may have borrowed the word “memory” from human cognition, but what these systems remember is the product of engineering, product, and increasingly, legal decisions about what gets saved, how it is interpreted, and when it resurfaces. Unlike your friend who remembers your gluten sensitivity because she was with you when you got sick one time, an AI system that remembers you is simply designed to read selected notes. The more convincing it becomes, the more that distinction matters.