Zero. That is the number of LLM agents currently capable of maintaining a truly coherent, evolving user profile in a high-stakes environment without contradicting themselves within a few turns.
We have spent the last two years pretending that adding a vector database to a prompt constitutes “memory.” It does not. It is just retrieval. There is a massive difference between finding a document that says “User prefers low-risk bonds” and actually integrating that preference into a cognitive model that evolves as the user’s life changes. This is the gap that FinPerMA tries to measure.
The benchmark focuses on financial advising, a domain where memory isn’t just a convenience—it is a liability if handled poorly. If an agent forgets that a client just liquidated their primary asset or ignores a newly stated risk tolerance, the result isn’t a slightly awkward conversation; it is potentially catastrophic financial advice. FinPerMA tests whether agents can maintain and update an individualized user model over long horizons, rather than just reciting a static bio.
The current industry standard for “personalization” is essentially a glorified grep (which is usually just a JSON blob in a Redis cache). You store a fact, you retrieve the fact, you shove it into the context window. But humans don’t work like that. We synthesize. We update. We overwrite old beliefs with new ones. If a client tells a human advisor they are now terrified of volatility, the advisor doesn’t just append that to a list of facts; they fundamentally shift the lens through which they view every other piece of data about that client.
The real problem is that LLMs are fundamentally static. They are frozen snapshots of weights. To make them “remember,” we rely on the context window, which is a finite and expensive resource. Even with the massive windows we see now, the “lost in the middle” phenomenon persists. Moreover, the latency involved in processing 100k tokens just to remember a user’s birthday is absurd. You end up paying a “token tax” every single turn just to maintain the illusion of a relationship.
Why are we pretending that retrieving a document is the same as remembering a person?
It is like a waiter who remembers you are allergic to peanuts but forgets you are vegan. He has the “data” in his head, but he lacks the synthesis to apply it to the current order. Most agents today are that waiter. They can pull a fact from a database, but they cannot update the internal state of the user’s profile in a way that influences future reasoning without explicit, manual triggers. They are simply matching strings, not managing state.
Or maybe I am being too harsh—but the data suggests otherwise. If agents can’t pass a theory-informed benchmark like FinPerMA, they can’t be trusted with a portfolio.
The industry is currently obsessed with the size of the context window, but that is the wrong metric. The real battle is over state management. We need agents that can perform “lossy compression” on user interactions—deciding what to keep, what to discard, and how to merge conflicting information—without needing a human to curate the memory logs. This isn’t about how much you can fit in the prompt; it’s about how much you can ignore.
The current approach to personalization is a failure.
By Q4, we will see a shift toward state-managed memory architectures rather than simple RAG for these high-stakes agents. We will see a move toward “working memory” layers that sit between the vector store and the LLM, acting as a filter and a synthesizer. These layers will likely handle the “forgetting” process—which is just as important as the remembering process—to prevent the context window from becoming a noisy landfill of outdated preferences. Until then, every “personalized” financial assistant is just a fancy filing cabinet with a chat interface.
