In 2025, we began using LLMs to rethink how we understood users on a content platform.

I previously wrote about the starting point of this system in “From Labels to Memory: Rebuilding How Internet Apps Understand Users”. Traditional user profiles resemble a collection of sticky notes. They record who a user is, what they clicked, and which segments they belong to. An LLM can read a sequence of behavior over time, distinguish long-term preferences from short-term intent and one-off actions, and turn user understanding from discrete labels into semantic memories backed by evidence.

That article focused on why the system worked. This one explains how it entered real business workflows and how online experiments changed our view of its value.

We ran many online experiments across new users, returning users, dormant users, and the full user population. Most produced positive results that were strong enough for broader rollout, improving activity, retention, content consumption conversion, and paid conversion. To keep the cases anonymized, I will not share the exact lifts. I will focus on what we tested and which conclusions changed as a result.

Why 2025

The idea had existed for years. User profiling has always tried to answer two questions: who is this user, and why do they behave this way? Traditional recommendation systems are already good at predicting the next click from behavioral sequences. But that understanding usually remains inside features and task-specific models. It is difficult to express as evidence-backed situations and intent, and difficult to reuse across recommendation, search, marketing, and Agents.

By 2025, several conditions had changed at the same time.

First, open-source models such as DeepSeek and Qwen had become capable enough at long-text understanding and structured output. Companies could deploy them in their own environments without sending user behavior to external APIs. For a high-privacy, high-volume task such as user profiling, private deployment was a prerequisite.

Second, capable small models were emerging quickly. Processing every user with the strongest model would still be too expensive, even with private deployment. But a strong model could generate high-quality samples, which could then be used to fine-tune or distill a smaller model. User summaries, interest and intent inference, and structured profiles could become a focused task. The model did not need to do everything. It only needed to produce these outputs reliably.

Third, engineering approaches had made inference cheaper. Profiles could be produced offline in batches, with the online path reading only the results. Memories could be updated incrementally instead of rereading the full history every time. Users with different activity levels could also be assigned different update frequencies.

Open-source models solved the deployment and capability constraints. Fine-tuning and distillation made specialized small models practical at scale. Offline and incremental architectures brought total cost within range. Together, these conditions let us treat LLM-based user profiling as a production system rather than a proof of concept.

How LLM-based profiles are produced

The full pipeline can be reduced to a few steps:

Raw behavior
  → Clean and convert behavioral sequences into text
  → Route inference by user lifecycle
  → User summary + long-term memory + short-term memory
  → Structured profile + semantic vectors + Agent Context
  → Recommendation, search, marketing, and Agents

The first step is not to send raw logs directly to an LLM. Logs contain repeated events, technical fields, and invalid actions. Even after reading them all, the model would struggle to tell what matters. We first organize searches, browsing, content consumption, saves, payments, and negative feedback into a timeline. We merge consecutive repeated actions and preserve the time, object, behavioral strength, and context.

The resulting input is no longer a list of event IDs. It becomes a behavioral narrative the model can understand. For example, a user may consistently consume financial content on weekday mornings while browsing lighter content in the evenings. That contains more than two content labels. It shows stable interests, consumption periods, and a recurring shift between contexts.

The second step is to route inference by user lifecycle. More profiling is not always better. For every inference task, we consider how much the existing data can already answer, what traditional models already do well, what the LLM can add, and how much the computation will cost. These constraints determine which prompt to use, how much context to read, when to trigger inference, and how often to update the profile.

Active users generate rich behavior that changes constantly, so their profiles need frequent updates. But rereading their full history is also the most expensive approach, while traditional features are already strong at predicting short-term behavior. Each update therefore reads only the existing memories and the latest incremental behavior, using the LLM to separate stable preferences from short-term exploration.

Returning users do not need continuous computation. When a user returns, the system reads the historical memories and recent signals to determine whether earlier interests still hold. The inference goes deeper, but it runs only when there is a clear business opportunity.

New users provide little evidence, so the input is shorter and traditional recommendation has fewer signals to work with. The marginal value of an LLM is higher during cold start. But the system does not treat inference as fact. It produces probabilistic judgments with confidence scores.

The third step is to generate three outputs.

The user summary is a compact piece of natural language that answers, broadly, “Who is this user?” It supports retrieval, operational analysis, and Agent personalization.

Long-term memory is a set of independently updatable and retrievable memory fragments that capture relatively stable user understanding. Every memory must include behavioral evidence. Only signals repeated across multiple days, or strong actions such as saving, paying, and repeated consumption, can enter long-term memory. One accidental click cannot rewrite our long-term view of a person.

Short-term memory also uses fragments, but records intent that is emerging now. It updates and decays faster. It captures changes such as: “This user has always liked finance, but this week they have been exploring one new topic intensively.”

The demo below preserves the structure of the actual profiles. User attributes, content names, and memory fragments have all been synthesized and anonymized. They do not correspond to a real user.

User summary:

The user lives in a major city and is at a stable stage in their career. They have an established habit of consuming information-dense content. Their main preferences include:
1. Macroeconomics and market analysis, followed consistently on weekday mornings, often through complete consumption of daily content;
2. AI and the technology industry, with a preference for in-depth interviews and industry analysis around midday;
3. Music and lighter content, consumed more heavily in the evening, forming a stable shift in context from daytime information gathering.
Interest in humanoid robots and market volatility has risen noticeably in recent weeks, representing a short period of concentrated exploration within existing long-term interests.

Long-term memory fragments (excerpt):

- The user is a consistent consumer of information-dense content, with long-term interest in finance and the technology industry, alongside stable music consumption. Evidence: all three content categories appear repeatedly across multiple days, with high completion rates for in-depth finance and technology content.
- The user consistently follows "Market Morning Brief" and "Technology Industry Interviews." The former serves as a morning information source, while the latter provides long-term views from founders and industry experts. Evidence: both series show week-over-week following and repeated consumption.
- On weekday mornings, the user tracks market and macroeconomic changes. At midday, they consume in-depth technology industry content. In the evening, they switch to music and lighter content. The user appears to use continuous information intake to follow external change and understand how new technologies reach the market. Evidence: this time-of-day distribution and content choice recur across multiple weekdays.

Short-term memory fragments (excerpt):

- Progress and commercialization of humanoid robots: over the past week, the user repeatedly searched for humanoid robots and consumed a sequence of in-depth content on embodied intelligence, the industry chain, and commercialization, mostly around midday on weekdays.
- Asset allocation during market volatility: over the past three days, the user concentrated on market changes, risk, and allocation strategies, preferring content with clear conclusions and logical support, mostly on weekday mornings.

These three outputs serve different purposes. The user summary lets a person or Agent understand a user quickly. Long-term memory preserves repeated, relatively stable judgments. Short-term memory records only recent changes. They are not long, medium, and short versions of the same profile.

During initialization, the model reads a longer behavioral sequence and builds the first set of memories. Later updates do not reread the full history. They send the existing memories and new behavior to the model and update only the fragments that changed. When new behavior conflicts with an earlier judgment, it is first recorded as a temporary change so that one accidental click does not rewrite a long-term preference.

Generating the profile is not the end of the work. Recommendation and search systems need structured fields and semantic vectors. Agents are better suited to reading user summaries and memory fragments directly. We express the same user understanding in different forms for each business workflow.

Before launch, we evaluate the profiles separately. We check whether conclusions have behavioral evidence, whether they overreach, and whether short-term behavior has been mistaken for a long-term preference. People then review low-scoring samples.

New users: What is missing is usable signal

New users have almost no in-product behavior. This is where traditional recommendation is weakest. Without enough clicks, content consumption, or payment history, collaborative filtering struggles to find reliable similar users. But new users are not entirely without signals. Interests they select, acquisition sources, external interest data, and their first few actions all provide clues.

In the production pipeline above, new users enter a sparse-profile branch. Before inference, we filter out broad, low-discrimination signals. Otherwise, the LLM can easily turn noise into generic interests. The remaining clues come from different channels and vary in strength and time horizon. The LLM reasons across them and organizes them into a set of interest judgments with evidence and confidence. It does not directly decide what to recommend.

The existing recommendation system still consumes these judgments. They become new input features alongside popularity, content quality, and real-time behavior in retrieval and ranking. In online experiments, both activity and paid conversion improved for new users.

The value of the LLM here is to produce a testable understanding of the user while behavioral evidence is still sparse.

Returning users: Filling the behavioral gap

Returning users do not lack history. The problem is the gap between their history and the present. A user may not have opened the product for several months. It is difficult to tell from their last few clicks how much of their old preferences still holds or why they have returned now.

These users enter the returning-user profile branch. We first compress their long-term behavior into a semantic summary, then vectorize it to find active users with similar interest structures. Content currently consumed by those active users becomes a candidate set for recommendations to the returning user.

This approach produced a positive improvement in retention. Its role is specific: when real-time behavior is insufficient, a more complete semantic view of long-term interests helps the recommendation system find an interest it can restart.

All users: Semantic memory and behavioral features should work together

At one point, we hoped to replace traditional features with vectors generated by the LLM. In practice, that was the wrong direction.

Real-time clicks, recent consumption, and frequency statistics remain strong predictors of the next action. The LLM is better at something else: extracting stable preferences, consumption contexts, and latent intent from a longer time horizon and a broader range of behaviors.

The later architecture retained signals at three timescales. Real-time behavior answers, “What did the user just do?” Short-term memory answers, “What has the user been exploring recently?” Long-term memory answers, “Which preferences are stable, and who is this user over time?” In recommendation experiments across the full user population, semantic retrieval based on long-term memory produced incremental gains in content consumption conversion and retention.

Personalized push notifications: Deciding whom to reach, with what, and when

The value of user memory in push notifications goes beyond generating more human-sounding copy.

Traditional push notifications often rely on predefined labels and fixed audience segments. The system may know that a user is interested in technology, but not that they have recently been exploring humanoid robots intensively or when they are more likely to consume that content. A coarse “technology interest” label cannot decide the content, timing, and message at the same time.

With user memory connected, long-term memory first determines whether a topic fits a stable interest. Short-term memory checks for clear recent intent. Context memory then identifies a more suitable time to reach the user. The system sends a push notification only when these signals align. The copy also focuses on the specific question the user is exploring rather than a broad trending topic.

For the user in the demo, long-term memory shows sustained interest in the technology industry. Short-term memory shows recent intensive exploration of humanoid robots. Time-of-day patterns show that midday is better suited to in-depth content. Together, these three signals produce one specific outreach decision.

This strategy produced a clear increase in push click-through rate in online experiments and was later expanded. The user’s clicks and content consumption then become new behavioral signals, return to short-term memory, and influence the audience, content, and timing of the next decision.

We also tried using user memory solely for personalized subscription messaging. The results varied by lifecycle stage. It produced incremental revenue for users already showing churn risk, but was less stable for other groups. Taken together, the two experiments showed that personalization creates value by changing the full outreach decision. Changing only the copy is not enough.

What these experiments changed our minds about

First, the value of an LLM-based user profile does not lie in the profile itself. A human-sounding user summary has limited business value if it only sits in an internal dashboard for operators to inspect. It needs to become recommendation features, retrieval vectors, or Agent Context and enter a real execution workflow.

Second, LLMs should be used first where traditional systems lack signals. New users, returning users, long-tail content, and complex intent are better candidates than predicting the next click of a highly active user. The latter already generates rich real-time behavior, so the marginal gain from an LLM is smaller.

Third, online experiments are the final Eval. We can check whether a user summary is accurate, whether it has evidence, and whether it mistakes short-term behavior for a long-term preference. Those evaluations can only establish that the profile meets a quality bar. Whether it helps users stay, consume more content, or pay still requires a controlled experiment.

Can LLM-based user profiles drive growth?

My answer is yes. Their clearest advantage over traditional profiles is the ability to organize behavioral signals scattered across systems and timescales into semantic assets with evidence, context, and motivation. That understanding no longer remains trapped in the features and models of one task. Recommendation, search, marketing, and Agents can all use it.

This understanding is especially useful when traditional signals are insufficient. New users have not accumulated enough clicks, so an LLM can produce confidence-scored interest judgments from a small number of clues. Returning users have a break in their real-time behavior, so long-term memory can help the system recover interests that still hold. For complex intent, semantic vectors and memory fragments can also match suitable content more effectively than fixed labels.

It does not replace traditional user profiling. Real-time clicks, consumption frequency, and statistical features remain good at predicting the next action. The LLM adds semantics that those features struggle to express explicitly or reuse across systems. Raw behavior becomes evidence-backed user memory, which then enters recommendation, search, and marketing systems as structured data, semantic vectors, or Context. I prefer to think of LLM-based profiling as a semantic layer on top of traditional profiling. Where traditional signals are sparse or intent is difficult to express directly, it adds a set of judgments that business systems can use and online experiments can validate.