How will long-term user modeling (opens in new tab)

Long-term user modeling captures persistent interests, cross-vertical behavior, and signals beyond what short-term recommendation logs can reveal. 당근 built a Transformer-based user encoder that learns from tens of billions of actions across its local marketplace, jobs, real estate, and other services, then exposes the resulting embedding as a shared feature for ranking, retrieval, and advertising models. The approach improved scalability and reuse, but introduced freshness and representation-transfer limitations.

Why Long-Term User Modeling Matters

• Recent actions reveal immediate intent, but miss recurring interests such as seasonal shopping or repeated moving-related searches.
• Long-term, cross-vertical activity can connect behaviors such as:
  • Searching for real estate
  • Looking for furniture and appliances
  • Reading neighborhood moving advice
• Longer histories can reduce selection bias caused by training only on items previously exposed by recommendation models.
• Simply adding more history is insufficient because ranking systems are latency-sensitive and long sequences increase computation and infrastructure complexity.

Shared User Embeddings as a Common Feature

• A separate user encoder processes long-term history offline.
• Home-feed ranking, candidate generation, and advertising models consume the resulting embedding as a shared user feature.
• Benefits:
  • Downstream models avoid directly processing massive histories.
  • The encoder can scale independently in model size, data, and compute.
  • One embedding can be reused across multiple recommendation surfaces.
• Limitations:
  • A downstream model receives only a fixed vector, so it cannot fully exploit the encoder’s richer representations.
  • Batch inference means recent actions are not reflected immediately.
• Possible future improvements include more frequent or real-time updates, fine-tuning, and distillation.

Contrastive User Modeling

• The encoder uses a two-tower architecture:
  • A causal Transformer converts the user’s action sequence into a user embedding.
  • An MLP converts item features into item embeddings.
  • InfoNCE loss trains the user embedding to predict the next interacted item.
• In-batch negatives provide alternative items for contrastive learning.
• Training uses clicks and conversion actions across all major verticals and surfaces.
• The dataset contains tens of billions of actions—around 150 times more than the existing home-feed candidate model’s training data.

Item ID Embeddings vs. Content Embeddings

Problems with Item ID Embeddings

• New items have no learned ID embedding, creating a cold-item problem.
• Hundreds of millions of item IDs require enormous embedding tables.
• In the ID-based model, embedding tables accounted for over 99% of parameters, leaving little GPU capacity for the Transformer.
• Hashing and embedding-sharding techniques were considered but did not provide a sufficient solution.

Content Embeddings

• The system switched to LLM-generated embeddings based on post metadata.
• This enables:
  • Representations for newly created items
  • Much larger Transformer models, with Transformer parameters becoming roughly 1,000 times larger than in the ID-based setup
• Large-scale lookup required two memory-efficient techniques:
  • memmap loads only needed embedding segments from disk and benefits from shared OS page caches during distributed training.
  • bbhash maps item IDs to embedding locations using roughly three bits per key, reducing mapping memory by about 97% compared with Python dictionaries.
• Together, these methods made training with hundreds of gigabytes of item embeddings practical.

Region-Constrained Batch Sampling

• Standard in-batch negatives assume that other items in the batch were visible but not selected.
• This assumption fails in a local service: users generally cannot view items outside their geographic area.
• More than 86% of transactions occur within five kilometers, yet random batches mixed users and items nationwide.
• Consequently, about 98% of random in-batch negatives were “impossible negatives”—items users could never have seen.
• These negatives teach geographic unavailability rather than user preference, weakening the contrastive signal.

RCBS Solution

• Region-Constrained Batch Sampling (RCBS) groups users from the same region into a batch.
• This reduced impossible negatives from 98% to 30%.
• The remaining impossible negatives mainly came from differences in viewing radius or users’ historical activity in other regions.
• Feasible negatives are harder because they represent items users could have viewed but rejected, forcing the model to distinguish genuine preferences among similar local items.

Why Sampling Was Better Than Masking

• Masking impossible negatives would remove most of the batch, drastically reducing effective batch size.
• Hard-negative mining would require checking feasibility separately for each user and could be expensive and complex.
• RCBS naturally produces more feasible and difficult negatives without changing the loss function or adding specialized mining.

Applying the Embeddings

• For home-feed and advertising ranking, the embedding is projected and concatenated with existing features.
• The long-term encoder supplies persistent preference signals, while existing ranking models continue handling short-term and real-time signals.
• For retrieval models, the best-performing approach used the user embedding alone to generate candidates rather than merely adding it as another feature.
• The separate candidate source appeared to improve recommendation diversity.

Embedding Refresh and Serving

• Offline tests showed little difference between frozen embeddings and 12- or 24-hour refreshes.
• Online A/B tests favored periodic updates, with shorter intervals performing better.
• A 24-hour refresh cycle was selected as the best cost-performance trade-off.
• GPU inference runs through a Beam pipeline on GCP Dataflow.
• Only users who acted during the refresh window are reprocessed, avoiding unnecessary inference for inactive users.
• Near-real-time inference remains a major future engineering challenge.

The overall recommendation is to treat long-term user modeling as a separate, reusable representation system rather than forcing every downstream model to process extensive histories directly. For geographically constrained services, the training data pipeline—especially negative sampling—must reflect actual item visibility, making region-aware batching as important as the model architecture itself.