techlist.io
KREN
line

Solving the Cold-Start Problem in Search Reranking Through Embedding Stabilization: A LINE Part Time Jobs Case Study (opens in new tab)

machine-learningapache-sparkonline-learningtwo-tower-embeddingsvdorthogonal-procrustesreal-time-rerankingcold-start

LY Corporation improved LINE Part Time Jobs’ real-time search reranking by stabilizing user and item embeddings produced by a two-tower recommendation model. The approach addressed both cold-start degradation and daily embedding-space drift without changing the underlying model or training pipeline. Offline and online evaluations showed substantial gains, including a 4.7% overall KPI increase and 6.5% revenue growth.

Search Reranking at LINE Part Time Jobs

  • Search consists of:
    • Retrieval, which finds listings matching a query.
    • Reranking, which orders the retrieved candidates.
  • The previous system ranked listings by cosine similarity between precomputed user-to-item two-tower embeddings.
  • This approach was computationally simple and captured broad user preferences, but:
    • It ignored query-specific information, such as the distance from a selected station.
    • Its embeddings combined behavior from multiple services and recommendation modules, not just search activity.
  • The team therefore introduced a dedicated real-time reranking model.

Challenges with the Dedicated Reranking Model

Cold Start

  • Most job listings are replaced at the beginning of each month.
  • New listings initially lack sufficient interaction data.
  • As a result, reranking quality dropped until enough training data accumulated.

Embedding-Space Drift

  • Two-tower models were regularly retrained from random initialization.
  • Each training run produced a different embedding space.
  • Using embeddings as downstream features caused a mismatch between training-time and inference-time data, reducing model performance.

Stabilizing the Embedding Space

  • Each day’s embeddings are aligned with the previous day’s stabilized embeddings.
  • The first day’s embeddings are used without stabilization.
  • This preserves continuity across retraining cycles and allows embeddings generated on different days to remain comparable.
  • Downstream models and embedding generation no longer need perfectly synchronized update schedules.

Low-Rank SVD

  • User and item embeddings are converted into a more standardized low-dimensional representation.
  • Instead of decomposing the enormous user-item score matrix directly, transformation matrices are derived from the embedding matrices.
  • This makes the procedure practical for large-scale data.

Orthogonal Procrustes Alignment

  • The transformed embeddings are aligned to the previous day’s stabilized space.
  • The orthogonal transformation only rotates or reflects the space.
  • Distances and inner-product relationships are therefore largely preserved, maintaining the two-tower model’s scoring behavior.

Scalable Implementation

  • The algorithm was implemented with Apache Spark to handle LINE Part Time Jobs’ large datasets.
  • For low-rank SVD:
    • The original QR decomposition was optimized using Cholesky decomposition.
    • The Gram matrix (G=A^\top A) is decomposed to obtain the same upper-triangular matrix (R) as QR decomposition.
  • For Procrustes alignment:
    • The large matrix multiplication (M=B^\top A) is distributed across Spark.
    • The resulting (e \times e) matrix is small enough for SVD on a single node using NumPy.

Evaluation Results

Embedding Stability

  • Before stabilization, embeddings from randomly selected days had correlations close to zero.
  • After stabilization:
    • Similarity remained around 0.88 after one week.
    • Similarity remained around 0.87 after one month.
  • This reduced performance loss caused by embedding drift.

Offline Evaluation

  • Unstabilized embeddings reduced nDCG by approximately 1–5% when training and inference used different days.
  • Stabilized embeddings improved:
    • Conversion nDCG by about 9.0%.
    • Click nDCG by about 4.5%.

Online A/B Test

  • Search-page KPIs alone did not show statistically significant improvement.
  • Across the entire service:
    • KPIs increased by 4.7%.
    • Revenue increased by 6.5%.
  • The results suggest that the embeddings captured long-term user preferences that influenced later actions across the service, not only behavior on the search page.
  • The added embedding features also helped mitigate the initial cold-start problem.

Practical Benefits and Future Work

  • The solution required no changes to the two-tower model itself.
  • Stabilization was added as post-processing, minimizing changes to existing pipelines and reducing deployment risk.
  • LY Corporation plans to test the method as the service expands its sources of job listings and to reuse the approach across other services through its internal machine-learning platform.

Overall, sequential low-rank SVD and orthogonal Procrustes alignment provide a relatively simple way to make frequently retrained embeddings reliable downstream features while improving real-time reranking and business outcomes.