Applying Spark on Kubernetes to process large-scale advertising data for LINE services

LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation.

Large-Scale LINE Ads Data Pipelines

• The data pipeline supports:
  • Real-time advertising-event processing
  • Abuse and validity checks
  • Machine-learning systems and model training
  • Analytics and system integration
  • Advertiser reporting
• The platform must handle hundreds of billions of events per day and hundreds of thousands per second.
• It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery.
• The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added.

Limitations of Spark on YARN

• Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components.
• Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity.
• JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features.
• Applications became tightly coupled to the Hadoop infrastructure.

How Spark on Kubernetes Works

• Kubernetes replaces YARN as the cluster manager.
• Spark drivers and executors run as separate Kubernetes pods.
• In cluster mode:
  • spark-submit requests a driver pod.
  • Kubernetes schedules the driver on an appropriate node.
  • The driver creates a SparkContext, builds the DAG, and requests executors.
  • Executors run as independent pods with individually allocated CPU and memory.
  • The driver divides the DAG into stages and distributes tasks to executors.
• Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured.

Advantages over YARN

• Containerized execution: Docker images package application dependencies, improving reproducibility and CI/CD integration.
• Infrastructure independence: Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster.
• Simpler autoscaling: Kubernetes can scale pods and integrate with cloud VM autoscalers.
• Unified platform: Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster.
• Governance and isolation: Namespaces, ResourceQuota, and RBAC provide flexible team-level controls.
• Operational automation: Helm, ArgoCD, GitOps, and rolling updates enable more automated application management.

LINE Ads’ Kubernetes-Based System

The platform is organized into four layers:

• Deployment layer

  • GitHub Actions runs CI workflows based on repository events.
  • ArgoCD monitors desired and deployed states and supports easier rollback and synchronization.

• Compute layer

  • Kubeflow’s Spark Operator deploys applications through the SparkApplication Kubernetes custom resource.
  • Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling.
  • LogSender forwards pod logs to OpenSearch.
  • ClusterMonitoring sends Prometheus metrics to the company’s monitoring system.

• Storage layer

  • Kafka provides high-throughput, low-latency storage for real-time advertising actions.
  • Hadoop remains available for large-scale, long-term analysis.

• Monitoring layer

  • Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging.

The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.