Introducing TabFM: A zero-shot foundation model for tabular data
TabFM is a foundation model for tabular classification and regression that uses in-context learning instead of dataset-specific training. It processes an entire table in one forward pass, eliminating much of the usual hyperparameter tuning and feature engineering. The authors report that it performs competitively with or better than tuned tree-based models, with planned integration into BigQuery. ## The Challenge of Traditional Tabular ML - Tabular prediction supports applications such as churn prediction and fraud detection. - Algorithms including XGBoost, AdaBoost, and random forests remain widely used. - Building reliable models typically requires: - Hyperparameter optimization - Cross-validation - Domain-specific feature engineering - Repeated model training for each dataset ## Tabular Prediction Through In-Context Learning - TabFM treats tabular prediction as an in-context learning problem. - Users provide historical training rows and target rows together as a single input. - The model learns relationships between rows and columns during inference without updating model weights. - This enables zero-shot predictions on previously unseen datasets. ## Hybrid Architecture TabFM combines ideas from TabPFN and TabICL to address the two-dimensional, order-independent nature of tables. - **Alternating row and column attention** - Attends across both features and examples. - Learns feature interactions and dependencies directly from the table. - Reduces the need for manually engineered features. - **Row compression** - Converts the contextualized information for each row into a dense vector. - **In-context Transformer** - Operates on compressed row representations rather than the full table grid. - Reduces computation and supports larger datasets efficiently. ## Training with Synthetic Data - Large, diverse real-world tables are difficult to obtain because they often contain proprietary or sensitive information. - TabFM is trained entirely on hundreds of millions of synthetic datasets. - Structural causal models generate tables using varied random functions and relationships. - The goal is to reproduce the broad range of distributions and feature interactions found in real-world data. ## Benchmark Results and Model Variants - Evaluation used TabArena, an Elo-based benchmark covering: - 38 classification datasets - 13 regression datasets - Dataset sizes from 700 to 150,000 samples - **TabFM** - Works out of the box. - Produces predictions in one forward pass without tuning or cross-validation. - **TabFM-Ensemble** - Adds cross features and SVD-derived features. - Uses a 32-model ensemble with non-negative least squares to optimize weights. - Applies Platt scaling for classification calibration. - The authors state that TabFM consistently outperforms heavily tuned conventional supervised algorithms. ## Availability and BigQuery Integration - TabFM is available through Google’s Hugging Face and GitHub repositories. - Google plans to integrate it into BigQuery. - Users will be able to run regression and classification through an `AI.PREDICT` SQL command without specialized machine-learning expertise. TabFM’s main promise is to make high-quality tabular prediction as simple as supplying a dataset and requesting predictions. Its zero-shot approach could reduce the engineering burden of traditional workflows, though practical adoption will depend on benchmark reproducibility, inference costs, and performance across specific production datasets.
Read original(opens in new tab)