Self Supervised Learning

metaJul 15, 20263 min readCurated summary

Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization

Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems. ## Purpose and Role in Deep-Funnel Optimization - The system aims to identify people with genuine latent interest in an advertiser’s offerings. - It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model. - It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences. - The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance. ## Technical Challenges ### Sparse Engagement and Large-Scale Graphs - Users, advertisers, products, services, and campaigns are modeled as graph nodes. - Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users. - Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence. ### Long-Range Relationships - Useful signals may come from indirectly connected users and entities rather than direct interactions. - Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms. ### Dynamic and Unseen Entities - The ads ecosystem changes rapidly, while individual entities may have little historical engagement. - Representations must generalize to rare and previously unseen businesses and products. ## Core Design Properties ### Dimension Reduction - The raw graph is projected into a configurable “super-graph.” - Learned latent interest primitives act as super-nodes. - Sparse user-ad relationships become denser connections at the interest level. - The primitive graph is more stable and stationary than the constantly changing ads vocabulary. ### Knowledge Enrichment - Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes. - Vision and language models process this multimodal content. - Content helps the system understand what a product or business is, not merely how users interacted with it. - This enables better generalization to new or sparsely observed entities. ### Unified Relational Representation - Users, advertisers, products, and latent interest primitives are embedded in a shared metric space. - The system can estimate: - Relationships between interest primitives - Similarity between users, ads, and products - A user’s proximity to particular interests - Which interests an advertiser or product serves - Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling. ### Multiple Hierarchical Granularities - Coarse representations capture dense, stable, high-level interests. - Finer representations capture sparse and specific deep-funnel intent. - Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision. ## Architecture and Training - The architecture combines: - An in-house transformer-based graph learner - Bias-aware attention - Self-supervised cross-view distillation - Sparse attention for long-range graph relationships - It combines real-world semantic knowledge with users’ temporal engagement histories. - The model learns multi-hierarchical interest representations across a large graph. - Training is performed end-to-end on real Meta Ads data involving billions of interactions. ## Outputs and Potential Applications - Universal embeddings for users and ads entities. - “Bag-of-Meaning” interest tokens representing latent interests at different granularities. - Potential uses include: - Ad retrieval - Personalization - Ranking - Specialized ranking architectures - Training supervision - Cross-entity similarity and discovery Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.

Read original

Self Supervised Learning Large Language Models Multimodal AI Transformer+4

googleJul 9, 20263 min readCurated summary

SensorFM: Towards a general intelligence and interface for wearable health data

SensorFM is a large foundation model designed to turn wearable-device signals into a reusable representation of human physiology. Trained on more than one trillion minutes of multimodal data from five million people, it performs well across 35 health-related prediction tasks while requiring relatively few labeled examples. The authors argue that scaling both data and model size enables a general-purpose interface for wearable health data rather than isolated, task-specific models. ## The challenge of wearable health data - Wearables continuously capture heart rate, movement, temperature, blood oxygen, sleep, and related signals. - Interpreting these measurements is difficult because: - Baseline physiology and lifestyle vary substantially between individuals. - Reliable labels such as diagnoses, lab results, and validated questionnaires are costly and difficult to collect. - Traditional models usually target one health outcome at a time and generalize poorly. ## Training on more than a trillion minutes - SensorFM was trained on de-identified data from five million consenting participants collected between September 2024 and September 2025. - The dataset includes: - More than 100 countries and all 50 U.S. states. - Over 20 Fitbit and Pixel Watch models. - More than two billion sensor-hours of data. - The model processes 34 minute-level features from: - PPG - Accelerometry - Electrodermal activity - Skin temperature - Altimetry - These signals represent heart rate, heart-rate variability, blood oxygen, sleep, movement, skin conductance, and temperature across full days. ## Learning from incomplete sensor recordings - SensorFM uses self-supervised masked reconstruction rather than relying on medical labels. - Its Adaptive and Inherited Masking framework treats naturally missing data as part of the learning problem. - This avoids: - Imputing gaps, which can introduce bias. - Discarding incomplete windows, which wastes real-world data. - The resulting representation is explicitly aware of missingness and can learn from fragmented wearable recordings. ## Scaling data and model capacity - Experiments varied training data from roughly two million to two billion sensor-hours and model size from 100,000 to 100 million parameters. - Larger models trained on more data consistently improved both reconstruction and downstream health prediction. - The largest model: - Reduced reconstruction loss by 31% compared with the smallest version. - Improved classification performance by an average of 9% in AUC. - Improved regression performance by 21% in Pearson correlation. - Won on 33 of 35 downstream tasks. - Scaling data and model size together produced near-linear gains with no observed saturation. ## One representation across many health domains - SensorFM was tested on 35 tasks from three prospective studies involving 13,985 participants. - The tasks covered: - Cardiovascular health - Metabolic risk - Mental health - Sleep - Demographics - Lifestyle - With the encoder frozen and only a lightweight linear head trained, SensorFM embeddings outperformed engineered-feature supervised baselines on 34 of 35 tasks. - Larger models appeared to learn physiologically relevant demographic and individual differences without being explicitly given demographic inputs. - The model showed particular value for difficult-to-measure conditions such as depression and anxiety. - It also reached strong performance with relatively small quantities of labeled data, addressing a major constraint in healthcare modeling. ## Automated adaptation through an agentic “classroom” - The authors introduce a collaborative system of LLM agents intended to automate the creation of prediction heads. - This approach aims to reduce the manual work traditionally required for: - Feature engineering - Architecture selection - Hyperparameter tuning - The supplied article excerpt ends while introducing this system, so its detailed workflow and results are not described here. SensorFM demonstrates that large-scale, missingness-aware pre-training can produce a broadly useful representation of wearable physiology. Its strongest practical promise is label-efficient adaptation across many health applications, potentially providing a foundation for personalized health agents and more general wearable-data interfaces.

Read original

Self Supervised Learning Healthcare Ai Foundation Model Wearable Health+2

googleMar 24, 20263 min readCurated summary

Mapping the modern world: How S2Vec learns the language of our cities

S2Vec is a self-supervised framework that converts buildings, roads, businesses, and infrastructure into general-purpose geospatial embeddings. By rasterizing these features into S2 Geometry cells and training a masked autoencoder to reconstruct missing areas, it learns the spatial “character” of neighborhoods without manually labeled data. It performs especially well for socioeconomic predictions in geographically unseen regions, while environmental tasks benefit from combining it with satellite imagery. ## Turning Geospatial Data into Images - Geospatial data is multimodal and unevenly distributed: urban blocks may contain hundreds of features, while rural areas contain few. - S2Vec uses hierarchical S2 Geometry cells to divide the Earth into regions at different resolutions. - It counts feature types within each cell—such as buildings, parks, roads, and businesses—and organizes them into multilayered raster images. - This makes complex geographic information compatible with computer vision methods developed for ordinary images. ## Learning with Masked Autoencoding - S2Vec masks portions of the rasterized map and trains a model to reconstruct the missing features from surrounding context. - Repeated training across global locations teaches relationships among urban elements, such as the likelihood of shops near residential buildings and transit stations. - The resulting embeddings are compact numerical representations of each location’s built environment. - Because training is self-supervised, S2Vec does not require worldwide labels for income, air quality, population, or other metrics. - The model can identify similar neighborhood types without being explicitly told concepts such as “financial district” or “suburban residential area.” ## Evaluation and Socioeconomic Performance - S2Vec was compared with models including SATCLIP, GEOCLIP, RS-MaMMUT, Hex2vec, and GeoVeX. - Tests covered population density, median income, carbon emissions, tree cover, and elevation. - Models were evaluated using mean squared error and both: - Interpolation, using random train/test splits - Extrapolation, predicting conditions in geographically unseen regions - S2Vec was generally the strongest individual model for zero-shot socioeconomic prediction, including population density and median income. - It performed competitively with established image-based approaches and exceeded GEOCLIP in the reported comparisons. ## Benefits of Multimodal Fusion - Combining S2Vec with satellite-image embeddings generally produced better results than either modality alone. - Built-environment data captures structures and infrastructure, while satellite imagery adds information about vegetation, terrain, and transportation patterns. - Fusion was particularly valuable for environmental prediction tasks. ## Limitations on Environmental Tasks - Built-environment features alone do not fully explain factors such as tree cover and elevation. - S2Vec was competitive for carbon-emissions prediction but weaker on some environmental metrics. - Satellite imagery embeddings improved performance by supplying information unavailable from counts of buildings, roads, and businesses. S2Vec points toward scalable geographic foundation models that replace task-specific feature engineering with reusable representations. In practice, it is most effective when its built-environment embeddings are combined with complementary imagery, especially for environmental analysis.

Read original

Self Supervised Learning Computer Vision Foundation Models Geospatial Data+4

googleJul 21, 2025Original article

LSM-2: Learning from incomplete wearable sensor data (opens in new tab)

LSM-2 introduces a paradigm shift in processing wearable sensor data by treating naturally occurring data gaps as inherent features rather than errors to be corrected. By utilizing the Adaptive and Inherited Masking (AIM) framework, the model learns directly from fragmented, real-world data streams without the need for biased imputation or data-discarding filters. This approach allows LSM-2 to achieve state-of-the-art performance in health-related classification and regression tasks, maintaining robustness even when sensors fail or data is highly interrupted. ## The Challenge of Pervasive Missingness * Real-world wearable data is almost never continuous; factors such as device charging, motion artifacts, and battery-saving modes create frequent "missingness." * Traditional self-supervised learning models require complete data, forcing researchers to use imputation—which can introduce artificial bias—or aggressive filtering that discards over 90% of potentially useful samples. * In a dataset of 1.6 million day-long windows, research found that not a single sample had 0% missingness, highlighting the impracticality of training only on complete datasets. ## Adaptive and Inherited Masking (AIM) * AIM extends the Masked Autoencoder (MAE) framework by treating "inherited" masks (naturally occurring gaps) and "artificial" masks (training objectives) as equivalent. * The framework utilizes a dual masking strategy: it employs token dropout on a fixed ratio of tokens to ensure computational efficiency during encoding. * To handle the unpredictable and variable nature of real-world gaps, AIM uses attention masking within the transformer blocks for any remaining masked tokens. * During evaluation and fine-tuning, the model relies solely on attention masking to navigate naturally occurring gaps, allowing for accurate physiological modeling without filling in missing values. ## Scale and Training Architecture * LSM-2 was trained on a massive dataset comprising 40 million hours of de-identified wearable data from more than 60,000 participants using Fitbit and Google Pixel devices. * The model learns to understand underlying physiological structures by reconstructing masked segments across multimodal inputs, including heart signals, sleep patterns, and activity levels. * Because it is trained on fragmented data, the resulting foundation model is significantly more resilient to sensor dropouts in downstream tasks like hypertension prediction or stress monitoring. LSM-2 demonstrates that foundation models for health should be built to embrace the messiness of real-world environments. By integrating missingness directly into the self-supervised learning objective, developers can bypass the computational and statistical overhead of imputation while building more reliable diagnostic and monitoring tools.

Self Supervised Learning AI Transformer Foundation Models+3