🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked.
- Representation Learning: A Review and New Perspectives - Reviews probabilistic models, autoencoders and manifold learning as approaches to learning representations. Bengio et al. 2013.
- A Survey of Dimensionality Reduction Techniques - Categorizes dimensionality-reduction techniques and explains their mathematical foundations. Sorzano et al. 2014.
- Feature Selection and Feature Extraction in Pattern Analysis - Reviews the theory, applications, and numerical implementations of feature selection and extraction methods. Ghojogh et al. 2019.
- Text vectorization - Sparse word/character n-grams, TF-IDF, and feature hashing. docs.
- Image descriptors - SIFT describes local keypoints; HOG summarizes local gradient orientations. Lowe 2004, Dalal and Triggs 2005.
- Time-series statistics - tsfresh combines a large feature bank with relevance filtering; catch22 provides 22 selected time-series characteristics. Christ et al. 2018, Lubba et al. 2019; code, code.
- Wavelet scattering - Cascades of wavelet transforms, nonlinearities, and averaging. Kymatio implements scattering for 1D, 2D, and 3D inputs. Andreux et al. 2020; code.
- Convolutional transforms - ROCKET, MiniROCKET, and MultiROCKET produce convolution-response statistics for linear classification; extraction can expand dimensionality. Dempster et al. 2020, Dempster et al. 2021, Tan et al. 2022; code.
- OpenFE - Expands tabular features with predefined operators, scores their incremental predictive value using feature boosting, and prunes candidates in two stages. Generation and supervised selection are coupled. Zhang et al. 2023.
- TabPrep - Jointly tunes feature generators and downstream models, unlike OpenFE's separate feature selection. With four model families and post-hoc ensembling, improves the best score on 19 of 51 TabArena datasets against newer model baselines. Tschalzev et al. 2026.
- PCA (principal component analysis) - Orthogonal projection maximizing retained variance of centered data. Pearson 1901; docs.
- Dual PCA - Computes PCA through the sample Gram matrix. Ghojogh and Crowley 2019.
- Incremental PCA - Batchwise PCA updates. docs, code.
- Truncated SVD / latent semantic analysis - Low-rank approximation without PCA's centering step, allowing sparse document-term inputs. Deerwester et al. 1990; docs.
- NMF (non-negative matrix factorization) - Additive factorization of non-negative data. Lee and Seung 1999 🔒; docs.
- Factor analysis - A linear latent-factor model with feature-specific Gaussian noise. docs.
- ICA (independent component analysis) - Linear separation of statistically independent sources. Hyvärinen and Oja 2000, Hyvärinen et al. 2001; docs.
- Topic LDA (latent Dirichlet allocation) - Document representations as topic mixtures. Blei et al. 2003, Hoffman et al. 2010.
- Robust PCA / principal component pursuit - Separates low-rank structure from sparse corruption (
X = L + S), with recovery guarantees under rank, incoherence, and corruption assumptions. Candes et al. 2011.
- Sparse PCA - Sparse component loadings, selecting which input variables contribute to each component. Zou et al. 2006; docs.
- Dictionary learning / sparse coding - Sparse sample codes over learned, possibly overcomplete dictionaries; sparsity applies to the codes rather than PCA loadings. Mairal et al. 2010; docs.
- Fisher LDA (linear discriminant analysis) - Supervised projection maximizing between-class relative to within-class scatter. Fisher 1936, Rao 1948, Martinez and Kak 2001; docs.
- Supervised PCA - Response-based predictor screening followed by PCA. Bair et al. 2006.
- CCA / PLS - Canonical correlation analysis maximizes correlation between projected views; partial least squares extracts cross-covariance-based components, including for regression. docs.
- Kernel PCA - Component analysis in an implicit kernel feature space. Schölkopf et al. 1998; docs.
- Kernel PCA extensions - Kernel PCA for novelty detection and robust kernel PCA. Hoffmann 2007, Nguyen and De la Torre 2008.
- Kernel Fisher discriminant analysis - Supervised class separation in a kernel feature space. Mika et al. 1999.
- Nyström approximation - Sampled data landmarks approximate the kernel matrix; an alternative to Fourier sampling. Williams and Seeger 2000; docs.
- Random projections - Gaussian or sparse random maps approximately preserve Euclidean distances under Johnson-Lindenstrauss conditions. Achlioptas 2003; docs.
- Random Fourier features - Sampled Fourier maps approximate shift-invariant kernels. Unlike distance-preserving random projections, these construct nonlinear features and may expand dimension. Rahimi and Recht 2007.
- MDS (multidimensional scaling) - Fits coordinates to dissimilarities or their ordering. Kruskal 1964, Little et al. 2018, Little et al. 2022; docs.
- Sammon mapping - Weighted distance reconstruction emphasizing smaller input distances. Sammon 1969.
- SOM (self-organizing maps) - Organizes adaptive units into a low-dimensional array whose responses reflect input topology. Kohonen also examines conditions where ordering fails. Kohonen 1982 🔒.
- Isomap - MDS on graph-approximated geodesic distances. Tenenbaum et al. 2000; docs.
- LLE (locally linear embedding) - Fits coordinates using fixed local linear reconstruction weights. Roweis and Saul 2000; docs.
- HLLE (Hessian eigenmapping) - Manifold coordinates from local Hessian estimates. Supported by scikit-learn's LLE interface. Donoho and Grimes 2003.
- Laplacian eigenmaps / spectral embedding - Graph-local similarity represented by Laplacian eigenvectors. Belkin and Niyogi 2003.
- Maximum variance unfolding / semidefinite embedding - Maximizes spread while preserving local distances through semidefinite optimization. Weinberger et al. 2005.
- Diffusion maps - Spectral coordinates reflecting random-walk connectivity across scales. Coifman and Lafon 2006.
- PHATE - Embeds distances between transformed diffusion probabilities to visualize continuous and branching structure. Moon et al. 2019 🔒; code.
- t-SNE - Local neighborhood probability matching. van der Maaten and Hinton 2008, van der Maaten 2014; docs, docs.
- Extensions - Tree-SNE for hierarchical clustering; LEt-SNE for hyperspectral imagery. Robinson and Pierce-Hoffman 2020, Shukla et al. 2019; code.
- LargeVis - Approximate neighborhood-graph construction and probabilistic layout for large collections. Tang et al. 2016.
- UMAP - Embeds a fuzzy neighborhood graph. McInnes et al. 2018; code, docs.
- Parametric UMAP - Trains a neural encoder with the UMAP objective. Sainburg et al. 2021; docs.
- densMAP - Adds relative local-density preservation to UMAP. Narayan et al. 2021 🔒; docs.
- MAPLE - Adds self-supervised manifold modeling to UMAP using maximum manifold capacity representations. Huang et al. 2026, Huang et al. 2026.
- TriMap - Relative similarity preservation through triplets. Amid and Warmuth 2019; code.
- PaCMAP - Combines neighbor, mid-near, and distant pairs to preserve structure at different scales. Wang et al. 2021; code.
- LocalMAP - Adjusts the neighborhood graph during optimization to improve local cluster separation. Implemented in the PaCMAP repository. Wang et al. 2025.
- ParamRepulsor - Hard-negative mining and stronger repulsion address loss of local detail in parametric embeddings. Huang et al. 2024; code.
- New observations - Ordinary UMAP and openTSNE place new points relative to a fitted embedding. Parametric UMAP instead evaluates a learned encoder; new-point support alone does not imply a neural mapping. docs, docs.
- Competitive learning - Rumelhart and Zipser study how competition among units discovers feature detectors without target labels, including multilayer representations of nonlinearly separable patterns. Rumelhart and Zipser 1985.
- Modular autoassociative learning - Couples autoassociative modules into hierarchies to learn internal representations. An early approach to organizing learning in networks with multiple hidden levels. Ballard 1987.
- Autoencoders - Encoder-decoder learning through reconstruction. Hinton and Salakhutdinov 2006.
- Denoising - Learns representations by reconstructing clean inputs from corrupted versions. Stacked denoising modules can initialize deeper networks. Vincent et al. 2008, Vincent et al. 2010.
- Contractive autoencoders - Adds a penalty on the encoder's input Jacobian to reconstruction loss, encouraging local insensitivity of the representation to input changes. Rifai et al. 2011.
- Variational autoencoders - See Generative Models for the latent distribution, approximate posterior and variational objective.
- Masked - Encodes visible image patches and reconstructs missing pixels with a lightweight decoder. Representation learning operates on the visible subset, separating encoder cost from full-image reconstruction. He et al. 2022; code.
- Autoencoder objectives - Learns an encoder and reconstruction decoder under restrictions that prevent unrestricted copying. Covers bottlenecks, regularization, and links to latent-variable models. Goodfellow et al. 2016.
- Contrastive Predictive Coding - Predicts future latent representations from autoregressive context using a contrastive loss and negative samples; learns sequential features without reconstructing raw observations. van den Oord et al. 2018.
- SimCLR - Contrasts augmented image views through a nonlinear projection head. Augmentation composition and batch size affect learning. ImageNet linear evaluation favors encoder features before the head over its output. Chen et al. 2020; code.
- Alignment and uniformity - Relates contrastive learning to positive-pair alignment and uniform spread on the unit sphere. The analysis is asymptotic; normalization alone does not guarantee uniform learned features. Wang and Isola 2020.
- BYOL - An online network predicts a moving-average target network's representation of another augmented view, without explicit negative pairs. Grill et al. 2020.
- DINO - Uses self-distillation without labels, a momentum teacher and multiple crops. Studies semantic structure in Transformer features through nearest-neighbor and linear classification. Caron et al. 2021.
- DINOv2 - Combines curated image data, scaled self-supervised training and distillation to produce reusable image-level and patch-level features. Oquab et al. 2024; code.
- DINOv3 - Uses Gram anchoring to address deterioration of dense features during extended self-supervised training. Studies scaling and post-training adaptation of visual representations. Siméoni et al. 2026; code.
- Barlow Twins - Drives cross-correlation between two augmented views toward identity, combining agreement with redundancy reduction. This objective also underlies TLDR. Zbontar et al. 2021.
- VICReg - Separates view agreement, per-feature variance control, and covariance regularization. Unlike Barlow Twins' cross-correlation objective, it explicitly constrains each branch's variance. Bardes et al. 2022; code.
- Self-predictive representations (SPR) - Predicts future encodings from features and actions as an auxiliary loss for Rainbow. Improves Atari 100k performance over a matched augmented Rainbow baseline without using the learned dynamics for planning. Schwarzer et al. 2021; code.
- I-JEPA - Predicts representations of masked image regions from visible context, rather than reconstructing pixels as in MAE. Assran et al. 2023.
- JEPA learning bias - In deep linear models, JEPA's feature-learning order depends on regression coefficients and input-target covariance; masked reconstruction is led by covariance. Theory assumes diagonal covariances, small aligned initialization and stop-gradient targets. Littwin et al. 2024.
- LeJEPA - Combines a joint-embedding predictive objective with sketched isotropic Gaussian regularization of embeddings. Balestriero and LeCun 2025.
- LeVJEPA - Extends LeJEPA to video using one encoder and 95% patch-token dropping. At matched pretraining FLOPs on 20% of K710, ViT-B improves ImageNet probing over V-JEPA 2 (61.0% vs 51.6%) but trails on Something-Something-v2 (40.4% vs 42.5%). Kuhn et al. 2026; code.
- JEPA predictive closure - In CIFAR-10 sequences with three-dimensional SIGReg latents and no stop-gradient, an unpredictable cue becomes more decodable when it helps predict other features. Altering predictive links shifts which feature sets are represented. Levy et al. 2026; code.
- V-JEPA 2.1 - Dense prediction over visible and masked tokens, plus intermediate-layer self-supervision, extends JEPA features to spatially structured image/video representations. Mur-Labadia et al. 2026; code.
- Spectral Ghost - Develops a spectral account of representation sufficiency connecting component analysis and self-supervised objectives. Dai et al. 2026.
- Cross-modal agreement - See Interpretability for the Platonic representation hypothesis, similarity calibration and JEPA's scene-pairing results.
- Limits of unsupervised disentanglement - Locatello et al. show that recovering independent factors requires inductive assumptions. Across seven datasets, stronger disentanglement does not consistently reduce downstream sample needs; factor recovery is a distinct goal. Locatello et al. 2019.
- Identifiable VAE / nonlinear ICA - Conditions a factorized latent prior on an observed auxiliary variable. Identifiability holds up to specified transformations under variation and regularity conditions; an ordinary VAE with an unconditional prior lacks this guarantee. Khemakhem et al. 2020.
- Visual representations - See self-distillation for DINO and predictive representations for JEPA. Transfer Learning covers adaptation.
- Text embeddings and retrieval - See NLP for sentence vectors, token scoring and document-context-aware chunk embeddings.
- wav2vec 2.0 - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. Baevski et al. 2020.
- BEATs - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. Chen et al. 2023; code.
- TS2Vec - Contrastive learning of contextual timestamp features, pooled for subsequences or whole series. Yue et al. 2022; code.
- MOMENT - Masked time-series pretraining supports classification, forecasting, imputation, and anomaly detection. Goswami et al. 2024; code.
- MantisV2 - Uses synthetic pretraining, layer selection and token aggregation for frozen time-series features. UCR ablations favor intermediate layers after more updates, but the final layer with 100K samples and 100 epochs; fine-tuning favors all layers. Feofanov et al. 2026; code.
- Joint representations - See Multi-View Learning for CLIP, CLAP, SigLIP 2 and multimodal retrieval.
- Matryoshka representation learning - Trains nested vector prefixes at multiple dimensions. Compression is built into representation training; arbitrary truncation of ordinary embeddings is not equivalent. Kusupati et al. 2022; code.
- TLDR - Compresses features using neighbor pairs and the Barlow Twins objective. On ROxford retrieval, the reported 128-dimensional codes gain 4.1 mAP points over whitened PCA with the tested GeM features and fitting data. Kalantidis et al. 2022; code.
- Retrieval dimension and margin - For unit-vector retrieval supporting every k-subset of n documents, optimal-order margin requires dimension of order k log(n/k). This worst-case relevance family does not determine the dimension needed for a particular corpus. Bangachev et al. 2026.
- Multivector expressivity - Proves a worst-case representation-size separation between Chamfer similarity and single-vector inner products; this is not an empirical retrieval ranking. Jayaram 2026.
- Neighborhood preservation - Trustworthiness penalizes introduced neighbors; continuity penalizes lost neighbors. These measure geometry rather than held-out task performance. Venna and Kaski 2006; docs.
- Intrinsic dimension - Global and local estimators characterize data geometry; their estimates are not task-optimized embedding dimensions. Bac et al. 2021; docs.
- Local versus global structure - On tested 2D layouts, t-SNE/art-SNE preserve neighborhoods better, while PCA/TriMap/PaCMAP/ForceAtlas2 preserve distance ranks better on most datasets. These visualization results do not rank general predictive representations. Huang et al. 2022.
- MTEB / MMTEB - Text-embedding evaluation across tasks, domains, and languages. Muennighoff et al. 2023, Enevoldsen et al. 2025; code.
- Prediction targets for control - Latent prediction is more robust than observation prediction to added Gaussian distractors in four MuJoCo tasks. Observation prediction instead improves aggregate learning on 20 sparse-reward, partially observed MiniGrid tasks. Ni et al. 2024.
- Python (reduction and dimension estimation) - scikit-learn, umap-learn, openTSNE, PaCMAP/LocalMAP, PHATE, scikit-dimension. docs, docs, docs, code, code, docs.
- Python (extraction) - Sentence Transformers, timm, Kymatio, tsfresh; evaluation: MTEB. docs, docs, docs, docs, code.
- R - dimRed, dyndimred, intrinsicDimension, Rdimtools; discriminant analysis: MASS, sda. You 2022; code, code, code, code, code, code.
- JavaScript (extraction) - Transformers.js runs compatible pretrained models through ONNX Runtime with WASM/WebGPU. docs.
- JavaScript (visualization) - tsne, umap-js, dimred; interactive tool: StatSim Vis. docs, docs, docs, docs.
- C++ - Tapkee. code.
- Prediction and control - MuZero predicts rewards, values and search policies. EfficientZero adds future-latent matching. V-JEPA 2-AC plans with image-goal distances after self-supervised training.
- Sparse autoencoders for interpretability - Overcomplete sparse features from internal model activations, evaluated for reconstruction and interpretability rather than compactness alone. Gao et al. 2024.