title: From Axes to Anchors:
Finite-Anchor Geometry for Text Embeddings
abstract: Representation spaces in language models are often interpreted geometrically, and
task-relevant information is often linearly recoverable. However, linear recoverabil-
ity does not imply that such information is organized along raw coordinate axes. We
therefore study a stronger and more relational diagnostic: whether task-label struc-
ture survives when embeddings are forced to be represented as convex mixtures
of a finite set of anchors. Instead of seeking global coordinate directions aligned
with label information, we fit an unsupervised finite-anchor convex decomposition
and analyze its anchor coefficients, convex reconstruction, and residuals. We find
that task-label structure selectively survives this constraint: topic datasets with
embedding-specialized representations often form compact anchor-relative regions
or low-dimensional faces, whereas finer-grained or multilabel emotion settings
show broader anchor support. Random-anchor controls indicate that this alignment
is not an artifact of arbitrary convex mixtures. We therefore position finite-anchor
convex decomposition as a diagnostic for locating task-label information under a
strong geometric constraint. We further show that anchor mixtures preserve rela-
tional label structure: coarse topics can be separable while remaining multi-facet
unions of finer category-level regions.
language of the presentation: English
|