Hyperbolic Embeddings.
Meaning has layers ↘From broad intent to fine meaning, represented in Poincaré space.
A research implementation of hierarchical text representations using a frozen Transformer encoder, learnable token-attention pooling, an iterative HRM-style refinement module, and a radial schedule that maps different semantic levels into the Poincaré ball. Retrieval uses hyperbolic distance, with a cosine baseline for comparison.
The problem, and the approach.
What needed solving?
A single flat embedding must compress topical category and fine-grained meaning into the same geometry. This project investigates whether successive levels of refinement can represent semantic specificity explicitly, rather than treating every text item as one undifferentiated point.
What was built?
Encode token states with a frozen backbone, learn their pooled representation, refine across several segments, normalize and radially scale segment vectors, then exponential-map them into the Poincaré ball. Train across levels using contrastive hyperbolic NCE and compare retrieval against a backbone-only cosine baseline.
Architecture, without the noise.
This is a simplified conceptual diagram of the components documented in the repository; it is not a claim that all operations use a single synchronous pipeline.
Token hidden states from BGE/BERT.
Trainable attention weights aggregate token features.
Low-level and high-level recurrent updates over M segments.
Normalize vectors, apply segment radius, exp-map to hyperbolic ball.
Hyperbolic NCE + distance-based ranking versus cosine baseline.
How it works, step by step.
- 01
Encode text
Run a frozen Transformer backbone; token-level hidden states become the input to the trainable pooling layer.
models/hrm_encoder.py - 02
Pool important tokens
A ScoreNet attention pooler learns a weighted representation rather than using a fixed token choice.
models/hrm_encoder.py - 03
Refine coarse to fine
Alternating low-level and high-level MLP updates produce multiple segment-level tangent vectors.
models/hrm_encoder.py - 04
Map into curved space
Normalize and scale each tangent vector according to the radial schedule, then apply expmap0_poincare.
models/poincare_utils.py - 05
Train and evaluate
Use coarse-to-fine hyperbolic NCE, retrieve by Poincaré distance and compare rankings to a cosine baseline.
scripts/evaluate.py
The choices behind the system.
Freeze the backbone
Train refinement layers and pooling while retaining a reusable pretrained Transformer representation.
Encode hierarchy by radius
Later segments receive progressively larger radial scales, so geometry represents granularity by construction.
Keep a fair comparison path
The evaluator supports backbone-only cosine retrieval, avoiding a comparison solely against another custom hyperbolic method.
Inside the implementation.
Geometric design
Multi-level embeddings
Each text yields multiple points, one per refinement segment, rather than a single pooled embedding.
Hyperbolic geometry
The Poincaré ball uses negative curvature and distance functions implemented in models/poincare_utils.py.
Coarse-to-fine contrastive learning
Training optimizes a hyperbolic NCE objective with explicit negatives across segments.
Evaluation settings
The runner lists 11 retrieval tasks and Hits@K, Recall@K, NDCG@K, MRR@K and MAP@K.
Training configuration
A documented example from the README; actual optimization requires datasets, compute and the repository environment.
python scripts/train.py \
--backbone_name BAAI/bge-small-en-v1.5 \
--epochs 1 \
--batch_size 128 \
--num_segments 4 \
--num_negs 4 \
--temperature 0.05 \
--hyp_c 1.0Results in context.
Breadth of evaluation support
The repository documents an evaluator supporting 11 MTEB retrieval datasets, including FEVER, SciFact, NFCorpus, DBPedia, HotpotQA, Natural Questions, SCIDOCS, FiQA, CQADupstack, ClimateFEVER and Arguana. It does not publish a reproducible summary table proving improvement on all 11 tasks. The page therefore reports evaluation coverage, not benchmark superiority.
Numbers here are reported by the project author in the linked public repository or CV. They are not independent tests performed for this portfolio.
Trade-offs & open questions.
Supported does not mean won
Eleven supported tasks are a statement of implementation breadth, not a reported 11-task average or positive gain.
More computation
Multiple refinement segments and hyperbolic distance computations add training and serving complexity.
Geometry is a hypothesis
The radial schedule imposes a hierarchy prior that should be tested per corpus against normalized cosine embeddings.
Source directory.
These source paths are identified in the project implementation or accompanying documentation. Links open the corresponding GitHub source location; availability can change as repositories evolve.
Attention pooling and refinement stages
models/hrm_encoder.pyExponential mapping and distances
models/poincare_utils.pyHyperbolic NCE loss
training/losses.pyRanking scores
evaluation/retriever.pyHyperparameters and optimization
scripts/train.py