TL;DR: A new interactive visualization maps GPT-2's 32,070 tokens into a hyperbolic Poincaré ball, revealing the natural tree-like structure of its vocabulary.
Summary: This interactive tool visualizes GPT-2-small's raw token embeddings within a hyperbolic space, specifically a Poincaré ball. It allows users to explore the 32,070 tokens, which naturally form a forest-like structure, with a large main tree and numerous smaller family trees. The visualization uses Möbius translations for navigation, demonstrating how hyperbolic geometry is well-suited for embedding tree structures.
Why it matters: Understanding the geometric structure of token embeddings can offer insights into how LLMs process and relate language. This visualization provides a novel way to explore the internal organization of a foundational model's vocabulary, potentially inspiring new approaches to embedding analysis or model interpretability.
Source: reddit