Limits of the Transformer Architecture and a QCD-like Alternative
The transformer architecture has no physics below the token scale. You cannot ask "what is the next character" if you trained on subword units — the question is literally undefined.
Topic
Science
A collection of 6 issues
The Transformer as Renormalization Group Flow
The forward pass through a transformer implements a Kadanoff-Wilson renormalization group flow, coarse-graining microscopic token representations into stable semantic attractors.
A Stationary Action is Stable Information
There is a maximum amount of information that any observer can extract from a physical system. This limit emerges from the structure of phase space itself. A system's state occupies a region defined by its position and momentum, and quantum mechanics forbids this region from shrinking below a
The Hard Problem as Hidden Relationality
There is a pattern in the history of physics. A puzzle appears insoluble for generations. Brilliant minds propose baroque solutions — new entities, hidden mechanisms, radical revisions to metaphysics. Then someone notices that the puzzle rested on an unexamined assumption. Remove the assumption, and the puzzle doesn't get solved.
It from Bit, Bit from It
If you have ever felt uneasy reading about quantum mechanics, you are in good company. For nearly a century, the idea that a cat can be both alive and dead — or that an electron exists in a cloud of probability until "looked at" — has struck even world-class physicists
The Synchronization Tax
A physicist, a computer scientist, and a banker walk into a bar. They cannot agree on who got there first.
This is not a joke. This is a central problem of physics, computer science, and economics.
The physicist points out that simultaneity is relative — who was there "first"