Limits of the Transformer Architecture and a QCD-like Alternative
The transformer architecture has no physics below the token scale. You cannot ask "what is the next character" if you trained on subword units — the question is literally undefined.
More issues
The Transformer as Renormalization Group Flow
The forward pass through a transformer implements a Kadanoff-Wilson renormalization group flow, coarse-graining microscopic token representations into stable semantic attractors.
A Stationary Action is Stable Information
There is a maximum amount of information that any observer can extract from a physical system. This limit emerges from the structure of phase space itself. A system's state occupies a region defined by its position and momentum, and quantum mechanics forbids this region from shrinking below a
The Hard Problem as Hidden Relationality
There is a pattern in the history of physics. A puzzle appears insoluble for generations. Brilliant minds propose baroque solutions — new entities, hidden mechanisms, radical revisions to metaphysics. Then someone notices that the puzzle rested on an unexamined assumption. Remove the assumption, and the puzzle doesn't get solved.
It from Bit, Bit from It
If you have ever felt uneasy reading about quantum mechanics, you are in good company. For nearly a century, the idea that a cat can be both alive and dead — or that an electron exists in a cloud of probability until "looked at" — has struck even world-class physicists
The Synchronization Tax
A physicist, a computer scientist, and a banker walk into a bar. They cannot agree on who got there first.
This is not a joke. This is a central problem of physics, computer science, and economics.
The physicist points out that simultaneity is relative — who was there "first"
Estimation Culture
Reason recently published a lovely article by Davidson Heath about the history of statistical significance and estimation. I enjoyed the article very much, but I'm afraid that even for people who use the tools described in the article in their day-to-day work, the message might be a little
The Causal Arrow of Altruism
I recently explored group selection and kin selection models in an essay on Radamacher Complexity and Group Competition, attempting to reconcile seemingly irreconcilable differences using measures of model Rademacher Complexity. The essay chose to use the definition of group selection provided by Nowak, Tarnita, and Wilson (NTW) 2010, which also
Rademacher Complexity and Models of Group Competition
What do Radamacher Complexity and mean field approximations have to do with kin and group selection models in evolutionary biology? And what might consideration of the Radamacher Complexity of kin and group selection models suggest to us about cooperation and competition in human groups? Here are tentative answers to what
The Coasean Singularity in Patents
Patent rights exist to solve a fundamental economic problem: Inventors need incentives to innovate, but after revealing an invention, others can copy freely. The patent system grants limited monopolies to inventors in exchange for public disclosure. This is the patent system as God intended. But the patent system has struggled