how to sooth your gpu!
turboquant and the kv cache wall
—hiring—
browse the job posts, and if any look like a match for you, just reply to this and if relevant i’ll make an introduction
lora -pre-seed
new consumer app reimagining how people understand themselves and connect with the world
stealth - seed
building the ux layer for ai agents
—on a side note—
me and my friends hosted a world models hackathon this past weekend
a lot of incredible builders, showed up and built cool stuff
from explorable 3d experiences to rapidly generating robotic training data from a single screenshot
special shout out to world labs, soma capital, lovable
planning to host even bigger ones to meet more cool people
stay tuned!
ps. if anyone would like to sponsor a track in the upcoming hackathons, hmu!
the real villain: memory
if you’ve actually shipped anything with a big model,
you know the real bottleneck isn’t compute,
(if you’re a nvidia employee you better spend those tokens)
it’s your kv cache melting your hardware.
every message you send to a transformer isn’t just “new input.”
it’s the model dragging the entire conversation back onstage,
re-reading all the tokens it’s ever seen in that session.
to avoid recomputing everything from scratch,
it stores intermediate representations in this kv cache
ELI5 -basically short-term working memory.
and that thing grows linearly with context.
128k tokens on a 70b model? ~40 gb of gpu memory just for the cache.
at long contexts, the cache can consume more memory than the model weights.
this is why your “one gpu, many users” dream dies the moment you turn on long context.
this is also why a lot of long-context marketing is…
let’s say “aspirational.”
google enters the chat!
on march 25, google research drops turboquant.
just a paper and pseudocode.
headline:
compress the kv cache from 16 bits to 3 bits per value. ~6x less memory. up to 8x faster attention. basically no accuracy loss.
wall street sees “6x less memory” and goes full silicon valley parody:
sandisk: -11%
micron: -7%
sk hynix: -6.2%
samsung: -4.7%
nvidia: -4.2%
philadelphia semiconductor index: -4.8%
all while the nasdaq is up.
someone compared it to “aramco crashing because toyota made a better hybrid engine.”
which is a great line and also kind of wrong.
wtf is turboquant?
it compress almost as much as physics allows,
without the usual “whoops, we broke your model”
two main pieces:
polarquant
rotate the vectors so their distribution becomes predictable, then quantize in that nicer coordinate system.
translation: you precompute the quantizer once, you don’t need calibration, and you avoid wasting 1–2 bits on metadata like other methods.qjl
whatever error is left, you squash it down to basically a sign bit and correct bias so attention scores stay clean.
result: the compressed output is statistically identical to full precision.
anobody a fan of silicon valley the show? kind of like that
but not “we invented compression.”
rather
it’s “we got rid of overhead and crept up to the shannon limit.”
there’s a line in the paper: any quantization algorithm has a hard lower bound on error given the rate.
turboquant is already flirting with that boundary.
benchmarks that matter
the numbers:
memory: 6x reduction minimum. 70b kv cache from ~80 gb to ~13 gb in their setup.
speed: up to 8x faster attention on h100 (4-bit vs 32-bit).
accuracy: perfect needle-in-a-haystack retrieval, even at aggressive quantization.
cost: 50%+ cut in inference spend for long-context workloads.
compat: plug-and-play on llama, mistral, gemma, etc., no retraining.
caveats included:
8x speed is for attention kernels, not full-model latency.
most tests are on sub-10b models; 70b+ is still “we expect it to work, but to be proven.”
this is absurdly impactful.
hacker culture at it’s finest
google didn’t release official code.
the internet shrugged and shipped.
within 24 hours:
custom triton kernel in pytorch, tested on gemma 3 4b on an rtx 4090 with byte-identical outputs at 2-bit compression.
a 35b model running on apple silicon via mlx, hitting 6/6 needle-in-a-haystack at every quantization level.
multiple cuda and c implementations wired into llama.cpp, one reporting 18/18 tests passing.
someone used gpt-5.4 to write an mlx implementation in 25 minutes.
of course, there was a footgun: naive qjl implementations produced garbage. without proper bias correction, quantization errors explode and your model turns into a word salad generator.
you actually have to respect the math.
but the bigger signal is this:
a serious research paper can now go from “pdf” to “running in prod-ish environments” in a weekend.
if you’re shipping agent infrastructure, you can’t afford to be three pdfs behind.
hitting the compression ceiling
everyone’s obsessing over the 6x.
but……..
you probably won’t get 20x from another clever quantization trick.
rough compression journey so far:
no compression: 1x baseline
basic quantization: 2–3x
outlier-aware methods: 3–4x
turboquant in real systems: ~4–4.5x effective, with error rates near the theoretical minimum
the paper literally proves a mathematical lower bound: you cannot beat shannon. turboquant is hovering near that floor.
so:
if your strategy doc says “we’ll just keep compressing more every year,” that’s not a strategy,
the next big unlocks will have to come from new architectures, new attention schemes, or a different way of storing and retrieving context.
someone will say: “but we always find new tricks. look at hardware over the last 40 years.”
sure. but information theory is not vibes. at some point, you actually hit the wall.
turboquant is a giant “objects in mirror are closer than they appear” sticker on that wall.
the jevons paradox:
cheaper memory, more memory
this is where wall street got the story backwards.
on paper: 6x less memory needed per request.
in reality:
if you make memory effectively cheaper, people don’t use less
they use more.
jevons paradox 101:
save 6x on memory per session → cool, now you can afford models 6x bigger or context 6x longer.
suddenly, real-time video, million-token docs, always-on multimodal agents stop being “lol maybe someday” and become line items.
one gpu that could serve 1 heavy session now serves 6. demand will not politely stop at 6. it will grow to 60.
analysts are already saying the quiet part: turboquant-like tech lowers adoption friction and expands total demand. memory vendors likely win over a long enough horizon.
this is exactly what happened with cheaper training (deepseek vibes): everyone said “this will kill gpu demand.” instead, more people could train, so more people did.
if you build infra, this matters: your unit economics change, but your volume assumptions should too.
your hardware just got a upgrade
for indie builders.
no new chip. no new device. just software. and suddenly:
mac mini: 100k-token conversations with no quality drop—aka full book-length context on a ~$600 box.
smartphones: 32k+ token context windows via software alone. no hardware refresh.
rtx 4090: workloads that previously needed multi-gpu rigs can now squeeze onto a single card.
enterprise: for long-context stuff, you can cut the gpu count for a workload dramatically, potentially slashing cloud spend by 50%+.
the gap between “local ai” and “cloud-tethered ai” just shrank hard.
running serious models at home is slowly shifting from “cute hack” to “default assumption.”
if you’re building ai-native products and you’re still designing everything as if local is a toy, you’re about to get leapfrogged by someone who isn’t.
strip away the charts and stock tickers, and you’re left with a few uncomfortable-but-useful truths:
context is no longer the excuse
“we can’t afford long context” is a worse excuse post-turboquant. if your product relies on stateless prompts and short-term memory, that’s a design choice now, not a physics constraint.architectures, not just tricks, are the next frontier
you won’t be able to ride an endless wave of compression hacks. if you’re serious about long-horizon agents, you should care about new attention, retrieval, external memory, all the boring stuff.local-first ai is going from fringe to default
when a mac mini can handle book-length conversations and a 4090 can host models that used to need 2–4 gpus, you get a new class of indie infra. the stack shifts from “rent everything” to “own more than you think.”agent design has to assume cheap-ish, persistent context
the interesting agents won’t be prompt macros—they’ll be systems that keep rich, long-lived state and learn over time. turboquant doesn’t solve that for you, it just removes one major excuse.
someone will say: “yeah but it’s just one paper, and real workloads are messy.”
fair. but the community already has working implementations.
and even if this exact algorithm ends up with rough edges, it proves a point: compression is basically solved enough that the next constraints are on you, not the hardware.
to sum it up
turboquant isn’t just “6x memory savings.”
it’s the moment we quietly hit the compression ceiling and were told
ok, no more free lunches from quantization
if you want the next 10x, you’ll need to rethink how your systems actually work.
if you suddenly get 4–6x more context and throughput for free, what would you build that felt impossible last week—and what’s stopping you from starting on it now?