Kog is trying to squeeze more inference out of standard GPUs

French startup Kog is betting there’s a lot more performance to be squeezed out of the GPUs enterprises already own. In May, the company posted a tech preview claiming extremely fast single-request decoding on standard datacenter GPUs, using AMD MI300X and Nvidia H200 chips, and landed on the front page of Hacker News in the process. CEO Gaël Delalleau told TechCrunch the preview generated 200 tangible business leads.

Delalleau expects software engineering to be the first real use case: Claude Code users sometimes wait hours for results, and Anthropic charges a price multiple for its Fast Mode, a sign that speed itself has value. Kog also has design partners letting users generate games and apps from a prompt, where faster output means more revenue for them. But the market isn’t mature yet — Kog found its prospective customers weren’t prepared to fine-tune small models, so the company has focused entirely on accelerating larger models to meet the demand it has seen since launch.

That’s a big leap from Kog’s demo, which reached 3,000 tokens per second per request — but on a small, purpose-built model of around 2 billion parameters, the now-open-sourced Laneformer 2B, far short of the 30x faster LLM inference Kog is promising. Delalleau insists the same approach can scale to large language models: newer GPUs have growing memory bandwidth that mostly goes unused, he said, dismissing the idea that GPUs are ill-suited to decoding as a misconception. France’s ZML has taken a different path, building hardware-agnostic software that bypasses Nvidia’s CUDA; Delalleau compares Kog’s work instead to Stanford lab Hazy Research, with an even deeper focus on GPU-level acceleration.

That focus traces back to Delalleau’s background: he studied solid-state physics at France’s École Polytechnique before moving into offensive cybersecurity, becoming a four-time finalist at DEFCON’s CTF hacking tournament, which he says taught him “to reverse-engineer things at a very low level — down to assembly language and binary code — to understand how it works, and to try to use it to achieve a goal for which it wasn’t necessarily designed.” The approach is slow — Kog spends several weeks or even months digging into each new GPU, which limits how many chips its 11-person team can support for now. Delalleau expects to hit 10x speed on a first major model by September, which he says would let Kog demonstrate customer traction and raise a Series A.

Why it matters

If Kog can prove its approach works on full-size language models, it could offer enterprises a cheaper path to faster AI inference than buying specialized chips — a bet that will live or die on results Delalleau says are coming by September.