The Last Profiler

Unzeitgemäße Betrachtungen

Now Written In Rust

LiteLLM has a page on its docs site called "LiteLLM is moving to Rust." It has a progress bar. 39% of Major APIs built, 47% planned by November 14, a dashed line heading toward a milestone on December 1. A progress bar for a language, as if Rust were a destination you reach by percentages.

Under the bar are the benchmarks. July's forwarding-only test, against a mock upstream with no logging callbacks, spend tracking, or persistence enabled. Added latency at p99: Rust 0.7 ms, Portkey 2.3, Bifrost 4.5, LiteLLM Python 257.7. Four gateways, one chart, one loser. The loser is the company's own product from last year. The company is not worried about this.

I.

You're going to say Python is slow. You know it's slow, you've always known, everyone knows, it's the thing you say at parties so people think you've worked on something large. Here's what's going on. They profiled one chat completion on one pinned core. Python spent 33.7 ms on it with a cached key. Rust spent 1.7. Twenty to one. Now look at what changed before you decide what caused it. The same post, in the middle of the Rust-is-fast victory lap, lists what they actually did: kept router cooldowns in an in-memory cache, kept auth lookups in an in-process cache, ran admission hooks in parallel instead of serially, and stopped reserializing the request body on passthrough.

Read that list again. Which of those is a language feature?

They also showed you how much memory a 2.45 MB request body takes in the Python version: four copies. Parsed, raw, json.dumps'd for logging, outbound bytes. 10.95 MB, for one stream. In the Rust version they kept the body once, or not at all. The benchmark doesn't show you a Python version that holds the body once. It shows you four copies losing to one, with the languages helpfully labeled.

There's a GitHub issue, #19921, from a user who traced 100% CPU to a bug: a new Router constructed on every request, which then walked the entire Prometheus registry on every request to check whether a metric existed. A bug. Not a language. It sat open for months. LiteLLM's own docs say DEBUG logging is "the #1 cause of latency with large payloads." Their own docs. Those are implementation problems, not measurements of a language limit.

And then there's the line in the November post, the revealing one: they were "very clear" that the earlier benchmarks were meant "to validate how much better Rust could do than Python as a language."

Validate. Not compare the implementations. Not isolate the bottleneck. Validate how much better Rust could do. The conclusion already has a reserved seat; the benchmark is checking its ticket.

II.

"Yeah, but Bifrost is written in Go, and it's close."

It is close. In November's separate test at 50 requests per second, with common proxy features enabled, Bifrost is at 1.2 ms median, 4.2 p99. Rust is at 0.9 and 1.8. Portkey, running on Node.js, is 2.1 and 5.3. LiteLLM labels Portkey "no features": its open-source gateway lacks the auth, keys, limits, budgets, logs, and metrics used in this test, though it still load-balances and retries. A garbage-collected language, written by people who designed the thing around performance, is within spitting distance of the Rust version. That's the finding. The finding isn't "Rust is fast." The finding is "design is fast," and the benchmark can't tell the two apart, because it doesn't include the control: LiteLLM's new architecture in Python.

Their own October post, as I read it, says that porting just the router and translation to Rust gets you from about 48 ms to about 32. They say that even if other parts get 10 to 50 times faster, you'd see roughly a 2x gain overall. The 1.7 ms number is what you get when you throw out the whole request path and rebuild it with a caching and hook model designed around the problem. Both changed. The chart doesn't tell you how to divide the credit. The caption has no such hesitation.

If you've ever read one of these before you know how this goes. Discord, 2020: Go service spiking every two minutes, rewritten in Rust, no more spikes. The footnote: "we don't think you should rewrite everything in rust just because." Also they said they'd only put "very basic thought into optimization" on the Rust version, which is the sentence you remember. A separate footnote says the graphs used Go 1.9.2; they also tried Go 1.8, 1.9, and 1.10 without improvement. Cloudflare, Pingora: 70% less CPU, and then, in the same post, the sentence that nobody quotes: "This is not because we run code faster... The savings come from our new architecture which can share connections across all threads."

Grab ran a more direct comparison, a Go service rewritten in Rust, same shape, and wrote: "Myth 1: Rust is blazingly fast! ... Verdict: Disproved." CPU savings, yes. Latency, no. "Rewriting a system solely for the purpose of 'rewriting it in Rust' is not a strong enough business justification."

Read past the headline and the language story gets complications. The complications are where the engineering is.

III.

So why does the page exist.

Not necessarily to deceive you. You don't need a dishonest engineer for this. You need an organization in which "we need six months to clean up" sounds like failure and "we're moving to Rust" sounds like a strategy. Same maintenance budget. Different story. One gets a progress bar. I don't know what happened in their planning meetings. I know which pitch I'd rather have to make.

And you, reading the progress bar, you're not skeptical, you're relieved. Because if the answer is Rust, then the answer is a thing you can go and learn. It isn't "make the thing you already own better, carefully, with measurements, for no applause." "We Finally Profiled Our Python and Fixed Four Bugs" is a harder story to sell. The reader might ask why you hadn't done it already. "We're moving to Rust" lets everyone look forward instead.

What do you want from the chart? You want to be the person who picked the fast thing. You want the next system you build to carry a number like that on the front.

To be fair to Rust: 357 MiB against 609 for Portkey and 2.8 GB for Bifrost is real. 0.15 cores against 0.41 is real. No GC, one process, no fork-and-duplicate. Some of that is a language advantage, some is a process-model choice. Either way, it will save somebody a pod bill. It's also not the part on the poster.

Rust can be the right engineering decision and the wrong explanation of the benchmark.

The poster says 257.7 ms. It doesn't tell you how much of that number belonged to Python and how much belonged to the way they wrote it.

You can ship the faster gateway without answering that question. You cannot use the speedup to answer it.

The progress bar says 39%. It doesn't say 39% of what the Python version would have done with the same architectural changes.