The differences were interesting. One provider rejected nearly half the requests on DeepSeek. Another had to process previously cached prompts again 165 times. And the fastest option wasn’t always the cheapest.

Here’s what we found.

How we tested providers

Instead of sending short, isolated prompts, we replayed real agent conversations from the public armand0e Claude Code dataset on Hugging Face.

These were long conversations with tool calls, tool results, and growing context windows. The median request contained approximately 40,000 tokens, making the workload more representative of agents working through complex tasks than a typical single-prompt benchmark.

We used eight recorded sessions, replaying assistant turns with their original tool results. Each provider received the same conversations and pauses.

We tested two models:

• DeepSeek V4.1 Flash: November 3, 2026, with 8,640 turns across six inference setups.
• GLM-5.3 Flash: November 4, 2026. We replayed the 8 sessions 3 times over 14 hours and report the third run, 1,440 turns in total.

The goal wasn’t to find the highest possible speed under ideal conditions. We wanted to see how providers behaved over extended agent workloads.

1. Reliability: most providers completed the workload, but not all

The first thing we measured was straightforward: how many turns received a complete response?

Most providers answered everything we sent on both models.

Baseten was the notable exception. On DeepSeek, it rejected approximately half the turns with rate-limit errors.

The requests were large, often containing tens of thousands of input tokens. Even with only two concurrent sessions, we exceeded the limits available to our account.

This doesn’t necessarily mean Baseten’s infrastructure couldn’t handle the workload. It means our account couldn’t run it at the tested rate without additional capacity or different limits.

On GLM-5.3 Flash, Baseten rejected fewer requests because the longer pauses reduced the load.

Modal had a different issue. One DeepSeek response ended before the stream completed. An application using that endpoint directly would need to detect the interruption and retry the request.

A single interrupted stream isn’t enough to establish a general reliability pattern, but it illustrates something worth testing: how applications recover when inference doesn’t complete normally.

Why it matters: For an agent, a failed request can interrupt a workflow involving many previous tool calls and accumulated context. Recovery behavior matters alongside the speed of successful responses.

2. Prompt caching: the differences were larger than expected

Prompt caching can make a significant difference to the cost and latency of long conversations.

When an agent sends its next request, much of the prompt may be identical to the previous one. If the provider retains those tokens in its cache, it doesn’t need to process them from scratch.

But caching becomes more complicated when inference runs across multiple providers or when a conversation sits idle.

On DeepSeek, the best direct provider reached a 95% cache hit rate.

Keln reached 94.3%, switching providers on just one of 1,440 turns. That’s notable because moving a conversation between providers can cause previously cached context to be lost.

Fireworks showed a different pattern. It had 165 cold turns, the most in the DeepSeek run.

A cold turn meant that much of the prompt had to be processed again rather than read from cache.

That affects cost even if the underlying model’s published input price looks competitive.

On GLM-5.3 Flash, Keln reached a 90.5% cache hit rate, the highest among the six tested setups.

These results show why comparing inference providers solely by published token prices can be misleading. The proportion of tokens actually served from cache changes what you pay.

3. What happens to caching when an agent goes idle?

Most benchmarks send requests continuously. Real agents don’t always work that way.

An agent might wait for a tool to finish, pause for user input, or sit idle before continuing the same conversation.

We wanted to understand how much cached context survives those breaks.

For GLM-5.3 Flash, we added pauses of 1–20 minutes to 15% of turns. Every provider received the same pauses at the same points in the conversations.

After a 15–20 minute break, the results differed considerably:

For agents with long context windows, this can become expensive.

Imagine an agent that spends several minutes running a tool and then returns to a conversation containing tens of thousands of tokens. If its cache has expired, those tokens may need to be processed again.

The impact depends on the model, pricing, and frequency of pauses. But cache retention is worth measuring if your workload involves long-running conversations.

4. Decode speed: two providers stood out

Decode speed measures how quickly a model generates output tokens once generation begins.

For GLM-5.3 Flash, Keln had the highest median decode speed at 432 tokens per second.

The next fastest was Baseten at 247 tokens per second.

On DeepSeek V4.1 Flash, Together AI and Keln were nearly tied at 437 and 432 tokens per second, both well ahead of the other tested providers.

These results are useful, but decode speed isn’t the same as total request latency.

For example, an agent sending a very large prompt may spend considerable time waiting before the first token arrives. Network overhead, queueing, and prompt processing also contribute to the total time.

And speed alone doesn’t tell you whether a provider can sustain the same performance as traffic increases.

For production systems, it’s worth measuring time to first token, decode speed, error rates, and performance under your expected concurrency.

5. Reasoning output was more consistent this time

Inference providers don’t always expose model reasoning in the same way.

Some support different reasoning settings, while others report reasoning tokens differently. This can make it difficult to compare outputs across platforms.

For this benchmark, we requested maximum reasoning effort on every turn, using the format supported by each API.

The providers returned broadly similar amounts of reasoning tokens.

That’s different from our previous benchmark, where two gateways returned less than half as many reasoning tokens for the same requests.

We didn’t observe that issue here.

There was one reporting limitation: Modal doesn’t report reasoning token counts for GLM-5.3 Flash, so we excluded it from that comparison.

Reasoning token counts aren’t a direct measure of answer quality, but large differences can be a useful signal when checking whether providers are applying the requested settings consistently.

6. The cheapest token price didn’t always mean the lowest cost

Cost is where the results become especially interesting.

Inference providers typically publish prices per million input and output tokens, sometimes with separate rates for cached input.

But the actual cost of a conversation depends on how many tokens are cached, how many need to be processed again, and how the provider bills for them.

On DeepSeek, Fireworks was the most expensive option in our benchmark.

The main reason wasn’t simply its published price. Its 165 cold turns meant it processed substantially more uncached input tokens.

On GLM-5.3 Flash, Keln was among the more expensive options.

Its per-token price includes managing capacity, failover, and retries across multiple providers, while direct providers sell access to their own infrastructure.

Neither approach is automatically better.

If you have predictable traffic, can manage provider limits, and don’t need infrastructure redundancy, going directly to a provider may be more economical.

If you’re running agents that need consistent performance across varying workloads, the operational overhead of managing failures, capacity, and caching becomes part of the cost calculation.

The useful metric isn’t always the price per million tokens. Sometimes it’s the cost of completing the entire task.

What we learned

The most interesting finding wasn’t which provider generated tokens fastest.

It was how differently platforms behaved when we ran the same long conversations through them.

Some delivered excellent throughput but encountered rate limits. Others completed requests reliably but retained less cached context. And differences in caching had a measurable effect on total cost.

Three things stood out.

First, test with your actual workload. A provider that performs well on short prompts may behave differently when requests contain 40,000 tokens and conversations continue for hours.

Second, measure caching over time. Cache hit rates during continuous requests don’t tell the whole story. For agents that pause between steps, cache retention can matter just as much.

Third, look beyond raw speed and token prices. Completed tasks, retries, rate limits, and cache misses all affect the experience and the final bill.

Direct inference providers can be an excellent choice, especially when you have the engineering resources to manage capacity and recovery yourself.

Managed inference platforms offer another approach by handling some of that complexity behind an API, usually with a different cost structure.

The right choice depends on your workload, concurrency requirements, and how much infrastructure you want to manage.

Hey 👋I’m Kris Holysheva, co-founder of Keln. We are publishing benchmarks on AI inference and open models. Follow me here to catch the next one, or read the full method in original post on keln.ai.