DeepSeek open sources DeepInfer, the engine behind V4.1 Flash, for Nvidia and Huawei chips

Most of the attention in AI goes to models: who trained the biggest one, who tops which benchmark, who released weights. The quieter half of the business is inference, the work of actually running a trained model every time someone sends a request. The same model can be cheap or expensive to operate depending on the software that serves it, and labs usually guard that software closely. On 20 October 2026 DeepSeek did the opposite. It published DeepInfer, the inference framework that runs its DeepSeek V4.1 Flash model in production, on GitHub under the Apache 2.0 license.

The announcement on X described it as the company's "in house inference framework, now fully open source", shared "to help the community lower compute costs, reduce energy use, and make AI more affordable." The repository picked up more than 8,000 stars by Tuesday afternoon, according to The Next Web. When we checked the GitHub API on the evening of 21 October, Berlin time, it showed about 17,500 stars and roughly 1,800 forks, which is a lot of attention for a piece of infrastructure code.

This article looks at what is in the release, what DeepSeek kept back, what its own benchmark charts show and why an open inference engine matters more than it first sounds. Sources are the DeepInfer repository on GitHub, reporting by The Next Web, which also cites Bloomberg, and The Decoder. All performance figures are DeepSeek's own. Nobody outside the company has tested them independently yet.

What inference software actually does

When a model like V4.1 Flash answers a question, two things happen. First the system reads the whole prompt, a phase called prefill. Then it generates the answer one token at a time, a phase called decode. Prefill is heavy on raw compute because it processes thousands of tokens at once. Decode is limited mostly by memory, because for every new token the model has to look back at everything it has already seen.

An inference engine decides how to spread that work across many chips, how to group requests from different users into batches, how to keep the GPUs busy while the CPU schedules the next batch, and how to squeeze extra tokens out of each step. Good decisions here can change the cost per answer by several times without touching the model itself. That is why companies that sell model access through an API often treat their serving stack as a trade secret. It is the part of the business that turns a research artifact into a product with a margin.

DeepSeek V4.1 Flash is a mixture of experts model. Instead of using all of its parameters for every token, it routes each token to a small set of specialised expert networks. That saves compute, but it creates a coordination problem: experts live on different chips, and tokens have to be shipped to the right place and back very quickly. A large share of DeepInfer is about solving exactly that.

What is in the release

According to the README, DeepInfer is built for offline inference with the core pieces needed to serve at scale. It supports one model for now, DeepSeek V4.1 Flash, with the checkpoint DeepSeek V4.1 Flash 1017 from Hugging Face as the default. We covered that update in our article on DeepSeek V4.1 Flash 1017 and its open weights.

The main building blocks listed on the GitHub page are:

• Optimised kernels. DeepInfer pulls in DeepSeek's earlier open source kernel projects, FlashMLA for attention, DeepGEMM for matrix math and DeepEP for expert communication, plus TileKernels, DeepSelect and custom kernels written in TileLang.
• Parallelism. Attention can be split by data parallelism or balanced context parallelism, and the experts are spread with expert parallelism and an expert load balancer, so no single chip becomes the bottleneck.
• Draft tokens with smarter checking. A feature DeepSeek calls DSpark drafts several tokens ahead and then verifies them, using confidence estimates to decide when to check. This is a form of speculative decoding, a well known trick to produce more than one token per model step.
• Bounded replay for sliding window attention. The decoder only replays a short suffix of the prompt instead of all of it, which cuts prefill work. The default for V4.1 is 140 tokens.
• Asynchronous scheduling. The CPU prepares the next batch while the accelerator is still working on the current one, so the chips do not sit idle.
• Batch invariant and deterministic output. On Nvidia hardware, the engine can return identical results no matter which other requests share the batch, and identical outputs for identical inputs with a fixed seed.
• Structured outputs. Generation can be forced to match a regular expression, a JSON schema or a grammar, which matters for agents and tools that need machine readable answers.

DeepSeek also released deepseek recipe, a companion project that reproduces how its official API turns text into tokens and back, so a self hosted setup behaves like the hosted one. The README credits vLLM, the widely used open source serving engine, as the main foundation DeepInfer was originally built on, with inspiration from SGLang and Nvidia's TensorRT LLM. The vLLM team thanked DeepSeek on X and said it would use ideas from DeepInfer to improve its own support for V4.1 Flash, as The Next Web reported.

What DeepSeek kept back

The release is honest about its limits. The README lists three things that a complete inference service needs and that are not included: a KV cache system, the split of prefill and decode onto separate groups of machines, and load balancing across data parallel replicas. It notes that most of the engine functionality for that split already exists in the code.

These missing parts are not small. A KV cache system stores the intermediate results for prompts the system has already seen, which is what makes repeated long contexts cheap. Disaggregating prefill and decode lets an operator size each pool of hardware for its own bottleneck. Load balancing keeps thousands of requests flowing evenly. Together they are a large part of what separates a fast engine from a profitable service. So DeepInfer is best read as the engine block, not the whole car. Anyone who wants to match DeepSeek's production costs still has real engineering work ahead.

There is also the hardware requirement. DeepInfer targets Nvidia SM100 GPUs, the Blackwell data center generation, and Huawei Ascend 950 NPUs. It needs Linux, Python 3.12 and either CUDA 13 or Huawei's CANN 9.2. This is not a tool for a gaming PC. It is aimed at cloud providers, research labs and companies that already run serious GPU clusters.

What the benchmark charts show

DeepSeek published two benchmark charts with the release. Both were measured by DeepSeek on its own hardware, prefill on 4 SM100 GPUs and decode on 64 SM100 GPUs, each within one scale up domain. The values below are our approximate readings from those charts, rounded, not exact numbers published as a table.

The decode chart shows the classic tradeoff of serving: the more requests you pack onto each GPU, the more total tokens you produce, but the slower each individual user sees the answer appear.

At 8 requests per rank, each user gets about 325 tokens per second, which is very fast, but each GPU only produces around 2,600 tokens per second in total. At 256 requests per rank, the GPU produces about 26,300 tokens per second, roughly ten times more, while each user still gets about 103 tokens per second. That is still faster than anyone can read. As a rough cross check, our own calculation: 256 requests on each of 64 ranks is 16,384 streams, and at about 103 tokens per second each that adds up to around 1.7 million tokens per second for the whole 64 GPU setup, which matches 26,300 per GPU times 64.

The chart notes that these runs used three draft tokens per step with a per token acceptance rate of 75 percent, which DeepSeek translates into an average of 3.25 tokens per decoding step. In plain terms, the speculative decoding trick more than triples the output of each expensive model pass, if the acceptance rate holds on real traffic.

The prefill chart measures how fast the engine reads prompts that are not already cached.

With batches of 4,096 tokens per rank, each GPU processes around 60,000 input tokens per second at about 68 milliseconds per step. Bigger batches push that to roughly 82,000 at 16,384 tokens and close to 90,000 at 65,536 tokens, but the step time grows to over 700 milliseconds. The curve flattens quickly, which suggests the GPUs are close to saturated beyond about 16,000 tokens per batch. Both charts were run with a context length of 131,072 tokens.

Two cautions belong next to these numbers. First, they come from the vendor, on its own setup, and Bloomberg noted there is no independent benchmarking yet. Second, they are throughput charts, not cost charts. They do not tell you the price per million tokens, which depends on hardware prices, utilisation and the missing serving pieces. The repository does include a sweep script so others can try to reproduce the results, and an evaluation suite for AIME 2026, Humanity's Last Exam and GPQA Diamond to check that answers stay correct under the engine.

Nvidia and Huawei, side by side

The most strategic detail in the release may be the hardware list. Huawei's Ascend 950 is a first class target, right next to Nvidia's newest data center GPUs, with its own build path and requirements file. The Next Web points out that earlier this month DeepSeek also published tools for Huawei chips as an alternative to Nvidia's CUDA software ecosystem.

For Chinese AI labs, depending on Nvidia is a business and policy risk because of US export controls. A production grade engine that treats Ascend as a peer makes it easier for Chinese clouds to run frontier class models on domestic chips. It also fits a wider trend in which big players try to loosen CUDA's grip from different directions. Only days ago Amazon opened its own native PyTorch backend for Trainium, which we covered in Amazon TorchNeuron brings native PyTorch to Trainium. Google made a similar move with TorchTPU at the same PyTorch conference.

Some features still lean on Nvidia. Batch invariant inference and deterministic sampling are marked as Nvidia only in the README, so the two platforms are not fully equal yet.

The timing and the politics

The release lands in a tense week for DeepSeek in the United States. On 19 and 20 October, an order to remove DeepSeek products from the systems of civilian federal agencies was made public and then published in the Federal Register, which we explained in US bans DeepSeek from federal civilian agencies. That order concerns DeepSeek apps and services used inside the government. An Apache licensed engine on GitHub is a different thing: anyone can read the code, audit it and run it on their own machines without sending data to DeepSeek.

The Decoder argues that the move also adds pricing pressure on Western providers such as OpenAI and Anthropic, and that given the AI race between China and the US it could be politically motivated as well. DeepSeek's models are already among the cheapest to use through an API. Giving away the serving know how makes it easier for every host in the world to undercut closed models on price, at least for workloads where an open model is good enough. According to The Next Web, open weight models already account for 23 percent of spending on Vercel's AI Gateway.

Who gains from this

The most direct winners are cloud and hosting companies that already serve DeepSeek models. They can now use the same techniques DeepSeek uses in its own production stack, as Bloomberg reported. Companies that run V4.1 Flash on their own GPUs for privacy or compliance reasons could also lower their cost per token. The open source serving projects gain too: vLLM has already said it will borrow ideas, and improvements tend to spread from there to the many tools built on top of it.

For developers who never touch a GPU cluster, the effect is indirect but real. More efficient serving of a capable open model tends to show up as lower API prices across many providers, and as more choice of where to run it.

Why it matters

Open weights let you run a model. Open inference code lets you run it the way its makers do. Those are two different levels of openness, and the second one has been rare. Meta, Mistral and others publish weights, but the production serving stacks of the big labs mostly stay private. DeepSeek has now published a large slice of its own, including the scheduling, the expert communication and the speculative decoding scheme, and it says code for future models will land in the same repository.

That raises the floor for the whole open ecosystem. Even if DeepInfer itself is used by only a few dozen large operators, its ideas will flow into vLLM, SGLang and similar projects that power a large share of open model serving. It also shifts competition. If serving efficiency stops being a secret, labs have to compete more on model quality, safety, trust and product, and less on who has the cleverest private batch scheduler.

Dany's take

I like this release for a simple reason: it is useful and it is honest about what it is not. DeepSeek says clearly that the KV cache system, the prefill and decode split and the load balancing are missing, and it labels its benchmarks as its own. That is more transparency than many launches offer. The numbers are impressive, especially the roughly 3.25 tokens per step from speculative decoding, but I want to see someone outside DeepSeek reproduce them before treating them as fact.

The bigger story for me is the hardware line. An engine that runs the same model on Nvidia Blackwell and Huawei Ascend with one codebase is a quiet signal that the CUDA moat is getting shallower, and that the next price war in AI may be fought in serving software rather than in model size. If you run open models in production, this repository is worth a weekend of reading even if you never deploy it.

Sources

• DeepInfer repository and README on GitHub (DeepSeek, 20 October 2026)
• The Next Web: DeepSeek open sources DeepInfer, the engine that runs its AI models (20 October 2026, includes Bloomberg reporting)
• The Decoder: DeepSeek open sources the inference engine that powers its fastest and cheapest AI model (20 October 2026)