Amazon tells its engineers that token counts are a deeply flawed way to measure AI work

Amazon has told its software engineers something that many companies racing to adopt AI have been reluctant to say out loud: the number of tokens you burn says very little about how good your work is. In a revised role guideline for software development engineers and their managers, published on the company's About Amazon site on October 27, 2026, engineering leadership calls token usage, AI adoption rates and counts of code changes and reviews "deeply flawed" when they are used as targets.

The memo goes further than a general warning about metrics. It names the behavior directly. Chasing token counts, it says, "is at odds with frugality", one of Amazon's long standing leadership principles, and the company uses the internet slang for it: tokenmaxxing. At the same time Amazon makes clear that it is not stepping back from AI. "We are certain that whatever future best practices are, AI will be at the center of software engineering practice," the leaders write.

According to CNBC, the memo circulated internally in September before Amazon posted it publicly. An Amazon spokesperson told CNBC that the company has not put caps on employee token usage.

What the memo actually says

The public version is introduced by Terk Terkowitz, a VP and Distinguished Engineer for Worldwide Amazon Stores. He explains that Amazon's Distinguished and Senior Principal Engineers came together to rewrite what it means to be a software development engineer at the company, based on what they saw in their most successful engineers. Their conclusion is that "neither the developer role nor their output is defined by the use of AI."

The memo itself, addressed to "SDEs and SDMs", starts with a question the authors say they debated at length: how much should a role guideline focus on AI? For two decades Amazon kept these guidelines free of specific technology, even as the cloud, microservices and serverless reshaped engineering work. The senior engineers argued about whether this time is different, and they agree that AI is changing software engineering faster and more deeply than those earlier shifts.

Yet they still chose not to write AI practices into the guideline. Their reasoning is practical. Any definition of good AI engineering written today, they say, will probably look outdated within a year, and an outdated rule risks "freezing our engineering practices in amber while the world moves on without us." So instead of describing tools, the guideline focuses on what does not change: impact on customers, on the business and on the team.

That choice leaves an obvious question, which the memo addresses head on: if the guideline does not say how to use AI, how should teams measure whether they use it well?

Why token counts are a trap

The honest answer in the memo is that the best measure, long term outcomes, is also the hardest one. Outcomes arrive late, are difficult to measure and are hard to separate from other trends. That is why many teams, Amazon included, fell back on input metrics like token usage and AI tool adoption, or throughput metrics like the number of code changes and reviews.

The authors do not mince words about these numbers. "We are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters," they write. Used as a target instead of a measure, they warn, these numbers can push teams toward exactly the wrong behavior. Two examples stand out:

• Rewarding token usage drives up costs, which is why the memo says tokenmaxxing is at odds with frugality.
• Rewarding lines of code drives up complexity, which the memo says is at odds with another leadership principle, invent and simplify.

This is a textbook case of Goodhart's law, the old observation that a measure stops being useful once it becomes a target. If engineers know their AI usage is tracked and compared, the cheapest way to look good is to use more AI, not to build better software. A model can generate thousands of lines of code or run long agent sessions that look impressive on a dashboard and add little value for any customer.

A leaderboard that backfired

Amazon has firsthand experience with this. In May 2026, Business Insider and the Financial Times reported that the company shut down an internal leaderboard called KiroRank. It ranked employees by their activity with Kiro, Amazon's AI coding tool, and some staff reportedly started running agents on tasks that solved no real problem just to climb the ranks.

Amazon said at the time that KiroRank was an informal tracker built by a group of employees and was never meant to promote AI use for its own sake. A senior vice president, Dave Treadwell, put the lesson plainly to staff: "Please don't use AI just for the sake of using AI." The new guideline turns that remark into written policy for the whole engineering organization.

Not a retreat from AI

It would be easy to read the memo as Amazon cooling on AI. It is not. The document repeatedly tells engineers to learn faster. "If you haven't already, you should take the time to learn and apply AI into your daily work as soon as possible," it says. Engineers already using AI should "keep investing in finding better practice."

The memo also keeps the input metrics on the table, just with a warning label. Amazon says it has found that token usage and similar numbers "correlate well with the outputs that matter" and expects them to keep doing so for a while. Teams that embraced AI the most, which Amazon calls frontier teams, deliver more, and teams or individuals who do not adopt AI or raise their output are "falling behind." Using the metrics "with care and judgement" when thinking about careers and team performance is, in the memo's words, appropriate.

So the message is not that tokens are irrelevant. It is that tokens are a clue, not a score. The difference matters. A clue tells a manager where to look. A score tells an engineer what to optimize.

Look at the edges, not the average

One of the more concrete recommendations is about where to look. The memo asks leaders to pay special attention to two ends of the AI adoption curve:

• The 10th percentile: teams that are not using AI effectively. The goal is to understand why, not to punish them.
• The 90th percentile: teams that use AI heavily. Here the question is whether their delivery is actually rising to match their usage.

This is a sensible way to use a noisy metric. Averages hide the stories that matter. The low end may reveal teams with legacy systems, strict compliance work or tasks where today's models simply do not help. The high end may reveal real productivity gains, or expensive habits that only look like progress.

The memo borrows a line from Jeff Bezos's 2016 shareholder letter about understanding customers through many anecdotes rather than averages, and rewrites it for engineering leaders: good leaders deeply understand their team's work, study many anecdotes and carefully track outputs, rather than only surface level metrics.

"A finding, not a failing"

The line that has drawn the most attention is short. "If your work does not benefit from current AI tooling, that is a finding, not a failing." The memo tells engineers in that situation to document why, share it with peers, bring it to their manager and work on solutions. "An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for."

For engineers at a company that has pushed AI hard, this is a meaningful shift in tone. It gives them explicit permission to say that a tool does not fit a task, without that being read as resistance. It also turns those cases into useful information. Knowing where AI fails is valuable for a company that sells AI tools to other developers.

Managers do not get off easily either. The memo expects technical and organizational leaders to use the tools themselves on production work. Its bar for "sufficient understanding" is specific: a leader should be able to describe a task where the tools helped, a task where they did not, and explain that judgement to the team. Relying on what you hear from others is not enough.

What Amazon wants to measure instead

The memo stays fairly abstract about better metrics and admits that "working together, over time, we will find better metrics and set smarter goals." Marc Brooker, a vice president and distinguished engineer at AWS, gave CNBC a more concrete hint. He said Amazon has looked at "feature velocity" to judge how AI changes engineer productivity, meaning things like how many launches teams ship and how many successful changes they make in production.

Brooker framed AI spending as a question of return on investment. When there is an important problem for customers or the business, Amazon will invest "tools, time, tokens, human effort" into it. What it wants to avoid is "spending those tokens solving problems that don't matter," while still leaving room for experimentation.

The memo adds one more point that is easy to miss. It says engineers and managers who are close to their products and customers get the most out of AI and agentic software development, and admits that as Amazon grew, some of its engineers became less connected to their customers and business. It asks engineers at every level to reverse that. The idea is simple: AI makes it cheaper to build things, so knowing what is worth building becomes more important.

Why this matters beyond Amazon

Amazon is not the only company wrestling with this. As AI coding tools spread, token bills have become a real line item, and companies have tried different fixes. Some cap usage, some switch parts of their work to cheaper open models, and some, like Amazon earlier this year, have learned that internal rankings can create the very waste they are meant to prevent. Tom's Hardware notes that other large companies have also said publicly that higher token usage has not yet translated into clearly better products.

Amazon's approach is notable because of who it comes from. This is a company that sells AI infrastructure, models and coding agents to developers around the world. It has every commercial reason to celebrate token consumption. Instead, its senior engineers are telling their own staff that more usage is not automatically better, and that judgement matters more than volume. Terkowitz says Amazon is sharing the guideline publicly because it believes its vision for the developer role "can serve as a model for our customers."

For other engineering leaders, the practical takeaways are straightforward:

• Do not turn token counts or AI adoption rates into individual targets or leaderboards.
• Use those numbers to find teams worth a closer look, especially at the low and high ends.
• Measure what ships and what works in production, not how much code was generated.
• Treat honest reports about where AI does not help as useful data.
• Expect leaders to use the tools themselves before they judge how others use them.

The bottom line

Amazon's revised guideline does not pull back from AI. It pushes engineers to learn it faster and calls it the center of future software engineering. What it rejects is the idea that AI usage is a goal in itself. Tokens, adoption rates and lines of code are inputs. Customers only feel the outcomes. By writing that distinction into the role definition for thousands of engineers, and by publishing it, Amazon has put a clear marker down in the debate over how to measure AI at work.

Sources: About Amazon, "An update from Amazon engineering leadership on the role of the developer in the age of AI", CNBC, Business Insider, Tom's Hardware.