Software quality metrics at the scale of 45% of the web: Lance Willett of Automattic

Most companies treat quality as a feeling. Lance Willett's argument is that software quality metrics should carry targets the way finance carries targets for revenue: quarterly, annually, benchmarked against the industry, and reported to the board.

Willett is Chief Quality Officer at Automattic, the company behind WordPress and WooCommerce. WordPress powers close to 45% of the top hundred million websites. He and Michael go back nearly two decades, to Automattic's early days.

From a climbing store to WordPress

Willett was building websites as a hobby while working mail order at a mountaineering store in Tucson, shipping carabiners to people heading up Half Dome, and repeatedly failing to get onto the web team.

He left, started his own company, and one of his first clients was that same outdoor store, where he redesigned their commerce site. He ran his agency for five years, which he describes with affection as the pain of being a small business owner, and which built genuine empathy.

WordPress was one of the tools he deployed, and Automattic was on his radar for an unusual reason. This was years before remote work was normal, and he was living full time in an RV. There were very few companies that would allow that. Automattic stood out.

So he followed the people who worked there, met them at conferences, and eventually a role appeared that fit him precisely. He describes himself as a hybrid designer and developer who does not fit cleanly into either category, and the theme role suited exactly that. It became a decade-long tenure.

What the mission became

Michael raises the founding ethos: democratizing publishing, so that people in countries with oppressive regimes could have freedom of the press. He asks whether it survived.

Willett's answer is that it expanded. The mission now covers democratizing publishing, commerce and messaging, across a three-pillar business model.

His own learning curve is a good illustration of what global scale means in practice. Going from building one client site to building a theme downloadable by millions of people is a substantial shift. His example: right-to-left languages. Farsi, Arabic, Hebrew. Bugs would arrive and he had no way to test them, and had to hope someone at the company spoke the language.

On open source, he cites a recent Linux Foundation conference where a theme kept recurring: the economic value of open source software now runs into the trillions, computed as the replacement cost of the time it saves.

What a Chief Quality Officer actually does

Willett is candid that the role is technical operations, and that the title carries a certain gravity. He notes other companies have equivalents under different names, including a quality ombudsman at Sonos, a term he had to look up. His translation: a steward, someone who pays attention to what ships. Quality control, but more considered.

His framework has two Cs.

Craft. Design, code, product management, everything that goes into making software. Quality as the people building it see it.

Context. From the people it is made for, which may be external customers or internal employees using an internal tool.

Put them together and you have a feedback loop. Quality's job is first to make sure that loop is connected at all, and second to make it move faster.

What counts as quality

The misconception he addresses is that this is only about code. He pays attention to copywriting, correct typography, page speed, security, accessibility and how forms work.

His illustration came from a dentist's chair. His dentist's WordPress site had a video that loaded beautifully on desktop over wifi and took twenty or thirty seconds on a phone on mobile data. One unoptimized video.

The business example is sharper. Go-to-market lead generation with an intake form. Too many fields, too much friction, and the business grows more slowly. He notes this has been studied properly: more steps in the way means fewer completions, exactly like a funnel.

And the piece he says is newest to him: integration. WordPress is a platform, WooCommerce is a platform, and quality means integrating well with every layer of the technology stack and every player in the ecosystem.

Two thousand repositories, and the signal problem

Standing up a new team with a charter and a three-year plan, Willett started with inventory: which repository holds what, who owns it, how security updates and testing work.

He was surprised to find over 2,000 individual repositories. Which is more code than fits in anyone's head.

And that produces the problem he keeps returning to: signal versus noise. When you are watching that many things, if they are all showing red, none of it is actionable.

The contrast with the inner loop is instructive. While building, you get immediate feedback from your development environment and from opening a browser. Once something has shipped and is in public, tracking it becomes an entirely different exercise, spread across dashboards, metrics and alerts.

So his current work is audit and cleanup, simply to establish where the signal is, before handing anything to a team. He describes it as sense-making, or map-making, close to building a taxonomy of the whole thing so he knows where to apply pressure.

Why the WooCommerce case is hard

Willett's example of what makes this genuinely difficult is a typical WooCommerce store, which he says runs something like 45 to 50 plugins. His own blog runs three or four.

The complexity is structural. Some plugins are built by Automattic, some by third-party agencies, some by the merchant's own in-house developer. Add the host and the underlying language version.

Which means that when a new version of the core software or an Automattic-built extension ships, knowing how it will behave alongside everything else is a real problem.

He frames it as a grid. One axis is the whole stack: how will this behave anywhere it is deployed. The other is the end-to-end customer experience of the person visiting to buy something, where checkout has to be fast, mobile-friendly and translated.

Benchmarks, exactly like finance

This is the transferable idea.

Once the mapping and auditing is done, Willett sets benchmarks. Targets, on a quarterly and annual basis, the same way a business sets financial forecasts and plans, except applied to product quality and technical efficiency metrics.

The framework he points listeners toward is Core 4 from DX, which he says is common in engineering circles and starting to travel further. Its four dimensions are efficiency, speed, quality and impact.

What that gives you is a way to categorize your signals and then set targets against industry percentiles, so you can see where you want to aim relative to what comparable companies achieve.

And the part that makes it real: he instruments it, checks in on it, and reports to the board on it.

Michael's framing of why this matters at Automattic specifically is worth keeping. The overwhelming majority of the company is engineers, so smooth engineering operations, quality under control, human error reduced, and an environment where engineers can do their best work is directly load-bearing for the business.

Rituals before assurance

Asked how you get accountability at a company of around 3,000 people, Willett says it starts with rituals: the craft of how things get built, and getting to fast experimental loops.

Which means getting things to early beta testers, and finding immediate feedback before the ink is dry.

His position on quality assurance is deliberately unconventional. He puts the term in quotes, because it means very different things to different people, and rather than instituting a separate QA team and a separate step, his question to teams is how they are doing it themselves. Open the browser. Refresh. Check.

Supported by tooling. His example is component libraries that hold every visual element of an interface, buttons and links and the rest, against which automated tests can run every time the code changes.

The 5 things I took away from this conversation

1. Set quality targets the way you set financial targets. Quarterly, annually, benchmarked against industry percentiles, reported to the board. Almost nobody does this, and there is no good reason not to, because the metrics exist and the frameworks exist.

2. Craft and context, connected into a loop. Lance's two Cs are the cleanest definition of quality I have heard. It is not the standard of the work alone, nor the customer's opinion alone. It is whether those two are actually connected and how quickly the loop turns.

3. The intake form is a quality problem. This is the reframe that most operators will find useful. Copywriting, typography, page speed, accessibility, the number of fields on a form. All of it is quality, and all of it shows up in growth.

4. If everything is red, nothing is signal. Two thousand repositories with alerting on all of them is not observability, it is noise. Lance's first job was not fixing things, it was working out which signals matter enough to hand to a team.

5. Ask teams how they test rather than adding a testing team. A separate QA step creates a handoff and a place for responsibility to go. Asking each team how they verify their own work, backed by real tooling, keeps it where it belongs.

FAQ

What are software quality metrics? Measurable indicators of how well software is built and how well it serves its users, spanning speed, reliability, accessibility and defect rates. Willett's approach is to categorize the available signals and set targets against industry benchmarks rather than treating quality as a subjective judgment.

How do you measure engineering quality at scale? By first auditing what signals exist and which of them matter, then setting benchmarks. Willett uses the Core 4 framework of efficiency, speed, quality and impact, targets against industry percentiles, and reports progress at board level alongside financial metrics.

What does a Chief Quality Officer do? Willett describes it as technical operations with a specific mandate: connecting the loop between how software is built and how it is experienced, then making that loop faster. In practice this spans code, copy, typography, page speed, security, accessibility and integration across an ecosystem.

Is quality only about code? No. Willett explicitly includes copywriting, typography, page load speed, security, accessibility and form design. His business example is an intake form with too many fields, which is a quality failure that shows up directly as slower growth.

How do you run quality assurance without a QA team? By making it part of how each team works rather than a separate step. Willett's questions are about the rituals: how quickly you get to a fast experimental loop, how early you get something to beta testers, and what automated checks run whenever the code changes.

Also mentioned

• Automattic, WordPress and WooCommerce, and the three-pillar mission
• Core 4 from DX, the efficiency, speed, quality and impact framework
• Storybook, for automated visual component testing
• The Linux Foundation, and the trillion-dollar replacement-cost estimate of open source
• Right-to-left language support, and what global scale demands of a theme

Listen to the full episode

Lance Willett on Between Two COO's

Between Two COO's is hosted by Michael Koenig. Subscribe on Apple Podcasts, Spotify, or wherever you listen.