The Invisible Bugs AI Finally Finds (A Real Magento Case)

For six months, a Magento store I work on had a strange hobby. Two or three times a week, some of its categories, and sometimes all of them, would wake up completely empty. No products. Just a nice header, a nice footer and a lot of white space in the middle, like a supermarket after a hurricane warning.

The ritual was always the same. A message in Slack from the client: “Hey, the categories are empty again.” Then I would jump in, run the usual commands, put the products back in place, and everything was fine. Until the next time. We fixed the symptom so many times that it almost became part of the job description.

Then one day I stopped patching and gave the whole problem to Claude Code, Anthropic’s AI coding agent that works directly inside a codebase and a terminal. In two or three hours of heavy investigation, it went through the logs, the way the store works behind the scenes and the code itself, and found the pattern. The real cause. Not the symptom.

I was telling this story to a friend the other day, and he said what I was already thinking: this is a blog post. Because everybody talks about AI building new features faster. Fewer people talk about the other side, the one I think is even more valuable. AI is very good at finding the invisible bugs. The ones that were always there, that nobody ever identified, or that everybody just learned to live with.

The bug that only showed up when nobody was looking

Developers have a name for this kind of problem: a Heisenbug. It’s a bug that changes its behaviour, or disappears, when you try to observe it. You open the site to test, everything works. You check the logs from this morning, nothing obvious. You reindex, products come back, and you go to lunch feeling like a hero.

Ours was a classic one. When we tested, we could never reproduce it. It happened at specific moments during the day or the week, never when we were watching.

And that’s exactly what makes these bugs expensive. A study by Undo with a Cambridge Judge Business School MBA project found that developers spend about 26% of their time reproducing and fixing failing tests, and 41% of them named reproducing a bug as the biggest barrier to fixing it faster. Stripe’s Developer Coefficient survey put it in hours: the average developer spends more than 17 hours a week on maintenance issues like debugging and refactoring.

Seventeen hours. That’s almost half a week of someone sitting in front of a problem that politely refuses to happen.

What was actually happening

To understand the bug, you need three pieces of the store.

The PIM integration. A PIM (Product Information Management) is the system where the company keeps the “official” version of its products: names, prices, descriptions, categories. The store receives everything from it through an integration. This one runs a full sync every Tuesday and a partial sync every six hours.

The indexers. Magento doesn’t read products directly from the main tables every time a customer opens a page. That would be slow. It keeps “indexes”, pre-built tables that answer questions like “which products belong to this category?” very quickly. Adobe Commerce even has an official way to pause them: the indexer:set-status command, introduced in 2.4.7, can mark indexers as suspended, which temporarily stops their automatic, cron-triggered updates.

Live Search. Adobe Live Search is Adobe’s search and catalog service for Adobe Commerce. After the PIM sync, the store sends the catalog to it, so search and category pages show the updated products.

Now, the integration. Let’s just say it was not written with love. It rewrites a lot of Magento’s own resources, talks to the database through its own lower-level layer instead of Magento’s model layer, and has its own way of doing almost everything. And here’s how it worked:

1. It suspended all the indexers.
2. It removed every product from the website and from all categories.
3. It ran the import in batches and rebuilt all those associations from the PIM data.
4. It put the indexers back to work.
5. Then the Live Search sync sent the catalog out.

Step 2 was the developers’ clever idea to handle products that were removed from a category, or from the website, in the PIM. Instead of calculating what changed, they deleted everything and rebuilt it. It’s a lazy solution. It’s like taking every piece of furniture out of the house to clean the floor, every single week.

It works. Until you trip on the stairs carrying the sofa.

That integration was not prepared to handle errors. When an exception happened in the middle of the batches, the process just died. The associations removed in step 2 were never rebuilt. The indexers stayed suspended. And then the Live Search sync came along, did its job perfectly, and sent a catalog full of products that belonged to no category at all.

Result: empty category pages. Products still existed, they just didn’t live anywhere anymore. And because the integration only broke sometimes, every test we ran in a calm moment passed. The house was empty only on the days somebody tripped.

How AI found it in two or three hours

I want to be honest here, because this is where people imagine magic. There was no magic.

What Claude Code did was the boring, heavy work that a human needs days to do with full attention: read a lot of logs, line up timestamps, follow the code path of a third-party integration nobody wanted to open, compare what happened on good days and bad days, and connect the dots between three systems that each looked healthy alone.

My job was to give it context, point it at the right places, challenge its hypotheses and check every conclusion against real data. The AI’s job was to not get tired on log file number 14.

Once the pattern was clear, the fix came in two parts:

• fix what was making the integration crash in the first place;
• add guardrails, so the indexers are always resumed, even if another exception happens for a completely different reason in the future.

The second part is the one I care most about. You can’t promise a messy integration will never fail again. You can make sure that when it fails, it doesn’t leave the store naked. In simplified form, the idea is this:

<?php
// Simplified illustration of the guardrail idea, not the project's real code.
$this->suspendIndexers();

try {
    $this->runImportBatches();
} finally {
    // Runs on success AND on failure.
    $this->resumeIndexers();
}

Since then, no more Slack messages about empty categories. Six months of a recurring problem, closed in one afternoon.

This is not just my store: examples from the market

My case is small. The pattern is not. Some of the most tested software in the world has the same kind of hidden problem, and AI is finding it.

SQLite. Google’s Big Sleep team (Project Zero and Google DeepMind) used an AI agent to find an exploitable stack buffer underflow in SQLite, one of the most widely used databases on the planet. They caught it before it appeared in an official release. The team noted that “the issue remained undiscovered after 150 CPU-hours of fuzzing,” and the SQLite developers fixed it on the same day.

Firefox. Working with Mozilla, Anthropic reported that Claude Opus 4.6 discovered 22 vulnerabilities in Firefox over two weeks, and Mozilla classified 14 of them as high severity. Anthropic describes Firefox as “one of the most well-tested and secure open-source projects in the world.” Well tested, and still full of things nobody had seen.

curl. This one is my favourite, because it has the funny part and the serious part together. The curl project, the data-transfer tool used inside a huge number of apps and devices, was being flooded by low-quality “AI slop” bug reports. At the same time, security researcher Joshua Rogers used several AI scanning tools and sent reports that led to about 50 merged bugfixes. curl’s creator, Daniel Stenberg, called them “actually truly awesome findings” and summed it up perfectly: “Powerful tools in the hand of a clever human is certainly a good combination.”

Same technology. One group produced noise. The other produced 50 fixes. The difference was the human holding the tool.

The invisible bug can be very expensive

Some invisible bugs just cost a few Slack messages and a bit of dignity. Others cost a company.

In 2012, Knight Capital, a big American trading firm, deployed new code in which a repurposed flag reached old, unused code called “Power Peg” on one server where a technician forgot to copy the new version. In roughly 45 minutes, its systems produced about 4 million unintended executions, and the firm took a pre-tax loss of about $440 million.

Old code. A forgotten server. A flag reused because “it wasn’t doing anything anyway”. Every piece of it was invisible until the day it wasn’t.

I’m not saying AI would have saved Knight Capital. I’m saying this is the category of problem where a tool that can read everything, without getting bored, changes the odds.

Where these bugs live in your store or project

After 15+ years in Magento and Adobe Commerce, my list of favourite hiding places looks like this:

• Integrations that fail silently. ERP, PIM, marketplace or payment syncs that catch an exception, log nothing useful and leave data half-written.
• Cron jobs that collide. Two heavy jobs that work fine alone and fight each other once a week.
• “Temporary” workarounds. The script someone wrote in 2019 to fix stock every night, that nobody remembers why it exists.
• try/catch blocks that swallow errors. The code doesn’t crash. It just quietly does the wrong thing forever.
• Data drift. Prices, stock or categories slowly getting out of sync between systems.
• Performance that degrades a little every month. Nobody notices until Black Friday notices for you.

The common point: each system looks fine when you look at it alone. The bug lives in the space between them. That’s exactly where an AI agent that can read logs, code and configuration side by side is strong.

How to use AI to hunt invisible bugs (without fooling yourself)

AI is not a magic button. It’s a very fast, very patient investigator that needs a good detective in charge. This is how I work:

1. Write down the pattern you know. When does it happen? How often? What changes? “Categories empty two or three times a week” is already a clue.
2. Give it real evidence. Logs, cron schedules, timestamps of the Slack complaints, the code of the integration. Context beats clever prompts.
3. Ask for hypotheses, not answers. Then ask what evidence would confirm or kill each one.
4. Verify everything. AI can sound very confident while being wrong. Check its conclusions against the data before you change anything.
5. Fix the cause, then add a guardrail. The fix solves today. The guardrail protects you from the next failure you can’t predict.
6. Add monitoring. If it happens again, you want an alert before the client’s Slack message.

My take

For years we treated AI as a tool to write more code. That’s useful. But I think the bigger win for most companies is the opposite: understanding the code they already have.

Every store, every project, has a few invisible bugs. Problems that were always there, patched every week, accepted as “just how the system is”. Now we have a tool that can sit with those problems for hours, read everything and help us find the real cause.

The developer who knows how to use it doesn’t just build faster. They finally close the gaps everybody else learned to ignore.

So, tell me: what is the bug in your project that everybody already knows by name, but nobody has ever fixed?

Sources

• Undo / Cambridge Judge Business School: Debugging efforts cost companies $61B annually
• Stripe, The Developer Coefficient, via ADTmag: Developer survey (2018)
• Adobe Experience League: Manage the indexers
• Google Project Zero: From Naptime to Big Sleep (2024)
• Anthropic: Firefox security work with Mozilla (2026)
• The Register: Curl project, swamped with AI slop, finds not all AI is bad (Oct 2, 2025)
• Wikipedia: Knight Capital Group