Your team builds a product with AI. Then AI writes the tests and runs them. Everything passes, the team is happy, and the release goes out. A few days later, customer support starts hearing about problems.
Sound familiar?
AI in software testing is making it easier to create and maintain tests. We’re using those capabilities throughout ErikLabs’ end-to-end QA platform. When the same technology develops a product and checks its own work, though, we need to ask what those checks actually prove.
Google engineers have described a similar concern. A DORA analysis published in March 2026 examined 1,110 open-ended responses from Google software engineers. One respondent explained that AI helped them work faster in unfamiliar areas, while leaving them without enough context to judge whether the output was correct.
Delegating the work is the easy part. Knowing what has been missed takes a different kind of understanding. Your customers arrive with expectations, habits and circumstances that need to be reflected in the test coverage. Understanding your industry is part of that.
Where can testing AI-generated code go wrong?
AI can treat the code it has written as the source of truth. If it misunderstands a business rule, it may build the same misunderstanding into the test. The code and test agree, which looks reassuring. Both can still be wrong.
Researchers at the University of Luxembourg investigated this in a study released as a preprint in July 2026. They selected faulty AI-generated programs and compared two test-generation workflows.
In one, the model wrote tests within the conversation containing the faulty code. In the other, it received only the task description in a fresh conversation, without seeing the code. Tests from both workflows were run against the same faulty programs.
Tests generated with the faulty code in the conversation detected approximately 14% of the selected faults. Tests generated from the task description alone detected around 25%. The study deliberately selected difficult-to-detect faults, so these figures should not be read as overall success rates for AI testing.
The finding gives us a practical question to ask: does the test’s expected result come from the requirement, or from what the application already does?
Who explains the business rules to the AI?
Columbia University’s DAPLab documented a useful example while developing more than 15 applications with five coding agents and tools.
A shopping application was supposed to apply a 10% discount to the entire basket when the order exceeded $40. The agent instead applied the discount to individual items priced above $40. The application ran, but its pricing logic was wrong. Published in January 2026, the investigation covered experiments conducted in November 2025.
“Does the discount appear?” would be a weak test here. We need to check that it applies to the right amount, under the right conditions.
Then come the details particular to your business. What happens when a customer removes an item? Can the offer be combined with a loyalty benefit? How should a partial refund affect the discount?
Some answers sit in requirements documents. Others emerge in conversations with product, finance, operations or support teams. An experienced QA team knows where those gaps tend to appear and who can help resolve them.
Better context helps AI produce better tests. Recognising which context is missing is part of the expertise.
What are companies getting right with AI-powered test automation?
Meta’s Catching JiTTests system generates executable tests to expose behavioural differences introduced by code changes. The next challenge is deciding which failures reveal genuine defects.
Meta combines rule-based checks with assessments from multiple language models to filter false positives, with engineers defining criteria and reviewing outcomes. According to Meta’s own reporting, the system helped catch 74 serious privacy bugs before production in the first half of 2026 and contributed to a 20% reduction in serious privacy incidents during that period.
Ant Group’s work, presented at FSE 2026, addresses another problem: AI-generated unit tests that fail to compile and therefore cannot run. Its researchers identified failures caused by missing project context, alongside failures that persisted even when adequate context was available. Their approach combines context construction with repairs based on static analysis.
These examples tackle different problems, but both teams put substantial work into checking what AI produces.
That is where we see the opportunity. AI can help us generate tests, reduce maintenance and investigate more scenarios. The process also needs a way to establish whether those tests are meaningful, executable and relevant to the risks we care about.
How we put sector experience to work at ErikLabs
We start by understanding your business. Our experience across finance, e-commerce, telecoms and SaaS helps us recognise where to ask more questions. A promotion and refund journey needs different checks from a financial application’s transaction and permission rules.
Our AI also brings sector-specific scenarios accumulated through previous work into the process. Depending on the area, that can mean dozens or hundreds of scenarios to consider alongside your own requirements. Your product team’s knowledge helps us decide which are relevant and what needs adapting.
We look at existing tests, previous defects and release habits. A recurring support complaint, an unreliable automated test or a check that keeps getting postponed can change our priorities.
Before automating a scenario, we agree what the correct outcome should be. If the requirements leave something unclear, we talk it through with your team. Resolving an assumption early saves the work of correcting it across an entire test suite later.
With SmartCase’s AI-powered test automation, we turn those scenarios into repeatable automated checks, using AI to support test creation and maintenance. Adapting to a moved button is useful. Recognising that a changed business rule needs a new expected result matters just as much.
We then look at the same journey under other conditions. SmartLoad helps us assess load and performance; SmartFarm lets us check behaviour on real devices; SmartCrowd brings different users, habits and circumstances into the assessment.
Take a promotional checkout. The discount may be calculated correctly, but can the customer complete payment during a traffic spike? Does the order reach the fulfilment system? Can someone using an older phone get through the same steps?
Those findings belong in the same quality assessment. Our end-to-end QA approach connects requirements, functionality, performance, devices and customer experience so your team can make a release decision with a fuller picture.
Let’s avoid fixing the same problem next release
Everyone wants to get useful features into customers’ hands sooner. Testing needs to keep up, and the team making the release decision needs to know what has actually been checked.
We want you to finish a QA engagement with an assessment you can use. Which critical journeys were verified? Which devices and traffic conditions were covered? What remains unresolved? Engineers should have access to the detail, while the person approving the release should understand the consequences for the business.
A high pass rate can hide an important failure. If the one unsuccessful scenario prevents customers from completing their most common transaction, that deserves attention. We consider the effect of the defect, how often it occurs and whether it has appeared before.
We also need to know when a result was obtained. A payment provider, permission model or external service may have changed since the last successful run. Linking tests to requirements and affected systems helps teams decide what needs checking again.
Some releases can proceed with known issues. The important thing is to understand the risk, agree who owns it and make the follow-up clear. Untested areas should be visible in that discussion.
The work continues after release. A customer complaint should inform the next test plan. A production defect should lead to a check that can catch it again. Otherwise, the team spends the time saved with AI solving a familiar problem for another customer.
That is the quality system we want to build with you: one that retains what your team learns and uses it in the next change. We combine sector experience, automation and real-world testing to give your team more time for new development.
If AI has accelerated your development, let’s look at how your quality process can keep pace. Tell us about your product, your customers and the problems taking up too much of your team’s time. We can work out where to start together.