What the AI Tests Got Wrong
The tests passed because they tested the implementation, not the behaviour. When I refactored the module, half the tests broke — not because the behaviour changed, but because the internal structure changed. Tests that break during refactoring aren't safety nets. They're anchors.
The AI wrote tests that mirrored the code structure. Test for each function. Assertions that matched the current implementation. It had no way to know what the code was supposed to do — only what it did. That's a fundamental limitation, and it's not the AI's fault.
How I Use AI for Tests Now
AI is excellent at generating test boilerplate and edge case lists. Given a function signature and a description of what it should do, it will enumerate edge cases I might have missed — empty inputs, boundary values, null handling. That list is valuable. The actual test code, I write myself.
It's also good at writing integration test scaffolding — the setup/teardown code, the mock infrastructure, the repeated patterns. The actual assertions — what the behaviour should be — that's mine. Those assertions are where the engineering judgment lives.
Key takeaways
- AI-generated tests tend to test implementation details rather than behaviour — they break on refactors that don't change correctness, creating false alarms
- Use AI to generate edge case lists and test scaffolding, then write the assertions yourself — the assertion is where your knowledge of the correct behaviour lives
- Test behaviour, not implementation — a good test should survive an internal refactor that preserves external behaviour
Conclusion
AI is a useful test assistant but not a test author. The judgment about what correct behaviour looks like — that's still yours. Tests written without that judgment pass CI and provide almost no actual safety.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
