Vector retweetet
An analogy.
A software project is like an oddly shaped container that you are trying to fill with water. The shape is the required behavior, and the water is the software.
Prompts and plans attempt to define the shape for the AI, but AIs have very poor long term memory, and even their short term memory is time-biased. Things you told it a minute ago just aren't as important as they were when you said them. So the shape of the container is something the AI simply cannot hold in its "mind".
To hold that shape requires tests. Lots of tests. And this is why many people using AIs are trying to use TDD. But the AI can't reliably remember the rules of TDD. Those rules lose importance as the context window fills. So you have to run test coverage and direct the AI to cover all the uncovered lines.
That will stabilize the shape of the container; but it leaves holes. Remember that coverage does not prove that the tests cover everything with assertions, it only proves that the tests execute the covered lines.
Those holes are leaks that the AI can sneak through -- and it will. It will take advantage of any leak in the tests, and that will create undesired behaviors that are often very difficult to detect and unwind.
Mutation testing is the tool that plugs those leaks. It will find every missing assertion and you can direct the AI to cover them.
But mutation testing is a two edged sword. First it requires a lot of cpu time and a lot of wall time. Second, it stabilizes the shape of the container so strongly that the AI will have to work very hard when you want that shape to change.
It's the old trade off. Stability and reproducability vs speed. To the extent you want the one, you can't have the other.
So choose wisely.
English
