Sabitlenmiş Tweet
On AI and The Death of Software Engineering.
Caution: much techno-speak follows. Thinking caps are recommended. 😆
I recently read a couple articles describing how to get AI to author and fully verify software. The articles were well written, but overlooked (ignored?) a couple significant issues.
Writing new software is difficult. Mimicking, copying existing software is pretty straight forward. Where is new software needed? Lots of places.
Porting operating systems to new processors.
Running new sensors, cameras, peripherals.
Driving cars.
Flying airplanes.
Controlling robots. Robots for surgery, for assembly lines, for personal assistants.
Launching rockets.
Landing rockets where you launched them.
These are non-trivial projects, where safety IS CRITICAL.
Verification is essential, as pointed out in the articles. And for safety-critical fields verification must be complete and error-free. In their latest AI training courses, Google emphasizes the need for verification, and emphasizes human involvement.
"But," respond AI enthusiasts, "that is slow, error-prone! So we'll get AI (different agents) to verify the output!" But then, how do you verify the verification? And how do you verify the verification of the verification? Ad Nauseum. Call this "uber-verification."
For simple tasks, it is simple, even trivial. But as task complexity increases, like in the fields mentioned above, verification needed, and associated effort, explodes. The overall write-verify task is a variant of the SAT (Satisfiability) problem in algorithm and complexity theory. Ask @grok to discuss SAT. It's a great read.
SAT is known to be NP-Complete. That means HARD. Extremely HARD in general form. And time to run explodes super-exponentially, a fancy way of saying the variables in your system are in the EXPONENT of the equation describing how long a solution takes to be generated or run. @grok also has much to say about SAT runtime versus problem size.
Humans, trained in the fields, are quite good at spotting errors, better than AI agents. So as the verification problem explodes, the need for human-in-the-loop goes UP, not DOWN.
Examining software creation and verification from another angle reveals the design/verify issue and solution are also a variant of the Turing Halting Problem (THP). You are asking, "Does this program behave correctly, and does the verification system prove it?"
The THP was PROVEN to have NO SOLUTION by the genius Alan Turing in 1936, 90 YEARS ago, long before digital computers existed (sorry, Ada Lovelace & Charles Babbage, close but no cigar.) And @grok could write a novel about Turing, Lovelace, and Babbage, which is also fascinating.
This brings us back around to the core theses of these AI articles and shows that there are fundamental reasons these problems can never be "solved" by AI. We CAN improve on past tools and solutions, but we need to keep the facts firmly in focus:
1. While you MAY have a set of agents/prompts that work great for some, even many cases, there can be no general system to write, verify, uber-verify any problem you give it. This is PROVEN. A CENTURY(ish) ago!
2. The combination of SAT and THP means the field can have many "really good" tools etc, but NONE OF THEM will be the ultimate, universal solution.
3. This does NOT mean "don't try this." Many companies have made fantastic breakthroughs in faster solutions to complex problems. The problem in general remains NP-Complete AND Undecidable. AND ALWAYS WILL BE. But in some cases, a human in the process, in the right place, making the right decisions, yields FANTASTIC results.
English
