Ramith Hettiarachchi
2K posts
@ramith__
PhD Student @CMUPittCompBio / @SCSatCMU Interested in ML for science while striving for trustworthiness
Adversarial Sequence Mutations in AlphaFold and ESMFold Reveal Nonphysical Structural Invariance, Confidence Failures, and Concerns for Protein Design 1. A new adversarial study systematically evaluates AlphaFold 3's robustness by introducing point mutations (up to 70%) and deletions (up to 10%) across 200 proteins, revealing striking structural invariance that raises fundamental questions about the model's biophysical reasoning capabilities. 2. The most concerning finding: AlphaFold 3 maintains virtually identical predicted structures even when 40% of residues are mutated with deliberately destabilizing substitutions, or when 10% of residues are deleted—perturbations that would catastrophically destabilize real proteins. 3. This structural invariance persists even for experimentally validated fold-switching proteins, where specific mutations are known to induce alternative conformations. AlphaFold 3 fails to capture these biologically critical transitions, suggesting limited sequence-structure coupling. 4. Confidence metrics prove unreliable: AlphaFold 3's ranking score selects the most accurate structure only ~25% of the time, and these scores correlate more strongly with template availability in the training set than with actual prediction quality. 5. Comparative analysis with ESMFold reveals that the protein language model-based approach shows significantly greater sensitivity to mutations, with structures diverging more rapidly as sequence perturbations increase—suggesting superior learned sequence-structure relationships despite lower absolute accuracy. 6. The study's template analysis provides quantitative evidence that AlphaFold 3's confidence reflects structural similarity to training-set exemplars (Pearson r=0.39) rather than genuine biophysical assessment, indicating heavy reliance on memorized patterns over learned principles. 7. These findings have profound implications for the entire AlphaFold ecosystem: protein design tools like RFdiffusion, binder design methods like BoltzGen and BindCraft, and drug discovery pipelines may inherit these fundamental limitations, potentially generating non-physical sequences or missing viable candidates. 8. The work identifies critical gaps in current structure prediction—models trained primarily on stable, wild-type proteins lack exposure to destabilized mutants and misfolded states, limiting their ability to generalize beyond the training distribution. 📜Paper: biorxiv.org/content/10.648… #AlphaFold #AlphaFold3 #ProteinStructurePrediction #StructuralBiology #ProteinDesign #MachineLearning #Bioinformatics #ComputationalBiology #AIforScience #ProteinEngineering #DeepLearning #Biophysics
Marc Andreessen: Revolutionary technologies were often viewed as “trivialities” or “jokes” “If you read history, the great innovations of the past are now well understood as being very important. In almost every case, they were not widely understood as such at the time. In fact, I would assert that they were often actually viewed as trivialities or jokes.” He gives three examples: 1. The telephone. “When Thomas Edison was first working on the telephone, the assumption of the use case motivating his early work was the idea that telegraph operators needed to be able to talk to each other. It was considered implausible that you would have a system that would let any ordinary person pick up the telephone and talk to another person - that was clearly impossible… Completely missing the larger opportunity.” 2. The Internet. “I have personal experience with this one. The Internet was laughed at. It was heaped with scorn from 1993 to 1997-98. In fact, those of you who were in the industry at the time will remember the New York Times had a reporter on staff named Peter Lewis… I’m convinced he was specifically hired by the editors to just write negative stories about the Internet. It was all he did, and it was always the Internet was never going to be a consumer medium. The Internet is not nearly as big as these people think. Nobody is ever going to trust the internet for e-commerce.” 3. The car. “The car was absolutely viewed as a triviality and a toy when it first emerged. In fact, J.P. Morgan himself refused to invest in Ford Motor Company with the response that it’s just a toy for rich people, which is in fact what it was at the time. If you had one of the first cars, you had to be a rich person. You had to have a driver. You often actually had to also have a stoker with your early cars to keep the engine going. And then you also had to travel with a full-time mechanic because the thing would break down every three miles.” Marc concludes: “The great innovations of the present, I believe, are virtually guaranteed to be viewed as trivial and to be viewed as jokes. I think history 50 to 100 years from now will enshroud them in legend. In our time, they won’t be recognized as such. Of course, in the future, when they become legends, our descendants will themselves have their own trivial innovations to laugh at.” Video source: @MilkenInstitute (2013)
