
dmarxn
82 posts










Andrej Karpathy beautifully explains the fundamental difference of learning between a human and an LLM. > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” many of the best minds in this field think LLMs are not really learning anything, and therefore are incapable of surpassing human-level intelligence.






>an eval and benchmark would be great - that’s a great thing to publish! (and would bolster your alls position) I might not have the bandwidth at the moment but we can share this internally. Although if you really wanna prove your case you can try tweaking Aider github.com/Aider-AI/aider/ It has all standard benchmarks and based on my understanding they use Treesitter and other techniques and not rag chunks. You can keep everything else exactly the same where the search retrieval tactic is the only thing tweaked. Personally I do not trust the SWE benchmark and most benchmarks as they have been gamed but if you specifically A/B test between the search mechanism that will be very insightful.




What's your favorite technical interview question as an interviewer?























