Bryce

I tried to use it this way and basically failed, the models aren't at the level where they can productively iterate on nanochat in an open-ended way. (Though one of the primary motivations for me writing nanochat is that I'd very much love for it to be used this way as a benchmark for agents, and I'd love it if it worked over time). I'm open to this just being skill issue. E.g. here some of the things I'd be suspicious about: - the zoo of torch compile flags can knowingly be abused to get +1% gains but often at the cost of +30min compile time. This is why modded-nanogpt prohibits torch compile kwarg engineering and why I haven't done any in nanochat either. i wouldn't reliably expect the model to notice, consider, or flag this kind of an issue or seek clarification. - ns_steps=3 might be a tiny bit of speed, but does the model also volunteer to make sure quality doesn't fall too much? - same thing for deleting .float() cast - sure you can remove it and get VRAM/speed gains but it's there for a clear reason (extra precision in the loss function). Removing it means you absolutely have to make sure that the lower precision is ok validation loss wise, in a highly controlled experiment. Overall I'm still struggling with getting the models to do significantly more basic things. For example, Opus keeps incorrectly "cleaning up" my comments when it doesn't understand them even when it's completely unrelated to the task, rude! It keeps violating and ignoring CLAUDE .md instructions on coding style but when I ask, it correctly points out all the violations. I know, I'm supposed to be using some kind of a /cleanup. Yesterday it gave me a table of results and incorrectly reported which experiment worked best (the table showed xyz=20 was best and it incorrectly claimed that xyz=12 was). Basically - much simpler things still fail routinely than something open-ended like "improve nanochat". (I've been doing a lot of YELLING IN UPPER CASE and I think this could actually be a really good metric for A/B testing instead of the inline survey thing.). Still incredibly net useful with oversight and with clear, well-scoped tasks. I definitely haven't given up on automatic closed-loop experiments with the models. It would be so glorious. I had 2 iterations that basically didn't work but I have ideas for the 3rd.

现在有部分人担心的是，AI 把项目改成黑盒，架构自己理解不了了，从而以后不好维护了，所以还需要去读 AI 代码，识别 AI 代码的实现问题。 咋说呢，我觉得这也很正常。我甚至也不知道了，就是说 AI 代码到底应不应该去考虑架构、考虑实现。 现在更像是一个分水岭，从手工编程时代迈向 AI 编程时代的重要转折点，虽然推特上大多数应该都是毫无疑问的全 AI 编程，但是仍有大部分的群体，是手工编程的。 到底怎么看这个问题，我觉得我也没有特别能说服别人的答案。

Codex team is working on a few experimental projects that are starting to shape up and I’m excited to share more about soon. But I’m curious, what would you like to see ship or improved by the end of the year other than better models?

我经常钉别人，凌晨三点把领导钉醒起来审批活动流程。把12个产品经理叫在一起钉钉对线撕逼，一个活动页面需要三个功能，每个功能有3-4个相似赛马产品在竞争使用，12个产品经理能撕两个月没有任何进展。搞活动有时候会要用到100多个内部系统。和产品老大吵架，对方使用钉钉的撤销信息功能来销毁证据。不能怪我凶残，因为给到的都是不可能实现的任务，内耗这么多，做点事经常感觉自己在徒手钻木取火，在荒岛上建海市蜃楼。自己搞钱自己搞人搞资源，然后才能搞点事…大多时候都是你们这些容易有PTSD的同事在制造阻力。

I'm noticing that due to (I think?) a lot of benchmarkmaxxing on long horizon tasks, LLMs are becoming a little too agentic by default, a little beyond my average use case. For example in coding, the models now tend to reason for a fairly long time, they have an inclination to start listing and grepping files all across the entire repo, they do repeated web searchers, they over-analyze and over-think little rare edge cases even in code that is knowingly incomplete and under active development, and often come back ~minutes later even for simple queries. This might make sense for long-running tasks but it's less of a good fit for more "in the loop" iterated development that I still do a lot of, or if I'm just looking for a quick spot check before running a script, just in case I got some indexing wrong or made some dumb error. So I find myself quite often stopping the LLMs with variations of "Stop, you're way overthinking this. Look at only this single file. Do not use any tools. Do not over-engineer", etc. Basically as the default starts to slowly creep into the "ultrathink" super agentic mode, I feel a need for the reverse, and more generally good ways to indicate or communicate intent / stakes, from "just have a quick look" all the way to "go off for 30 minutes, come back when absolutely certain".