Motoki Wu 🌊🦈

All this compute power for language models but most of our knowledge are in screenshots.

2026 will be the year of Claude Code UIs.

Coding agents are so good now that you should be using it for everything. Text classification? Coding agent RAG? Coding agent Prompt optimization? Coding agent Recipe for next week's Christmas potluck? Coding agent

The only late night parties in SF are labeling parties 🥳.

oh no

"Owen" is here, the American version of Qwen!

If RL needs pre-training level scaling, are we admitting that RL learns really slow...

I know RAG is dead, but please chunk your docs for the browser :) platform.openai.com/docs/api-refer…

OpenAI chose Windsurf because Ctrl+N doesn't delete your prompt :P

Agents for recall, workflows for precision.

Post training making a comeback. MLEs can finally breathe.

@Thom_Wolf My favorite was τ-bench (arxiv.org/abs/2406.12045). It evaluates on states, so the trajectory of the conversation and the agent's implementation matter less, and there's no need to rely on LLM evals. It strongly influenced how I structure agent workflows.

What was the most impactful/visible/useful release on evaluation in AI in 2024?

I would say a skillful cook would have a sharper knife.

There's no reason to FOMO over a new model when you can wait one month and FOMO over a better model :)

Can o1 at least help us with Pydantic upgrades.

Now we can take coffee breaks during test time!

Some Bad: * you pretty much have to restart PyCharm if something goes wrong * it’s the only thing that makes the fans spin up on my M1 * not Cursor 😛

The Good: * strong autocomplete * codebase understanding with jump-to-definition * jump to previous cursor position * interactive tables * stable enough for remote notebooks Essentially, having all the IDE goodies in a notebook keeps me in the flow.

After 10(?) years, I think I finally found a better alternative to just running plain Jupyter notebooks: PyCharm's notebook integration. I've been using it off and on for awhile, but it's reached to a point now I can wholeheartedly recommend.

I just implemented agentic RAG. ☺️

@abacaj I've found loss masking to work well, but it does lower the number of tokens that the model sees. So you may have to compensate for that (more epochs / higher batch size).

Couple of questions that I haven't seen answered yet: 1. Why did stanford alpaca team set instructions to -100, forcing the model to condition on outputs? 2. Why do I see a better model (including proper validation loss) training on full sample, including instructions?

Why would SFT show increasing validation loss and still an "improving" model? Two runs, same hparams. Only difference? Training on "inputs". In the original stanford alpaca they set the instructions to -100 to condition the model on the outputs But removing this line, shows a steady validation loss (blue line) - and a model that performs better in my benchmarks... What gives?