Alex Strick van Linschoten

🎉 The @zenml_io LLMOps Database just crossed 1,000+ case studies! 17 months of curating real-world LLM production stories. Here's how the collection grew over time 📈 What started as a side project in July 2024 is now the largest open collection of production LLM case studies.

@Vtrivedy10 @mattpocockuk Would recommend you check out @sh_reya 's skill for error discovery. Quite similar to what you're talking about here I think, and it's directly targeted at it. github.com/shreyashankar/…

ok so it’s early but @mattpocockuk’s grill-me skill feels like great DX for iteratively building evals/environments with humans + agents in the loop understanding a repo + traces + definition of success is pretty much never a one shot thing, it’s many step collaboration between humans and agents idk how to measure & improve every agent so getting grilled helps from an agent definition + data dump, it’s impossible to know which axis of agent performance teams want to generate evals for measurement and hill-climbing specifics matter a lot here: - verifier design (do we have ground truth somewhere) - what tools in the environment should be simulated vs run (semi)-live - what characteristics do we care most about rn? cost, accuracy, etc? - what data needs to be sourced or synthetically generated to produce a realistic production like environment? how do we do that? data is the currency for building better agents humans sharing expertise/priors & controlling the loop on data design helps a lot today

New legal dataset up on the @huggingface Hub! Over the weekend I worked to finalise a snapshot of the official legal documents hosted by the Guantanamo Military Commissions. This is mostly court filings and transcripts from all the cases, around ~52GB of data. Link to the dataset at the bottom of the 🧵

When asking models to draft prompts (esp for things of more consequence like long-running processes / tasks), is it better to ask a model from the same family to write the prompt, or to use a different model family? Specifically, would a Claude model (Opus or Fable, e.g.) do better at writing such a prompt for ChatGPT Sol, or would Sol be best at writing a prompt for Sol? Any practical experience with this? (My gut tells me that probably staying within the family is best, but I don't know whether this applies to the latest GPT 5.6 generation.) Anecdotes + gut feelings welcomed!

Yeah I guess I'd agree with that in an abstract sense. I think I was looking more for experiential "this really worked for me" kind of gut responses (if they existed). It's also quite hard to run these kinds of evals across long-running tasks I think because it ends up being really expensive! but I totally agree with the premise of what you're saying!

@strickvl Imo it’s more about your ability to evaluate and close the loop on the prompts (which is the core thesis of DSPy) You can use any model to optimize the prompt so long as you can evaluate good and bad.

Published my first little environment on the @PrimeIntellect Environments Hub yesterday evening. Very happy to finally have that out and complete! (links and more comments below, and a blog to follow I guess)

Kitaru replay is pretty nice to use, and even better everything is hooked into the Kitaru MCP etc so you can have your agent drive these kinds of experiments as well!

@PrimeIntellect Updated my environment to use the new 'v1' API that they just released today. Congrats @PrimeIntellect! primeintellect.ai/blog/verifiers…

One little correction to the numbers I mentioned above about the GEPA gain... (After posting I went back to the original numbers and observed the following: - first off, the two numbers weren't measured on the same rows (so apples to oranges comparison; not useful!). - secondly, and more importantly, the GEPA score number was GEPA's own score :) (It wrote some candidate prompts and recorded its score on the same 50 rows and reported the best of those.) So I reran it properly. Same model, same reward, but using the new GEPA system prompt and across rows GEPA had never seen. New results: Baseline prompt: 0.803 reward GEPA prompt: 0.968 reward (+0.166, def a real signal IMO) Anyway, all of this is just to say, remember to really understand where your numbers come from otherwise you'll have to write :hides-face: updates like this haha

huggingface.co/datasets/stric… here's the dataset! Let me know if you end up using it in some agentic RL work or if you have any questions!

What's it useful for? Firstly it's just an archive of what exists. That has value in and of itself. Secondly, (and the main reason why I took the effort to assemble it) it's a really great open dataset for long-running agentic tasks. Think of all the amazing work @harvey (amazing posts recently from @gabepereyra / @nikogrupen / @ItsJulioPereyra 🙌) are doing around post-training of their models and their harnesses. They need interesting datasets to drive quality environments to help scale their efforts. I'll be doing my own work to build environments on top of this dataset (which I know the domain specifics of quite well) but I'd hope that others also find uses for it to improve long-running legal AI).

@pvncher yeah I def do that. but for long-running single-use prompts it's sometimes hard to make those comparisons. I'm sure we'll all get used to the new models + how to best coax them, but for the moment just trying to find something to hold on to haha

@strickvl I think either family is fine but you should understand the model’s behavior with that prompt to get best results. I like running some blind model stress tests and having the prompt refined from feedback from the model running the tests observing rollouts.