Ian Cairns

A great conversation between Noah Kravitz from the @nvidia team + @hwchase17.

@jxnlco @OpenAI @romainhuet @OpenAIDevs Very cool! Sounds like a good fit. Congrats @jxnlco @romainhuet 🙌

I’ve recently joined @openai to work with @romainhuet on @OpenAIDevs Now is the year of dogged pursuits But Back in 2021 i thought my technical career was over. I had chronic hand pain in both my hands and could barely tie my shoes let alone use my phone or write code. I spent a few years not thinking about what it mean for the value of my labor to go zero but to not being able to produce any labor at all… I gave up bjj. Pottery. Tech. Etc. Then, that one company that solved dota and hide and seek released chatgpt and whisper and all of a sudden with dictation and some determination I could write essays, build things, and make a living from twitter meeting great people like @eugeneyalt @dmdohan @humford @GEVS94 for my reintegration into the tech world after so many years away. From Canada advised companies for free until I had to ask them to pay me. I charged companies until I figured out pricing and asked for enough that I became an investor as well. I started a consulting business and a course business. Learning alongside @HamelHusain and @vig_xyz But through that time I learned a lot about running a business and felt like I’d stopping learning about everything else. I realized that last summer that I wanted to wrap things up and go somewhere and just get involved and be at the center of it all.

Full episode covers a lot more: * Why MCP felt more natural than RAG for their system of record * "LLMs are the most expensive switch statement on the planet" (when to skip MCP tools and use code) * Why they pulled evals out of CI/CD Kevin's a Distinguished Engineer at Sprout and has been there for 13 years building infra. This is a good listen for other people making the transition to building agents. Full episodes: * Spotify: open.spotify.com/episode/06SIk5… * YouTube: youtube.com/watch?v=3PTAGZ…

🎙️New Deployed episode with Kevin Stanton from @SproutSocial. They're building agents that process billions of social messages and turn it all into signal. He shares great lessons learned as an engineering leader. One example, the benefits of chat UX: it's the fastest way to "seed your evals" with real traces, and a product manager's holy grail to learn what customers actually want to do.

@mitchellbyars 🥳 A joyous day to all who celebrate.

Happy sixth birthday to the greatest tweet ever tweeted

Cisco's @duosec built their AI evals & quality practice without a blueprint. A year later, they're watching the industry catch up to what they figured out by doing the work: automated evals > cross-functional data review > improve > repeat. Proud to support their team. 🙌

Our team's been shipping a ton this year! This one's incredibly practical: Build automations to move data around your system so you don't have to do it by hand. * Create review queues * Refresh datasets * Run conditional evals * Send Slack alerts * ...more to come!

In the last year, lots of teams have been trying to get PMs and domain experts more involved in AI product development and evals. Folks like @HamelHusain and @sh_reya have evangelized how important this is. Chime has figured it out, read on. 👇 freeplay.ai/blog/chime%E2%…

New case study: How @Chime scales AI in production by letting domain experts own evals and prompt performance alongside engineering. If you're figuring out how to formalize AI ops across your team, this is a good blueprint.

This evals flashcard is all you need. 👇

@HamelHusain @sh_reya This is the best flashcard. 🙌

This deserved its own flashcard b/c I've seen bus stop ads from eval vendors encouraging the opposite in San Francisco 🤣 The only thing generic metrics do is waste your time. Links in reply.

It’s amazing how often people sound grumpy in Slack or email and turn out to be totally fine. Assume positive intent.

@skeptrune @mintlify Honestly it's been great. 🙌 Couple little things, I'll send you a DM. Blogs: I love the workflow and would want to use it for lots of marketing stuff. It's a little tougher because we're committed to Framer... LMK if you have ideas.

@cairns @mintlify yay, so happy to hear this! any feedback on things which would have made the experience even better? also, we are planning to build Mintlify for blogs soon. do you think the existing workflow for docs would work well there too in your opinion?

I updated our new @mintlify docs site using Cursor + Claude over the break. It was the best software experience I've had in a long time. Coding agents aren't just for code. Every CMS should work like this.

@mintlify If you've never used it: * All your docs are .mdx files * Everything deploys automatically with every commit * If you have an OpenAPI spec there's extra magic for your API docs (example: docs.freeplay.ai/openapi/introd…)

It was 3 years ago today that @ericwryan and I started showing up full time to a real office to build Freeplay. 🥳 I'm all for remote and know it can be great, but I can't shake the feeling that moving to IRL was foundational for bringing our company to life. Since then we've had probably dozens of video call conversations that sound something like this: Big company folks: "Wait, you all are in the same building in real life? We're jealous… We [gave up our office / no one comes in anymore].” Us: "Yep. We did remote for years and know it can work, but it's felt great to be together in person." Here's what's worked for us: ⏱️ Synchronous comms. Things move fast in startup life. Need to solve a problem? Tap someone on the shoulder. Pair on code, or go for a walk along Boulder Creek and talk through it. ✨ Serendipity. So many good ideas have come from discussion outside official channels like Slack or GitHub, where people just stumble onto a good topic together. NGL, the side chats / coffee chats / lunch chats / etc. matter. 🤝 Trust. Being together in person has helped us get to know each other. I keep hearing from people in remote teams who struggle to figure this out. 😎 Vibes. We walk out into the middle of downtown Boulder, have a space that belongs to us, friends from other companies stop by… Kinda hard to explain other than “it feels good.” And I’d argue that’s ok. What might surprise people too: We’re very open to remote hires, and we have designated WFH days. We’re not religious about being in office, but we’ve been intentional about defining our team culture. * For remote folks, we tell them up front we’re a synchronous culture and they pair as much as anyone else. We also have an open video call all day with a view to our office so folks can drop in. And we fly everyone to Boulder every six weeks to retro, plan, and spend time together. (3rd annual ski trip is in February!) * And for anyone who lives locally but prefers to WFH sometimes, everyone has full flexibility Wednesday and Thursday. But we commit to start and end the week together in person. It creates a strong rhythm. The balance has allowed us to hire some great people who don’t live close by, but the clarity on our approach to in-office vs. WFH has helped too. Remote folks know how we work up front, and they opt-in (which means they generally like it too). That’s what’s worked well for us so far… Curious to hear what you think, and what else has worked well in your context. PS: If this sounds interesting let’s talk, link is in the comments. 🙌

It's windy here today.

@HamelHusain @SHREYASHANKAR Progress for sure 💪

@cairns @SHREYASHANKAR Looks good 🙌

🚀 New at @freeplay_ai: Review Insights An agent that automatically clusters themes as you review data, then suggests actions like eval metric creation or automated prompt experiments. Check it out.

@HamelHusain @SHREYASHANKAR Read more on our blog here. freeplay.ai/blog/introduci…

The result: Faster root-cause analysis, tighter iteration loops, and a stronger data flywheel for improving AI agents. Big shoutout to @HamelHusain for the push to get everyone looking at data, and to @shreyashankar — her EvalGen paper inspired this direction over a year ago, and it keeps getting better. 🙌