Derek Haynes

@zeeg That UI is 🤌 ... how much back-and-forth did you have with Claude to put that together?

Shipped v1 of what I'm now calling Abacus (sorry not sorry). Pulls in Claude Code and Cursor usage, lets you look at them globally as well as per individual. Heavily focused on understanding consumption and cost. and its open source now, glhf github.com/getsentry/abac…

Speaking as someone who has maintained codebases over many years—and who cares deeply about code quality and long-term velocity—my first move today when enhancing a project is Claude Code, not opening my IDE.

Over a few days during the holiday break, I built Finchvox, an open-source debugger for Voice AI apps built with Pipecat. github.com/itsderek23/fin…

This is the kind of side project that never launched prior to Claude Code ... digging thru the OpenTelemetry schema, creating an OTEL collector, building a UI to align audio with trace spans ... this grunt work loses steam when it lives outside your day job.

@zeeg I like it ... similar approach I've used (jumps right to the implementation): #implementing-my-streamlined-edd-flow" target="_blank" rel="nofollow noopener">dlite.cc/2023/10/04/202… What is the output of assertLLMAgrees?

Im going to build a synthetic test suite that runs the production evals. I'll then take the results of those evals, and pass them through the LLM again to get it to "verify" the output roughly equals the output I want. assertLLMAgrees(output, "Indicates the game is 4 players");

Thinking more on this "how do I test my LLM implementation", with my "I dont know shit" approach...

@donnfelker @NathanSRobinson @hatchboxio Same! We're working on build.io (same Heroku build flow) with a generous free tier and fair pricing. DM if you'd like to try it.

@NathanSRobinson I found I hit the quotas on Heroku quickly, especially for rails apps. After moving my stuff over to @hatchboxio with Digital Ocean droplets, my costs plumetted, performance skyrocketed and I'm good to go.

My app broke because too many people used it (and because performance was an afterthought when I built it) Guess I need to prioritize some of those performance optimization tasks that's been on my todo for a few days... I honestly thought I could get away without it for much longer.

@zeeg How did Sentry emerge as the dominant exception monitoring service against A LOT of other products?

What should I write about next? I dont feel a need to keep things in a linear timeline, so was considering our journey on the Business Source License decision.

Ruby AI Survey 2023: docs.google.com/forms/d/e/1FAI… We'd really appreciate 2 min of your time to get a better idea of what problems Rubyists are experiencing building with LLMs. The results will be shared!

@BenRosmineML Hey Ben - also working on a natural language AWS CLI … can’t ever remember commands - github.com/opstower-ai/ll…

With so many different solutions for Chat with a PDF/knowledge database/website, how is there no Chat with AWS documentation?? It takes me at least 7 hours of searching to find anything in there

@zeeg From the post: > ...you’ll note Sentry existed for about five years before we launched the cloud service. That was half a decade of natural needs-driven growth... It's pretty crazy how much evals/monitoring/etc exist around LLM apps with relatively few AI apps deployed.

An excellent blog post on Sentry's beginnings by @zeeg - cra.mr/sentry-from-th… ... made me think A LOT about the LLM tools explosion.

@_cartermp > analyzing data relevant to a query I think this would be interesting to explore w/Honeycomb .. similar to ChatGPT's Advanced Data Analysis. Sometimes I upload a CSV and just ask it to explore.

@_cartermp So, I wouldn't bet on AIOPs to magically make sense of messy data.

@jwbiam1 Yes - thanks! Benchmarked here - github.com/opstower-ai/de…

@dhaynes23 have you seen this? release.com

Detailed blog post on my EDD flow: dlite.cc/2023/10/04/202…

Following this approach, I was able to get OpsTower.ai to the top of the DevOps AI Assistant Open Leaderboard. 😉 - I wrote the datasets for this leaderboard. See github.com/opstower-ai/de…

Last weekend I shared my streamlined Eval Driven Development (EDD) approach at the excellent AiPeaks.org. Evals were a big topic from @aiDotEngineer summit. What's worked for me: