@Cartoonhari Prompt caching: long system prompts, stop reprocessing every turn
Model routing: Haiku for simple fields, Sonnet only when reasoning complexity justified it
End-of-turn signaling to Deepgram: reduce STT/inference overlap
English
Wesley
20 posts
Wesley
@SumHern27200
thinking deeply, posting unseriously
San Francisco Katılım Ocak 2024
16 Takip Edilen23 Takipçiler
@SumHern27200 What optimizations did you make to Claude to achieve 78% latency drop?
English
@voiceflipme This is the exact wall we hit next. Static knowledge works until a patient asks about a specific clinic's hours or insurance accepted. What's your current approach scheduled scraping or live tool calls during the conversation? Curious how you're handling latency impact.
English
@SumHern27200 The GPT-4o to Claude jump is real - we got the same latency/cost win. The next wall wasn't speed, it was knowledge. 6 months on VoiceFlip: what moved bookings was the agent reading the live site - today's prices and hours - so it sells, not just answers fast.
English
@beknabdik "Fair! the orchestration did the structural lifting. But the model swap wasn't neutral. Tool use consistency under conversational variance improved meaningfully, especially on address disambiguation. The compression win matters less if structured data extraction is dropping calls
English
@SumHern27200 headline credits claude but your own thread answers it. the win was the orchestration work, not the model. compression alone cut the 40-turn cost. the swap mostly forced the rebuild that did the real lifting.
English
Code is open source: github.com/sumtzehern/hea…
Full writeup on LinkedIn.
Building voice agents for enterprise? The architecture decisions that matter aren't in most tutorials. Happy to talk.
English
What I'd do differently:
Build the eval harness first.
I built a p50/p99 benchmarking suite measuring inference latency, token throughput, and per-call cost, but I built it after the product.
Should have been the first thing. You can't optimize what you can't measure.
English
What surprised me most:
Tool use consistency under conversational variance matters more than raw speed.
In healthcare, missing a function call isn't a UX issue. It's a data integrity issue.
Claude's structured data collection held up better than expected under real conditions.
English
Optimization 3: Context compression
A full intake = 30-40 turns. Without compression, context balloons and costs scale linearly.
Solution: structured turn summaries. After each intake section, raw turns collapse into a structured summary.
Context stays under 10K tokens per call.
English
still refreshing my inbox like it’s a situationship 😂
Wesley@SumHern27200
failing a final round interview is worse than a breakup: 🥹 starts with “we really like your background” 👻 ends with ghosting so hard even your dad wouldn’t recognize you 10 toxic exes? light work. 1 recruiter on “we’ll get back to you”? absolutely soul-crushing 💔
English
especially in CS, the best way to learn is by building
if i could do it again, i’d still get the degree, but i’d spend even more time exploring things outside of class
there’s so much you can learn online for free. don’t waste your freedom.
Wesley@SumHern27200
as i wrap up my CS degree, here’s my biggest takeaway: the degree teaches problem solving, sure, but what it really gives you is 4 years to learn, build, and try things without heavy responsibility that freedom is the real value
English