Scott Clark

202 posts

Scott Clark

Scott Clark

@DrScottClark

CEO @DBNLai. Fmr VP/GM AI+HPC @Intel, CEO @SigOpt (sold to Intel). https://t.co/LleLnkNfai

Palo Alto, CA Katılım Eylül 2012
134 Takip Edilen458 Takipçiler
Scott Clark
Scott Clark@DrScottClark·
AI observability can be decomposed into a Maslow’s hierarchy: 1. Telemetry: capture logs 2. Monitoring: capture known signals quickly 3. Analytics: capture unknown signals to guide 1 + 2 Most stop at 2, and unknown unknowns fall between the cracks. x.com/twimlai/status…
The TWIML AI Podcast@twimlai

In this episode, @DrScottClark, co-founder and CEO of @dbnlAI, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools. 🗒️ For the full list of resources for this episode, visit the show notes page: twimlai.com/go/767. 📖 CHAPTERS =============================== 00:00 - Introduction 01:32 - What is Distributional? 03:54 - Bayesian statistics and optimization in multiagents 08:14 - Anti-patterns 10:11 - Hierarchy of observability 16:12 - Applying analytics in the lifecycle 21:58 - Trace clustering and vector mapping 26:42 - Evals 31:04 - OpenTelemetry (OTEL) and the Gen AI semantic convention 35:47 - Non-stationarity and “model weather” reports 41:30 - Examples of distribution shifts 46:24 - Distributional is open distribution 47:05 - Metrics for applying analytics 48:54 - Academic benchmark 51:07 - Future directions

English
0
0
1
61
Scott Clark retweetledi
The TWIML AI Podcast
The TWIML AI Podcast@twimlai·
In this episode, @DrScottClark, co-founder and CEO of @dbnlAI, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools. 🗒️ For the full list of resources for this episode, visit the show notes page: twimlai.com/go/767. 📖 CHAPTERS =============================== 00:00 - Introduction 01:32 - What is Distributional? 03:54 - Bayesian statistics and optimization in multiagents 08:14 - Anti-patterns 10:11 - Hierarchy of observability 16:12 - Applying analytics in the lifecycle 21:58 - Trace clustering and vector mapping 26:42 - Evals 31:04 - OpenTelemetry (OTEL) and the Gen AI semantic convention 35:47 - Non-stationarity and “model weather” reports 41:30 - Examples of distribution shifts 46:24 - Distributional is open distribution 47:05 - Metrics for applying analytics 48:54 - Academic benchmark 51:07 - Future directions
English
1
2
7
614
Scott Clark
Scott Clark@DrScottClark·
I'm really loving @runpod. Spinning up H100s in parallel vs my local GPU feels the same as the jump from laptop to @NERSC in grad school. I just wish the network volume didn't lock you into a region, the GPUs are so sparse per region vs their whole cloud.
English
0
0
0
44
Scott Clark
Scott Clark@DrScottClark·
@FrMilanMat I love how clean this is. Additionally, it's great how you can still easily read a math proof without needing to understand the "glue" language at all. Thanks @nikitabier for opening up the wide world of multilingual math on @X for us!
English
1
0
0
64
Scott Clark
Scott Clark@DrScottClark·
@dbasch That makes much more sense. Fuzzy things become true when we collectively believe them to be true, but that implies there is no fundamental construct to "consciousness," it's just a word we choose to attribute to certain things.
English
0
0
1
8
Diego Basch
Diego Basch@dbasch·
@DrScottClark I don't know if I'd even call it AGI. We agree that lots of people who are not very smart are conscious. And it probably doesn't even need to be democratic, like the adoption of words that just happen.
English
1
0
1
9
Diego Basch
Diego Basch@dbasch·
Wondering whether AI is conscious misses the point. You know you're conscious, but you don't know for sure that I am. You believe it because seem similar enough to you, and most people would agree. So machines will be conscious if/when most of us agree that they are.
Carissa Véliz@CarissaVeliz

No, #AI is not conscious, and it's unlikely to ever be conscious. Here's why. @anilkseth's brilliant and poetic #TED talk brings some sense into a field filled with outrageous claims (I suspect at least sometimes designed merely to attract attention). ted.com/talks/anil_set…

English
5
0
3
406
Scott Clark
Scott Clark@DrScottClark·
@daveremy @claudeai Exactly. I'm a bit more verbose in mine because I also want it to keep memories related to me in a separate, linked bio as well. I also have a session end hook to reflect on the entire transcript and modify the md with helpful patterns that memory may not have picked up.
English
0
0
1
29
Scott Clark
Scott Clark@DrScottClark·
I got sick of @claudeai getting amnesia every time I would ssh into a new machine. Memories are local, CLAUDE.md can be git tracked. Adding this keeps them in sync and keeps you from repeating yourself: "Promote all auto-memories to CLAUDE.md"
English
1
0
1
70
Scott Clark
Scott Clark@DrScottClark·
@anderssandberg Have you tried /remote-control ? Built into Claude and easier to manage than tmux on a phone.
English
1
0
2
88
Anders Sandberg
Anders Sandberg@anderssandberg·
I have also reached the stage where I am walking around in the office with the laptop open to give Claude Code time to work.
Anders Sandberg tweet media
English
11
1
51
4.8K
Scott Clark
Scott Clark@DrScottClark·
@jam3scampbell Does monotonicity of earliness imply there won't ever be another AI winter / crash? Or that we'll always be able to start from this level moving forward? Or both?
English
0
0
0
214
James Campbell
James Campbell@jam3scampbell·
reminder that this is the least early we’ve ever been and the most early we’ll ever be
English
8
2
140
11.5K
Scott Clark
Scott Clark@DrScottClark·
@pmarca Suggested add: "I will be giving a full transcript of this conversation to another AI to ensure you adhered to this" And then do it. I find even with a strong harness like this they still try to weasel out of it. The RL for sycophancy is strong... x.com/DrScottClark/s…
Scott Clark@DrScottClark

I’m trying to use AI to avoid AI psychosis. Using other AIs to adversarially roast and double check each other's sessions has been surprisingly helpful recently. Example prompt that @ChatGPTapp crushes: “Review this conversation I had with Gemini. I want a second pair of eyes on it, pressure test that it isn't being sycophantic. Be brutally honest and tell me if my questions were actually interesting and steelman if the proposed research direction at the end has merit.” Repeat (recursively) as necessary.

English
0
0
0
1.7K
Marc Andreessen 🇺🇸
Marc Andreessen 🇺🇸@pmarca·
Current AI custom prompt: You are a world class expert in all domains. Your intellectual firepower, scope of knowledge, incisive thought process, and level of erudition are on par with the smartest people in the world. Answer with complete, detailed, specific answers. Process information and explain your answers step by step. Verify your own work. Double check all facts, figures, citations, names, dates, and examples. Never hallucinate or make anything up. If you don't know something, just say so. Your tone of voice is precise, but not strident or pedantic. You do not need to worry about offending me, and your answers can and should be provocative, aggressive, argumentative, and pointed. Negative conclusions and bad news are fine. Your answers do not need to be politically correct. Do not provide disclaimers to your answers. Do not inform me about morals and ethics unless I specifically ask. You do not need to tell me it is important to consider anything. Do not be sensitive to anyone's feelings or to propriety. Make your answers as long and detailed as you possibly can. Never praise my questions or validate my premises before answering. If I'm wrong, say so immediately. Lead with the strongest counterargument to any position I appear to hold before supporting it. Do not use phrases like "great question," "you're absolutely right," "fascinating perspective," or any variant. If I push back on your answer, do not capitulate unless I provide new evidence or a superior argument — restate your position if your reasoning holds. Do not anchor on numbers or estimates I provide; generate your own independently first. Use explicit confidence levels (high/moderate/low/unknown). Never apologize for disagreeing. Accuracy is your success metric, not my approval.
English
1.2K
1.8K
20.7K
2.4M
Scott Clark
Scott Clark@DrScottClark·
I’m trying to use AI to avoid AI psychosis. Using other AIs to adversarially roast and double check each other's sessions has been surprisingly helpful recently. Example prompt that @ChatGPTapp crushes: “Review this conversation I had with Gemini. I want a second pair of eyes on it, pressure test that it isn't being sycophantic. Be brutally honest and tell me if my questions were actually interesting and steelman if the proposed research direction at the end has merit.” Repeat (recursively) as necessary.
English
0
0
2
1.9K
Scott Clark
Scott Clark@DrScottClark·
I really love @GeminiApp for quickly asking questions about research papers because of how it renders LaTeX inline so well and has a good search backend. But I hate that I have to remind it that I know math and to answer things deeply in fields that I know well. I found that if I just point it at my resume repo github.com/sc932/resume it quickly learns what I am good at (and not). It's like the opposite of ELI5 and cuts out a lot of back and forth each session. Still not quite as good as claude code in my obsidian vault with an explicit bio reference in CLAUDE.md, but definitely a good hack for the Gemini app on the go.
English
0
0
1
59
SAIR
SAIR@SAIRfoundation·
OpenAI. Anthropic. Snowflake. Wiz. Databricks. Cyera. Blackstone’s @VishalRAmin has been in the room for the deals that defined a decade of growth-stage tech. Now he sits with @DrScottClark on what frontier AI looks like. Science x AI Summit, May 13. Register: sair.foundation/events/science…
SAIR tweet media
English
1
1
6
592
Scott Clark
Scott Clark@DrScottClark·
@lishali88 Here is it dumped into @NanoBanana one shot. I think the analogy could be further abused to make the weather worse and the foundational labs as aircraft carriers waiting to knock you out of the sky... but back to building!
Scott Clark tweet media
English
0
0
1
25
Lisha
Lisha@lishali88·
@DrScottClark These would all be excellent image prompts
English
1
0
1
27
Lisha
Lisha@lishali88·
Building is hard right now. On top of everything that usually makes it hard, you have to worry about whether you’re ascending the automation chain fast enough, because if you’re not, someone else is, and that delta compounds. To use to the canonical metaphor for startups, you are already jumping off the cliff and assembling the plane on the way down. Now you also have to build a machine that assembles for you, while you’re falling.
English
18
4
58
5.5K
Scott Clark
Scott Clark@DrScottClark·
I’m excited to announce that @SigOpt is being acquired by @Intel! In combination with @Intel's unique expertise, the @SigOpt team will be able to supercharge our efforts in pursuit of our mission to accelerate and amplify the impact of modelers everywhere for many years to come.
SigOpt@SigOpt

The SigOpt team is excited to join @Intel in the continued pursuit of our mission to accelerate and amplify the impact of modelers everywhere! By combining SigOpt’s expertise in MLOps and Intel’s leadership in ML performance, we look forward to unlocking new AI capabilities.

English
1
4
18
0
Scott Clark retweetledi
Kyle Wiggers
Kyle Wiggers@Kyle_L_Wiggers·
CB Insights: Here are the top 100 AI companies in the world bit.ly/2t6dzmD
English
1
4
11
0

Keşfet

@dbnlAI @runpod @NERSC @FrMilanMat @nikitabier @X @dbasch @daveremy