Aaron

@Suryanshti777 Book

This AI System Design guide teaches RAG better than most courses. And I'm giving it away for free (Only for First 4500) Inside: • RAG fundamentals & chunking strategies • Hybrid retrieval (BM25 + vector search) • Production-level RAG architecture • Evaluation & RAGAS metrics • Hallucination reduction techniques • End-to-end LLM system design How to get it: • Follow me (must so I can DM) • RT + Like • Comment "book" I'll dm you

@ChrisLovejoy_ Honestly I am still feeling dumb when it comes to skills - I get the distinction between it and a context file but on a practical level it’s lost on me. I haven’t got the oh I should probably create a skill trigger in my mind yet to fully understand how to leverage it.

I feel late to the party but made my first skills for Claude Code today. Started with skills for loading in context (because it's all about having the right context): FROM LOCAL COMPUTER: 'read-daily-note' and 'read-weekly-note' skills - points to the relevant md file in my @obsdmd vault - I make these notes every day / week and they have the most relevant context for what I'm working on and thinking about FROM THE INTERNET: 'summarize-article' and 'summarize-video' skills - I pass in a URL or name + author, it gets the content, summarizes it, then discusses it with me in a socratic way. summarize article is just my prompting, for summarize video I'm using @steipete's CLI Starting simple. Planning to move more and more of my workflows into the command line and skills feels like a nice iterative way to do that. Going back to the terminal feels like coming full circle. Recollections of using MS-DOS growing up (although running programs rather than workflows, and obviously all human-driven).

@ChrisLovejoy_ @readwise @obsdmd Been thinking on this myself. I used to have hundreds of rss feeds or newsletters converted and a backlog. I somehow either lost or subconsciously just ignored them as the amount of stuff was overwhelming. The engaging with content actively I think is the key - and repetition.

One thing I'm looking to update is my 'content consumption' pipeline. Various iterations of this over time but until now basically: - save article to read-later (e.g., @readwise chrome extension / mobile app, @obsdmd web clipper) - read it and highlight interesting parts - auto saved as MD file inside @obsdmd vault for future reference Feels like this is too passive though and I want to bring LLMs into the process. Am considering something like: - save article to read-later (TBD where this is stored) - use skill to review and engage with these articles. summarize, read segments, discuss the ideas. - rather than 'highlights' focus is more on storing 'learnings' / 'takeaways' More active process, so should internalize the ideas better. And maybe use LLMs to re-surface and engage more with the ideas later too. x.com/ChrisLovejoy_/…

@propoflurane @DrRJWebb I don’t know the product but I guess It will make a decision for you based upon the three biomarkers whether it’s viral or bacterial. Logistic regression if they’re smart, probs neural bet based upon pt data.

@DrRJWebb They are quite open about what the blood tests are: TRAIL, IP-10 and CRP. The machine learning bit is a bit more opaque. me-med.com/memed-bv/

What is this blood test, and please tell me we’re not bleeding every febrile child who attends ED? This just reinforces over reliance on healthcare, when the majority could/should be looked after at home if health education was better.

@0xb33bs @ChrisLovejoy_ @shawnup No but I’m working on it.

@shawnup Is there any platform today that replicates this full workflow? Like analytics + RL + human curated test cases? I feel like that’s the missing piece for getting really good enterprise agents. (like this workflow from @ChrisLovejoy_ )

🧵 We acquired OpenPipe! OpenPipe's ART framework makes it easy to beat foundation models with smaller open models on real problems, using reinforcement learning.

@ChrisLovejoy_ So, have you broke that single digits yet ? Waiting with anticipation over here ;)

so close @ChrisLovejoy" target="_blank" rel="nofollow noopener">youtube.com/@ChrisLovejoy

@ChrisLovejoy_ I like that heuristic - i would also add you can compare on a test set on someone who’s trained up vs domain expert. Still subjective but a comparison can give confidence you’re not missing something. Expertise can also be prohibitive depending on outcome targeting.

Healthcare and law are clearly specialized verticals - but what about the rest? It's critical to incorporate domain expertise when building vertical AI products. But when do you need 'true' domain experts to review AI outputs and when can you just upskill a 'normal' person? At Anterior our product performs nuanced clinical reasoning. Using medical doctors to assess the AI is critical. My friend writes all the LLM prompts for one of the top meeting summary apps. She has no formal background in meeting summaries (lol). There's clearly a spectrum from "domain expertise is critical" (e.g. clinical reasoning, interpreting legal contracts) to "anyone could be the domain expert" (e.g. meeting summaries, email generation) So for any particular use case, what is the threshold for determining whether you need a "true" domain expert or not? My heuristic is: would it take more than 2 months for a smart, switched-on person (with no prior expertise) to generate top 5% judgement? If so, you probably need a domain expert.

@ChrisLovejoy_ This is supremely on point for my own work and covers a lot of my own UX instincts building the tool out. Thanks Chris very useful to hear more about your experience. Curious to hear more about the improving the product at time of review. Interesting idea.

This tweet was popular so I elaborated on it in a full article: chrislovejoy.me/review-dashboa…

@ChrisLovejoy_ Couldn’t agree more. Stop belittling your skill in other domains by using such language. Be what you want to become. In tech cvs I strongly urge people to remove pretty much everything clinical to no more than a paragraph. It does not define the role of a software dev.

For disillusioned medics looking for a career pivot: A mindset shift that helped me transition was to focus on meeting the bar for a different role regardless of the fact that I'm a doctor: - Not a "clinician who codes" but rather a "software engineer" (who happens to be a doctor) - Not a "clinical data scientist" but rather a "data scientist" (who has experience analysing medical data) - Not a "clinician scientist" but rather a scientist who's done interesting clinical research The aim is to meet the bar independently - and the background as a doctor to be a bonus (for non-technical skills, etc). Being a "clinician who codes" helps you make friends with other clinicians who code but not to land a serious job. Pitching yourself in a "doctor first" way makes it hard for hiring managers to know where to place you. You risk being seen as a defective software engineer / data scientist / graphic designer / whatever.

@ChrisLovejoy_ @AnteriorAI So there’s multiple outputs to ensure accuracy/faithfullness.. (RAG used) also llm answer vs student answer. I have a good sense of where the failure modes are but really wanting to encourage llm trace reading, I’m coming at it as domain expert with the chops to build.

@ChrisLovejoy_ @AnteriorAI We’ve been developing llm driven interactive experiences within our platform that engages students with pathophysiology content with a vignettes and veering into clinical depending on the institution. There’s multiple llm outputs and llm validation between student/llm answer.

Building a dashboard for domain experts to review AI outputs is critical for domain-specific LLM apps. Principles we followed at @AnteriorAI when building ours: (1) Optimise for providing all relevant context as intuitively as possible Our doctors and nurses say the hardest part of reviewing AI outputs is getting the same context the AI has. Our interface makes the medical record and guideline document accessible on right hand side with the AI outputs laid out on the left, providing clear mental separation. (2) Design the user review flow from first principles, not their existing workflows Different domain experts develop their own workflow over time through habit. Our opinion is that there is an 'ideal' workflow which you can guide them towards, optimising for both speed and accuracy. (3) Don't just get correctness, get failure modes and suggested fixes at the same time You want your reviews to help you improve your system, not just tell you how you're doing. Measuring multiple things at once prevents your domain experts from needing to keep re-loading context for an AI output. In our dashboard, you can see 'correct/incorrect', 'add domain knowledge' and 'tag failure mode'.

@ChrisLovejoy_ This is also great, I like the raw format and it’s great to just get thoughts out there in the world, takes courage! I liked your AI eval note, I’ve been majorly hovering over doing it for next cohort. Looks like a high quality course.

Just added a 'notes' section on my personal site (chrislovejoy.me/notes) The idea is to jot rough notes here which haven't been 'polished' into articles. Somewhere between a fully open 'digital garden' (where all your local drafting notes are publicly available) and a polished blog. So I still make the explicit decision to 'publish' but the content itself still acknowledged as a rough note. (Inspired by the digital gardens of @andy_matuschak, @neuranne, @tomcritchlow, @Mappletons and others) I tried with full public publishing before but found I wrote slower when I knew everything goes public by default. Just one note for now (notes from an AI evals course I just finished) but will add to here over time.

@ChrisLovejoy_ @karpathy Great find, will defo be using this.

TIL that you can just replace github.com with gitingest.com to convert a repo into a single block of text that you can paste into an LLM Just tried it with my YouTube video finder repo and worked perfectly h/t @karpathy from github.com/chris-lovejoy/… to gitingest.com/chris-lovejoy/…

@ChrisLovejoy_ @aiDotEngineer @AnteriorAI Love this Chris, really cherish you sharing these details. Looking forward to hearing you talk about it when I get a chance.

Making an LLM product MVP is easy -- building a product that works at scale is hard. Last week for the @aiDotEngineer conference I shared our journey at @AnteriorAI from prototype to production, now serving health insurers covering >50M Americans. Our special sauce? Real-time reference-free evals powered by expert clinical reviews. Let me explain...

@ChrisLovejoy_ Damn impressive Chris. I’m yet to make the Jump, what’s working for you ? Would love to see a note about your experience!

that feeling when you roll up to work after vibe coding 3 apps and 10 PRs on the weekend

@ChrisLovejoy_ Meta af

So many people out here using AI to write about how to do AI 😅

@ChrisLovejoy_ Good question it’s been on my mind! Aakash Gupta has written some good stuff on Substack but not specifically llm focused or healthcare’s. Can fire some of his articles to you as I subscribe if that would be of use. There are few books coming out, ‘building AI-powered products’

Anybody making good content on Product Management for LLM products? ie. what makes a good PM for these types of products, how that's different to 'conventional' products I can only find courses from big corporates like IBM

@ChrisLovejoy_ What tools have you found most helpful Chris ? I’ve also had similar thoughts about content creation.

Passive content creation might finally be possible thanks to LLMs In 2020, I made videos and articles on evenings and weekends alongside a full-time job. 1 video and 1 article for the full year. After a while, I built some momentum: ~10k YouTube subscribers ~1.5k email subs ~1M total video and article views top of hackernews x1 However, making content took time, which made it harder to actually do interesting stuff. (I wasn't at a scale where hiring a video editor or writers made sense.) So I decided to pause everything and live more first - and I can always come back with more life experiences to share. Since then I've toyed with the idea of sharing more but it's never been a high enough priority. I'm now convinced that LLMs can make it pretty frictionless to get the ideas out of your head and share them with others (while you keep doing cool stuff). For starters: - record every meeting you have and get an LLM to lift out the interesting ideas - convert writing between formats - eg. tweets to article or vice versa - think out loud to an AI (like @Voicepal_me) and get prompted for insights

@ChrisLovejoy_ Nice Chris! Will check out the gaelic version!

Just made Language Coach - my first #customGPT Pick any language, have a conversation, and then receive detailed feedback on how you can improve chat.openai.com/g/g-gVvgJcX76-…