Charles C. Lee

Recursive Language Models with Elixir. Of course. The solution was in front of us all this time.

if you’re building a RAG system in 2025 just build a good search engine backend + let the model query it

All I can say is we've had a blast working on this iteration - 🚀

How structured outputs work under the hood (via breakout at OpenAI DevDay) Guess why the first structured output request is slow, but the 2nd+ is fast? Engineering: * Unconstrained token decoding isn't good. The model could pick any token. * Limiting which tokens can be produced next is what works. They use "token masking" for this. Constrains the token which are picked at the end of sampling * After generating probabilities with a forward pass, ignore (or mask) tokens you know aren't applicable. * Mask tokens need updating at every step of the inference. Needs to be really fast. Mask determination happens on the CPU. Needs it to be as quick, as quick as time between tokens. <10ms * They build up an index with the JSON schema you apply to speed up mask making * Index Making: Grammar > Parser > Iterate over all tokens and parse states > Index. Index once, cache later. First query to structured outputs go slow, then the 2nd+ query is quick. * Recursive schema makes this hard. But they believe they will be a big deal so they are moving towards it. * CFG: Context Free Grammar approach. Regular expression + stack. Their attempt at indexing Research * Retraining and fine tuning gets you to 100% schema adherence API Design * Additional properties: What do you do with properties that were note defined in the schema? Decided to disallow them * Design principle, be explicit instead of implicit. Requiring developers to set properties. It's harder to use, but communication is more clear * Design principle: All properties are design by default. It makes optional params nullable * Decided to generate fields in the same order they were defined in the schema, this helps with requiring chain of thought for structured outputs

We made a step-by-step tutorial on how to finetune Llama-3 with Google Colab & deploy it to @Ollama Tutorial: docs.unsloth.ai/tutorials/how-… Colab notebook: colab.research.google.com/drive/1WZDi7AP… Blog post & video coming soon. 🦥

❓What is an agent? I get asked this question a lot, so I wrote a little blog on this topic and other things: - What is an agent? - What does it mean to be agentic? - Why is “agentic” a helpful concept? - Agentic is new Check it out here: blog.langchain.dev/what-is-an-age…

This is fast. Chrome running Gemini locally on my laptop. 2 lines of code.

@mortenjust Query expansion like this could be promising

This is an interesting overview of what it is like to actually build production software with today's LLMs. It also shows how weird LLMs are to work with from a software perspective & how much we have to learn. It is why I stress co-intelligence for now. oreilly.com/radar/what-we-…

@CharlesSchwab loving the updates to the web platform! When will existing users be able to use ToS with their Schwab assets?

@Apple the iCloud web beta is fantastic. Hoping this comment serves as one more data point to keep whatever investment in this type of work going. 🙏

@overstimulat3d @rails @stimulusreflex @flydotio @realkeydb +1 to a breakdown. Also curious if you had any thoughts around hotwire vs stimulusreflex (ie. dev ux).

I made a quick mini-app last week to see what low-latency geographically distributed pub-sub looks like in the context of multi-region web applications with live DOM updates. Built with @rails, @stimulusreflex, @flydotio and @realkeydb (just for fun). 😊 global-presence.fly.dev

@LtSquigs Gotta get one of those mobos that have visual error codes on board.

@wongmjane Really hoping this is an instance of "it's gonna get worse before it gets better".

Still zoning out trying to process the past 24 hours. Anyone else?

@mariaersh @ian_eck @jtmorehead_ @joulee Agreed with the hashtag based organization. Trade offs, oh trade offs 🙃

@ReallyChar @ian_eck @jtmorehead_ @joulee Totally agree, I use Bear.app a lot and offline mode is a huge differentiator from notion. One issue I find with Bear is if I don't hashtag a specific note it could be hard to search for a that note later on. It can get cluttered pretty quickly.

Which tech products do you use because it's more convenient, even though you know there's something higher quality out there that does the same job?

@mariaersh @ian_eck @jtmorehead_ @joulee I love notion but not having an offline mode like Bear.app or Evernote makes it feel slow since it seems to be fetching all the pages on load.

@ian_eck @jtmorehead_ @joulee I would disagree for a personal use case, haven't used it for professional work though. Notion substituted multiple excel spreadsheets that I keep and made it more use friendly for me.

@centralcomputer No need to be sorry, stock is limited. Glad the shops were taking safety precautions. Better luck next time. 🤞

@ReallyChar I'm so sorry 😔

Still looking for a RTX 30 series GPU? Look no further! 😉 We have restocked all of our stores with RTX 30 series GPUs. Get yours today when you visit one of your local Central Computers! centralcomputer.com/nvidia-30-seri… #RTX30 #RTX3080 #blackfriday #deals #GPU #videocard #holiday

@AMDRyzen absolutely killing it with the new line up.

@AndrewYNg DLSS 2.0 is insane for future engines in gaming & media - nvidia.com/en-us/geforce/…

What are your favorite AI technology breakthroughs or demos from the past year? Would love to hear particularly if there's something you think not many others have heard of yet.