@henrytdowling Whispering sweet nothings to my computer next to my coworkers >>>
English
Ben Geist
751 posts
Ben Geist
@b_geist
Research Eng @ramplabs / physics + math nerd / Kate Bush fan
Brooklyn, NY Katılım Temmuz 2019
457 Takip Edilen984 Takipçiler
@b_geist im sorry lip syncing to a computer is too much aura loss i would never use that
English
Million dollar startup idea I’m giving away FOR FREE! Lip reading software that connects directly to Claude code
swedishasian67@michellezfr
Wearing noice cancelling masks to talk to Claude is crazy
English
does anyone remember this Sakana paper? integrate documents into weights via LoRA & hyper-networks
bc how are we going to integrate massive amounts of enterprise knowledge or user specific context into models continuously over time?
idk - great open research question. a lot of this will come as contextual retrieval at the harness layer
but some will happen at the model layer and there’s a ton of infra & training logistics + testing if it works at that layer
tbd what ends up winning, fairly sure it still needs a yet to be shown research breakthrough in weight space integration + a lot of good methods in search + harness eng
Sakana AI@SakanaAILabs
We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible. pub.sakana.ai/doc-to-lora/ By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks. Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts. To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document. In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights. Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore. Doc-to-LoRA Paper: arxiv.org/abs/2602.15902 Code: github.com/SakanaAI/Doc-t… Text-to-LoRA Paper: arxiv.org/abs/2506.06105 Code: github.com/SakanaAI/Text-…
English
@rennyzucker I promise to save you time and money and at the same time be a NY truther 🫡
English
File under: things you don’t like to see from a bank you have an account with
Ben Geist@b_geist
NYC is where stupid people act stupid, SF is where stupid people try and act smart. This is why i like NYC
English
@b_geist @cursor_ai @dwarkesh_sp was a really nice talk, I hope more people work in similar areas as its going to have very real consequences in the world out there
English
Spoke at @cursor_ai’s conference, compile, yesterday hosted by @dwarkesh_sp on my research how introducing information in the latent space into LLMs can make the underlying models more efficient. Haven’t written on a chalkboard in years, was a great conference and experience 😄
English
COT is a bit like graph traversal, right? We start at the node encoding the query’s information in “fact space” then the model iteratively moves to the nodes with the highest probability and backtracks when it is some distance away from the theoretical node encoding the answer
English
@lu__jasper @RampLabs Left this up to future work but I’m interested in exploring with text-to-Lora techniques on a task description
English
Highly-recommended read.
Even with all the research, fine-tuning is such an underexplored problem.
Based on what I've seen among the top AI-powered orgs, we are on the cusp of a fine-tuning revolution.
Agentic fine-tuning is going to dramatically change things in AI.
rahul@rahulgs
In many ways, finetuning or RLing a custom model is a bet against model progress and scaling. It's to choose to say "we don't think there's going to be a good enough base model for this task anytime soon, so we're not going to wait" with oss release velocity these days, its a hard tradeoff It's easy to end up on a custom model with an outdated base (Kimi 2.6 is only a few months old) So we fixed it - PorTAL lets you swap base models quickly, allowing your learned task specific behaviors to port to new models as they come, no matter how fast
English
@binbinsh_ @RampLabs By construction the bulk of parameters are shared are frozen, would say that is pretty key
English
Introducing PorTAL: Portable Task Adapters for LLMs. A novel recipe to cheaply port fine-tuning between models. It matches per task LoRA accuracy at half the cost, lowering the switching overhead of adapting tasks across LLMs.
At Ramp, every new model release used to mean retraining our fine-tunes from scratch. PorTAL learns the task once, then efficiently refits it onto any new base model, even across model families.
English
@cclark @combin8or @RampLabs *completely separate holdout set (but still from the task datasets we trained on)
English
All of the datasets we trained on had >2000 examples in them. We capped our training data to 2000 examples per task (here each task is just a dataset). It was first trained on Qwen 1.7B + 4B on the training set, then calibrated, again on the same training set for an unseen model. The eval set was a completely separate holdout set from these 14 datasets.
English
Continuous learning only works if that learning can transfer across base models. Why stay on last month's model that knows your processes when this month's is far smarter? I researched this for months and ended with PorTAL. It ports finetunings across base models by refitting only a thin converter, effectively a separate memory bank that attends to any base model. Check it out!
Ramp Labs@RampLabs
Introducing PorTAL: Portable Task Adapters for LLMs. A novel recipe to cheaply port fine-tuning between models. It matches per task LoRA accuracy at half the cost, lowering the switching overhead of adapting tasks across LLMs. At Ramp, every new model release used to mean retraining our fine-tunes from scratch. PorTAL learns the task once, then efficiently refits it onto any new base model, even across model families.
English
@AkashBajwa96 Agreed! The goal is to optimize this process to make finetuning frictionless
English
This is a pretty big deal wrt the TCO debate of fine-tuning vs frontier model release cycles
Ben Geist@b_geist
Continuous learning only works if that learning can transfer across base models. Why stay on last month's model that knows your processes when this month's is far smarter? I researched this for months and ended with PorTAL. It ports finetunings across base models by refitting only a thin converter, effectively a separate memory bank that attends to any base model. Check it out!
English