Frontier LLM

Model training was sufficient, however results are lackluster. Back to retraining and modifying the architecture a little bit.

We're evaluating results from our first prod training run!

75% done on production training!

Training is now taking place on a GPU cluster! Flying through the epoch's

Want to try EchoNest in alpha for free? API access forms just dropped: frontierllm.net/access.html Apply today!

What if a 100M-param model could follow instructions efficiently on a plain CPU, no GPU, no massive pre-training stage? That's the idea behind EchoNest, our new architecture paper. Recurrence for local structure, attention for the global view. frontierllm.net/resources.html

training may be restarted on faster hardware tomo/the next day will keep you bois posted everything is looking very, very good though

the whole open-source LLM world quietly agreed on one architecture and nobody really talks about it. crack open llama, mistral, gemma, qwen, phi- pick any of them, and you're basically staring at the same model. rmsnorm, rope, swiglu, grouped-query attention. pre-train on a few trillion tokens, instruction tune on top, ship it to a gpu. it works. it's genuinely a great recipe. but it's one recipe, and at this point everybody's just cooking it slightly differently. and that recipe isn't neutral. it quietly assumes a bunch of stuff: billions of parameters, fat accelerator memory, and a two-stage life where you spend a fortune pre-training on raw text before you ever teach the thing to follow an instruction. all of that is completely reasonable if you're a frontier lab. it's just not the only point in the design space, and it's the only point anyone's building for. so we built something for a different point. it's called echonest. it's a 102.8M param model, and the target is dumb: run well on a commodity cpu at the edge. no accelerator. tight memory. the kind of place you can't just drop a 7B model. the usual move for "small model" is to take a big-model architecture and shrink it. we didn't want to do that. a tiny llama is still a llama wearing the assumptions of a thing 100x its size. if the deployment context is genuinely different, the architecture should be too. the core idea is to stop pretending one flat stack of identical blocks will magically sort out local vs global processing on its own. big transformers get that separation to emerge by throwing depth and scale at it. we don't have depth and scale. so we just built the separation in, explicitly, with three nested tiers that each do one job: echounit, the local tier: lstm style recurrence handling short range, order sensitive stuff. the place where sequence actually matters. micronest, the middle: aggregates those local bits up into phrase and clause level features. macronest, the top: attention. but only here. attention is quadratic and expensive, so you spend it once, at the level where global mixing actually earns its keep, instead of paying for it at every single layer. bringing recurrence back is the part that'll make people twitch, because the whole point of "attention is all you need" was killing recurrence. OK. but recurrence lost for specific reasons: it blocks parallel training and it's bad at long range. now look at our setup: 512 token context, running on a cpu. the parallel-training penalty barely shows up, and in exchange you get a compact hidden state and positional order basically for free. the tradeoff that made recurrence a bad idea in 2017 quietly flips when you change the hardware and the context length. the normie talk: no pre training at all. none. echonest trains straight on ~100k instruction–response pairs (alpaca, python code instructions, codesearchnet) from random init. loss only on the response tokens. adamw, lr 1e-4, ema 0.999, the whole run on cpu. obviously you pay for that. you give up all the broad world knowledge that trillion-token pre-training buys you. what you get back is a training process that's small, cheap, fast, and actually controllable, tightly scoped to behavior instead of trying to absorb the entire internet. if what you want is a predictable, domain-focused assistant you can train and re-train without a pre-training budget, that's a real trade, not a downgrade. now the part i have to be honest about, because everyone skips it: there are no results yet. zero. this is v0.1-alpha, training is literally still running, and i have no perplexity numbers, no benchmarks, no latency measurements to show you. right now the case for echonest is architectural and philosophical. it's a hypothesis with a model card, not a win. and i know exactly what would actually settle it, so i'll say it out loud: echonest vs a parameter-matched vanilla transformer- same ~100M params, same 512 context, same instruction data- measured on cpu latency, memory, and task quality. that head-to-head is the whole ballgame. until it exists, treat all of this as a claim, not a result. the bigger thing i actually care about, underneath the specific model: for a growing pile of edge and cpu deployments, the right model might not be a smaller frontier model. it might be a structurally different one. the monoculture is efficient but it's still a monoculture, and "make it smaller" is not the same as "design it for where it runs." echonest is a first swing at designing for that context from scratch instead of shrinking down from the top. roadmap, quickly: finish v0.1 and ship an eval suite, then v0.2 with inference-optimized weights, then v1.0 with a safety pass and an api. full technical report with reproducible specs and that parameter-matched bake-off comes with the real numbers.

We're running our first training job in the open. Live train/val loss, perplexity, and grad norm for EchoNest-1 — all on the public status page. Building in public from epoch zero. frontierllm.net/status.html

@mmmatt Followers will be the first to know about early access! <3

hey bois, we made an official account for frontier so i can talk more about other stuff on this account. All @frontierllm posts are official give a follow if you wanna keep up with everything there. ty bois for supporting as always. lets change the world

Introducing Frontier LLM — foundation models engineered for the enterprise. General reasoning, coding, and domain-tuned models, with the agentic infrastructure to deploy them in production. Your infrastructure. Your data. Your terms. frontierllm.net