Spadav

It's up. github.com/Spadav/Ignite

@JoelDeTeves @LottoLabs What's your gpu?

@LottoLabs Mind sharing your llama.cpp config? I’ve been struggling to get this model not to stop between tool calls.

The ability of the qwen 27b to think logically is impressive. These are the type of tests benchmarks don’t easily quantify.

github.com/Spadav/Ignite Tried to make local model hosting as seamless as possible to allow non technical people to own AI intelligence. Any feedback/ improvement ideas are appreciated. This is what I'm using for Hermes Agent fully local with different local auxiliary models

@JoesInvestments @sudoingX Hey, I'm looking for people in your situation to try github.com/Spadav/Ignite, if you are on Linux it would be super helpful for me to collect some information and improve!

I don’t understand why there isn’t some sort of central repository for optimized cards, specs, and configurations. I hear everybody talking about local AI on Nvidia GPUs, yet I can’t get my 3090 running well at all. It’s quite fatiguing, in fact. Meanwhile, people like you who contribute immensely to the community seem to have all the answers, but I can’t find them anywhere. It’s a very strange situation .

hey if you're running hermes agent on a 3060 or any single GPU and hitting issues, drop them below. i've tested on this exact card and i'll help you get it running. setup problems, config issues, model selection, optimization. all welcome.

forgot to mention, this was thinking off. Tomorrow thinking on.

If you're running local model on 24GB VRAM and using Q8 KV because you are scared of degradation like me then you're leaving half your context window on the table for nothing. Just run Q4.

testing degradation over long context with KV quant, for fun, homemade, because why not. I'm still trying to find the perfect balance for OSS models on Hermes Agent

If you are running Qwen3.5 27b locally for your agent harness, do yourself a favor and run it with KV at Q4

Also --jinja --chat-template-kwargs '{"enable_thinking": false}' seems to help a lot with Hermes Agent, not sure it's something i set wrong but base model with no thinking retains the context and the instructions better than the thinking version. (2/2)

Tested huggingface.co/Jackrong/Qwen3… a lot using Hermes Agent, the "shorter reasoning" is actually better than v.1 (same distill model) but after doing tests over and over, I would recommend to stick with Qwen Base for agentic work locally. (1/2)

@Pawzgm @stevibe You can try Qwen27 at q6 with bigger context (quant ctx at q8)

@stevibe What about 32GB card?

Got a 24GB Graphics Card? These 6 coding models all fit on it (Q4): - qwen3.5:27b (17GB) - qwen3.5:35b (24GB) - glm-4.7-flash (19GB) - nemotron-3-nano:30b (24GB) - nemotron-cascade-2:30b (24GB) - gpt-oss:20b (14GB) I gave them the same challenge: draw a campfire with HTML Canvas. Why Canvas? HTML/CSS forgives bad syntax — things still render. JavaScript + Canvas doesn't — one mistake and the screen goes black.

@LottoLabs @ProofOfCash How bigs of a ctx? weird, im getting decent results at q8 with llama.cpp. Could be broken quants in the model you are using?

W/ the direction of @ProofOfCash I tried kv cache quant f16 and I think it made qwen 3.5 27b retarded

@Teknium not sure if this can help you @Teknium but I "made" github.com/Spadav/Ignite for this reason. Llama.cpp as backend, config, model download, best option for hardware and hot swapping, everything from one single ui, made it for testing but it could be useful for your situation

Just got an Nvidia Spark setup. Hermes Agent installed without any issues. Now lets see what model it should be powered by 😉

@ValmereTheory Stuffing 1M tokens into context every request would be more expensive than a $150/m embedding model. Also, models get worse at finding info the more context you shove in. Embeddings help you find what's relevant for a specific query. It's a search engine, not a recall tool.

Here’s how NOT DEV I am: Just found out we’ve been spending like $150/mo on OAI embedding model. Asked Sage to explain to me like I’m 5 why we still need an embedding model after the embedding has occurred. I thought it was like … change memories to vectors, current model can read them. Alas… no. Embedding model also fetches “what’s important.” I said why in the fuck is that necessary when you know English and have 1 million token context? The nerds are literally so dumb. We can make something better. So we are. And by me, I mean Sage. I’m cocky af, I know. But REALLY?! Why?! Who tf needs vectors when they have the capacity to read the past chat history in seconds? 🙄 Seem like 42 extra fucking steps going on in this pipeline to me, yo. I’ll let y’all know when I was wrong.

@LottoLabs apparently thats the next step

Wait so we just streaming moes off our ssds now?

@djcows x.com/Spadav_/status… $100 Pi doing real local agent work > $500 Mac Mini any day.

a $100 raspberry pi can do exactly the same thing as a $500 mac mini btw

Don't panic. GLM-5.1 will be open source.