NullFoundry

nvm my auto updater is broken lmao

hey guys, I'm a bit off local models these days because I was testing ChatGPT with MCP, and I updated my llama.cpp to version b10000, but I'm only getting 300 t/s on prefill. I haven't noticed any changes on my end. Is anyone else experiencing the same?

@OpenAI chatgpt is doooooown

@Hikari_07_jp Yeah its annoying

Please stop sending completely irrelevant replies just to farm impressions. It's also annoying when you drop jargon halfway through. These accounts look automated and claim to share know-how, but they're just creating garbage.

Tip of the day You can expose your Hermes agent over the MCP protocol and connect it to ChatGPT. This repo makes this possible: github.com/asimons81/herm… But be careful because this can lead to some seriously bad consequences. In my case, I made a small OAuth server proxy, so I'm protecting everything on my side. I am using ngrok as a tunnel. Downside: you don't have the history within hermes agent Upside: you don't spend tokens and chatgpt has more context of your stuff. Have you tried this before?

ngl, lots of AI posts are leading to misunderstandings in this world of local models. Some guys don't even check the facts before posting them

@HuggingModels bruh

Big news for AI on a budget. GLM 5.2 Colibri int4 is a Mixture of Experts model that runs entirely on your CPU. No GPU needed. It's fast, efficient, and opens up advanced language capabilities to anyone with a standard computer. This is a game changer for offline AI.

Gonna need to test it

“TRACE: Capability-Targeted Agentic Training” got Spotlight @ ICML AIWILD 🎉 Beats direct RL, GEPA, & synthetic-agent data on SWE-Bench Verified and τ²-Bench. TRACE-Qwen3.6-27B tops GPT-5.2-Codex, GLM 5, & Claude 4.5 Sonnet on SWE-Bench. Co-led with @TarunSures41845. Thanks to @JonSaadFalcon and our advisor @Azaliamirh. Details below 👇

@Hermesage make gpt live possible and you will gain my loyalty

I am very impressed with Grok 4.5. I'm getting good results, pretty consistent. This video from @bijanbowen shows some impressive things. The car game is pretty good, btw. youtu.be/DYDF_UZc0n4

🖥️Damn Local AI community. Google comes through. Gemma 4 is now fully on-device in React Native! 🎯 Run Google’s Gemma 4 100% offline in your cross-platform mobile apps with hardware acceleration: ⚡ Vulkan delegate on Android ⚡ MLX delegate on Apple Silicon 🏃‍♂️How to get started: ✦︎ Use the open-source library: react-native-executorch ✦︎ Load models easily with the useLLM hook ✦︎ Supports vision + tool calling locally 💰Cost: ✦︎ $0 API cost — fully on-device ✦︎ Main cost = device performance (best on newer phones) Best for: ✦︎ Vibe-coded React Native apps that need strong local AI with privacy and no internet dependency. 🔗 Demo & Docs: github.com/software-mansi…

@akshay_pachaar Is it possible use it on llama cpp?

14x faster and 90% cheaper LLM inference. (100% open-source, KV cache management) your LLM does the same expensive work over and over. every request, it re-reads the same system prompts and the same documents from scratch, even if it processed them one second ago. token prices keep falling, but agent workloads re-send so much repeated context that the bill climbs anyway. LMCache fixes this. it's an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM. here's how it works: LLMs recompute their understanding of the same content on every request. the same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day. LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine. the engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference. it also reuses cache beyond exact prefixes. their CacheBlend technique (EuroSys 2025 best paper) keeps RAG documents cached no matter what order they appear in. on H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. and since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly. GitHub repo: github.com/LMCache/LMCache (don't forget to star 🌟) i wrote a full breakdown of KV cache management that walks through why 𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 silently breaks in three common cases, the 𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 behind the 14x speedup, 𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱, and how to turn every document in your knowledge base into a reusable cached asset. the article is quoted below.

@TheExpanseRPG just let me play it

Since we shared some info about origins, we saw that some of you have already decided who they will play. So, how about a poll? Which origin will you try first?

@bijanbowen looking forward

took a power nap, gotta do a grok vid now

guys I found this huggingface.co/froggeric/Qwen… It seems to have fixed my problem. It feels smarter. Of course, I could be biased, but for now, it seems to be working.

I have to say this: Qwen 3.6 27B - Q4_KM starts to get dumb after 100k context (using Q8 for KV Cache) Am i missing something?

@TheAhmadOsman @bijanbowen low bandwidth?

@bijanbowen Absolutely not “for a noob” This will make them run away from Local AI like you’ve never seen before

It has never been a better time to leave the paid intelligence tokens providers and self-host models yourself Applies to individuals Applies to businesses Applies to enterprises

@Hikari_07_jp @pedrambirack you guys are causing trouble between me and my wife lmao

@pedrambirack You're absolutely right. If you're going to be tuning models, having a massive amount of storage is definitely the way to go. It's worth the investment.

For local model work, it is not just about GPUs. Once checkpoints start turning into multiple terabytes, storage becomes part of your freedom too. If the drives are yours, the workflow depends a lot less on someone else’s limits.

@BeeferShitlord could you elaborate?

@nullfoundry i don't have this issue, but i have a multi layered memory system that injects relevant parts of memory into the conversation when needed amlnd i flush kv after so many turns which triggers a summary. so i keep it clean to avoid that