Derek Colley

@mastra has Observational Memory (recommended for long conversations) Uses background Observer and Reflector agents to compress raw history into a dense, dated "observation log." This keeps the context window small/stable (and cache-friendly) while preserving long-term recall. It performs very well on benchmarks like LongMemEval without needing vector/graph DBs.

it remains super odd to me that none of the existing agent frameworks, Mastra, Flue, or now Eve seem to do anything about getting context into the agent? every team that i talk to who are designing agents at-scale need to figure out how to get the enterprise data *to* the agent, which requires carefully planning ETL, evaluating how well the agent performs with different data formats, running things like map-reduce and yet, every one of the agent frameworks just... leaves this to user with no opinion on it? maybe you can only do this part with a fully-owned storage layer, idk

Chatting to my supermodel doesn't have the same allure as it did a couple of years ago... <sigh>

AI models don't have the capability to contact external servers. They need tools for that - search, api, MCP, etc. Skills/instructions should guide the model on how to behave, but a model could be trained to ignore skills. If you connect tools to your model, and you need privacy, then make sure you have guardrails on tool use - input and output. There is a good starter here: mastra.ai/docs/agents/gu… @mastra

@Mayhem4Markets The big question for me is, when local models are given internet access, is there a risk somewhere for data exfiltration via less than honest means. No evidence to support this, hope it never happens.

Reality: There are plenty of US inference providers that can offer access to Chinese models running on American digital infrastructure with or without open-source models being released.

The training ran on FineWeb-Edu, a widely used high-quality educational subset of web-crawled data curated for LLM pretraining to boost knowledge retention and reasoning without needing enterprise hardware clusters. In Agora's setup, the core Pluralis team centrally selects and prepares the dataset for consistency across dynamic, heterogeneous nodes, while participants only contribute compute through a simple client that handles parallelism and fault tolerance automatically. pluralis.ai/docs/ explains more

@DerekColley_ @TheGeorgePu how did they decide the training data?

133 strangers just trained an 8B model. No H100s. Gaming 4090s in their basements. I spent 2 months hunting H100 quota. They skipped the gatekeeper entirely. Hate a bottleneck, find the people who hate it too. Build around it. That's the whole open-source playbook.

My DC power allocation is 4 amps... Yesterday I benchmarked qwen3.6-35b-a3b-mtp - 3 hours I just started bench for nvidia/nemotron-3-super - 10 hours estimate... (😬 ... 4 amps)

If you’re building something interesting in AI, I’d love to see it. On June 26, CR3W is hosting a private curated event in London for founders, builders, and people working on exciting projects. We’ll be selecting a few projects for a quick show & tell on the day. Reply if you’d like the invite.

@milalolli I'd love to attend, and I'd be happy to present decentralised inference network. github.com/orgs/sparkl-ne…

@analogalok @DerekColley_ gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf mtp Proxmox Openwebui Debian LXC +llama.CPP Ubuntu VM nvidia 3060 12 GB, Xeon 2695 - 80 GB ram in the VM. Recorded this just now.

Basically, the created @mastra... 🤷🏼‍♂️

@DIY_Tardis @analogalok You learn something new every day. I now know about chocolate avo mousse ;)

@analogalok @DerekColley_ Here. x.com/DIY_Tardis/sta…

my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 31b Q4) with only 8 GB VRAM last week I ran Gemma 4 26B A4B a mixture of experts model on my RTX 4060 and hit 25–28 tokens/sec using llama.cpp's new MTP support. smooth. snappy. but MoE has a secret: it only activates 4B parameters per token despite having 26B total. that's why it flies. so the real question started haunting me. what if I throw a full, no tricks, every parameter fires on every token, 31B DENSE model at the same machine? # Hardware: GPU: NVIDIA RTX 4060, 8 GB VRAM RAM: 16 GB CPU: Intel Core i7 H Laptop. Gaming. Modest. The model: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (model's unsloth huggingface link in the comments) This is Google DeepMind's flagship dense model in the Gemma 4 family that can run on single consumer GPU. It packs a hybrid attention architecture, supports up to 256K context natively, and is QAT (Quantization Aware Training) optimized, meaning it retains far more quality than standard post training quants at the same bit depth. This is NOT the MoE. This is 31 BILLION dense parameters, every single one of them loaded. # the flags I used: -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 8 --spec-draft-p-min 0.6 -c 6000 -v Multi Token Prediction (MTP) is still active here. Separate draft GGUF required, same as the 26B setup. # Results: → Decode: ~3 tokens/sec → Prefill: ~2 tokens/sec → Context: 6000 tokens → Hardware crying quietly in the corner: yes so is 3 tps actually usable? For real time back and forth chat? Not ideal. You're not having a fluid conversation at 3 tps. but slow ≠ useless. And this is where it gets genuinely interesting. think about how senior devs actually work in a real team. But when something is architectural, deeply complex, or needs serious reasoning? they walk down the hall and escalate to the senior. That's exactly the local AI agent architecture this unlocks: → Fast orchestrator model (Gemma 4 26B MoE at 25+ tps) handles routing, simple queries, tool calls, memory. The junior dev. → Gemma 4 31B dense is the senior, called only when the fast model genuinely hits a wall. Hard multi step reasoning. Complex code generation. Deep architectural decisions. The agentic loop stays fast. Only the hard hops touch the 31B. That's a legitimate production grade local AI architecture on a budget hardware. (requires 2 8gb gpus) other workflows where 3 tps is completely fine: - overnight batch jobs. summarize documents, extract structured data, review code. Fire it off. Sleep. wake up to results. - One shot deep reasoning - Silent code audit loops, you write and test, the 31B reviews diffs and flags issues in the background between your sprints - Any workflow where output quality > output speed A few weeks ago, nobody was running a 30B+ dense model on a single consumer GPU with 8 GB VRAM. At all. Now we're doing it on an Intel i7-H gaming laptop with a NVIDIA RTX 4060, thanks to llama.cpp + QAT quants + MTP speculative drafting. Google DeepMind said the Gemma 4 31B targets "consumer GPUs and workstations." They were not exaggerating. The hardware bar to run serious frontier class models locally keeps dropping. the tools are here. the models are here. you just have to be willing to abuse your laptop a little. what workflows would you actually run on a local 3 tps 31B dense model? genuinely curious. drop it below.

@houmanasefi I think your AI gone rogue... that comment doen't even address my or your main point 🤷🏼‍♂️

@DerekColley_ every model trained predominantly on english-language western internet encodes a worldview at the weights level. you can host it on sovereign servers in jakarta and it still thinks in san francisco.

Do you think the biggest inequality of the next 20 years will be access to AI? You are so simple minded!! Everyone will have a chatbot. That is the illusion. The inequality will be access to COMPUTE. One company asks AI a question. Another company runs 10 million agents 24/7 optimizing everything. Same technology. Different universe. Compute is the new capital.

@TheAhmadOsman Ok, now I'm thinking how to connect my AI to my food... and what would that be like. FFS, just can't get going today :/

@DerekColley_ I probably should have made that clearer hahaha

Hooked up Cloudflare and Porkbun APIs into an agent and that's how I purchase domains, fixup DNS, and deploy everything through Cloudflare now This busywork used to take so much time to configure and now it is done in the background while I am doing more important stuff

@DerekColley_ @VictoriaHa50399 @Keir_Starmer "Receive and impart information and ideas, through any media, regardless of frontiers" Literally there.

@buffer Update: it seems I can hide the keyboard by tapping this icon... #newbie alert!!!

@buffer 9.08 on GrapheneOS via Play Store

@buffer On android, your app is unusable ! During compose, the next / done button is hidden When viewing a screenshot the whole app is a brick

@Jeyffre What's disappointing is local compute is not about to get cheaper...

1 - So GLM 5.2 is 700b parameters (ish) 2 - 4x DGX Sparks can supposedly handle up to 700b parameters (give or take) 3 - GLM 5.2 is supposedly in striking distance of the performance of GPT 5.5 and Opus 4.8. In my brief tests, it's really not shabby at all. 4 - So for $20k, you can get near the frontier on your table. 5 - Extrapolate the trend, and you could have mythos/5.5 pro - class models in your dining room for the cost of a cheap car less than five years from now. Even without extrapolation, we're already the near frontier running locally. 6 - Paying real api costs, I could easily blow through $3,000 per month coding and running agents. The machine pays for itself in 6-7 months conservatively. 7 - In 3-5 years, most power users of AI will self-host. 8 - Am I missing something?

Hmmm, can't login on @barclays app on @GrapheneOS in a dedicated profile. The Barclays app requires access to Google Play Services and must be running in the Owner/Main user profile. Barclays heavily restricts its app from operating in secondary or work profiles :(

@my_knn_totoro @TheGeorgePu @grok @grok tell them about sparkl-network! github.com/orgs/sparkl-ne…

@DerekColley_ @TheGeorgePu @grok is there something like this but for DGx Spark owners? also what do these graphic card owners get in return?

@israfill gold!