Patrick Devine

For your pandemic Friday viewing enjoyment. 'kubectl run -it --rm --image=ghcr.io/pdevine/thisis… tif' or for the docker inclined 'docker run -it --rm ghcr.io/pdevine/thisis…'

So glad to finally get this out! We also added a new LRU trie cache w/ Qwen 3.5 which makes using coding tools like Claude and Pi super usable running locally.

@KrakowiakK @awnihannun I believe MLX is faster than vLLM w/ batch size 1.

@awnihannun I am waiting for the moment when MLX will be ahead of vLLM.. might never happen ;)

You can now run LMs with Ollama + MLX! I've been waiting for this moment since MLX was first open sourced, so glad that it finally arrived.

@QuantCapitalX @ollama Not quite ready yet 😅

Why are nvfp4 Models not supported on the NVIDIA DGX-Spark? Should be running fine! "admin@dgx-spark:~$ ollama run qwen3.5:35b-a3b-nvfp4 pulling manifest Error: pull model manifest: 412: this model requires macOS"

Ollama is now updated to run the fastest on Apple silicon, powered by MLX, Apple's machine learning framework. This change unlocks much faster performance to accelerate demanding work on macOS: - Personal assistants like OpenClaw - Coding agents like Claude Code, OpenCode, or Codex

@D_Twitt3r @ollama You _may_ be able to get them to import if you create a Modelfile in the safetensors directory and then use `ollama create --experimental -f <path/to/Modelfile>`. You can use the `--quantization` parameter to quantize to nvfp4/int4/etc.

@ollama Will this updated Ollama support other mlx and/or nvfp4 models downloaded from Hugging Face? Or do we need to wait for you to do more adjustments to them and post in your own catalog?

@ivanfioravanti @ollama Definitely appreciate any feedback! This one took a while, but I think the wait will have been worth it. 😅

@ollama YES!!!!!!!! Thanks team! I'll benchmark this like crazy and send feedback!

@micheltamanda @hxiao @0xRaghuboi Take a look through the docs in docs/development.md (specifically the MLX Engine (optional) section). You'll need to install Xcode and also run `xcodebuild -downloadComponent MetalToolchain` and then the correct cmake commands (they're covered in the doc).

@pdev110 @hxiao @0xRaghuboi @pdev110 , thoughts?

uh..Qwen3.5-35B-A3B on llama.cpp re-prefill on every request, ~4x slower than it should be. anyone solved this? Thought people have happily deployed & used it locally? But if this is not solved yet, the perf is quite limited. Root cause: GDN layers are recurrent → pos_min tracks full sequence → but llama.cpp validates cache using an SWA threshold that defaults to 1 for non-SWA models → pos_min > 1 always true → cache always discarded → full re-refill every time?

@ivanfioravanti @dfi @0xSero Do you mean works better than the stock affine quants? I think the problem is the default groupsize is 64 which isn't super great.

My for mlx-openbench is working like a charm and crunching evals like crazy. My M3 Ultra are on fire! 🔥 Here testing Qwen3.5-35B-A3B-REAP-pile10k-15p-MLX-q8 by @dfi created with reap-mlx by @0xSero Don't ask me how this is possible, but REAP model is working better on evals 👀

@micheltamanda @hxiao @0xRaghuboi Just merged the two PRs. What are you running in to?

@pdev110 @hxiao @0xRaghuboi Working through this now, I having issues getting MLX to work with int4. Debugging it now.

@micheltamanda @hxiao @0xRaghuboi I get about 135 toks/sec eval w/ that on the new M5 Max. About 95 toks/sec eval on an M3 Max.

@micheltamanda @hxiao @0xRaghuboi github.com/ollama/ollama/… you will also need to `ollama pull pdevine/qwen3.5:35b-a3b-int4`. Development instructions for enabling MLX are in `docs/development.md`.

@micheltamanda @hxiao @0xRaghuboi Still in PR form, but will hopefully be merged soon 😅

@pdev110 @hxiao @0xRaghuboi Ok this is great, I will install the latest when I get home

@micheltamanda @hxiao @0xRaghuboi I just posted some updates to make it run a lot faster w/ Ollama w/ the MLX backend. Also, there are some caching changes coming too which should make it pretty zippy.

@hxiao @0xRaghuboi I am using Ollama to serve this model. I will try vLLM/llama.cpp next.

@ivanfioravanti Do you know if oMLX sets the presence_penalty correctly? Qwen3.5 is super sensitive to this and tends to really overthink if the hyperparameters are set incorrectly.

Yes, Qwen3.5-9B thinks a bit too much. 22 seconds to reply to Hi. Note: here I'm using oMLX (it seems really good!)

@ivanfioravanti @awnihannun I've definitely felt the "cognitive debt" issue already with LLMs writing something that "works", but it turns out it isn't the thing I actually asked it to write. The flip side is it's now really easy to just _ask_ the LLM to explain itself.

@awnihannun "As humans write less code our mental model of the underlying systems and algorithms will decay or never develop in the first place. This is cognitive debt." Very few people will be able to intervene in case of need, this could become a real issue in the mid term.

@awnihannun I think the main difference between models and hw is the cost of duplicating the weights is effectively zero. If you've got the hw, you can always run another copy model.

In some ways AI models are more like hardware than software. - Predictable scaling - Immense capex to produce - High per unit margin (each token is profitable for inference) - Finite shelf life. Shorter half life than hardware, to be sure - Updated on reasonably predetermined schedules. And at this point I think it’s fair to say those schedules are known at least a year maybe multiple years in advance - Distinct tape-out (training) and after that you can only improve / fix them in post training (firmware?) I wonder if AI models will ever be sold in the same way devices are sold.

@vickcodes Pro tip: We just added Ctrl-G (same as Claude and Codex) in `ollama run` so you can use your favourite editor to write long prompts.

👩‍🚀 Up and Running. 🚀 #llama #minimax

New @ollama update is god's work btw Give it a try Just run `ollama` and thats it

@ivanfioravanti In my testing on glm 4.7 flash, 4bit affine quants with a high group setting often makes it go of the rails (and I still see this even with mixed precision between the various tensors). 8 bit is definitely a lot more resilient.

Using 4bit coding models locally give you more generation speed, but less precision. This means model and harness will keep iterating on wrongly created file to fix them. TLDR: better using 6bit or above for local coding.

@MrBerneker @ollama it's not in the UI yet, but does work in the CLI. Here's z-image-turbo running on the MLX backend in Ollama in an iterm2 terminal running on macOS:

Hey @ollama, I have a feature request for you! I love that you added z-image-turbo and flux2-klein for use with Ollama, but it would be even better if you made your chat interface able to send it prompts and display generated images too! What do you think? Doable?

@wassollichhier @ollama I told it to "be so colorful that my eyes freaking bleed", so I can't blame it for the neon. :-D

@pdev110 @ollama Interessting, when I one shot a Pong game with GLM 4.7 Flash, it made a „Neon“ Pong as well. The model really like neon style for old school games ^^ Btw. I havent asked Flash to make it Neon

1-shot prompt Asteroids game w/ GLM-4.7-Flash on the experimental MLX backend for @ollama. This was w/ 8-bit affine quantization and a MBP.