murat 🍥

@caguilar51 @vikhyatk yes easy

@mayfer @vikhyatk I have an M3 with 48GB of RAM. Could I run it on it

what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty @vikhyatk here we have whisper+qwen+moondream triple model pipeline working offline flawlessly

@azlenelza meme

If you are into word games and interested in playtesting this and other word games I’m tinkering on, let me know!

Making new word game where adjacent cells have to be meaningfully related / associated with each other

@azlenelza yay

@nuwandavek @vikhyatk gemma could be better at certain tasks or prompts too i wouldn't be surprised

@mayfer @vikhyatk nice! i was playing around with a similarish idea (finding all untagged basketball/tennis courts in sf on google maps by browsing around) gemma4 + efficientsam3 was hilariously good for the size. will try qwen! github.com/SimonZeng7108/…

@mig4ng @vikhyatk m3 max tho i also use it on base m4 mac mini note that moondream requires 32gb

@mayfer @vikhyatk What apple machine is it?

@shubham_arora_0 @vikhyatk thank you sir

@mayfer @vikhyatk goat website!

@Laythe_li_suwi @vikhyatk converting user command to actions, in this case it's just "click <item>" but it can do more things like use keyboard shortcuts, applescript, or type things directly

@mayfer @vikhyatk i am curious, where is qwen involved all of this?

@chris_ineg @vikhyatk defo not haha

@mayfer @vikhyatk this is really cool and the future of computer/browser use but when i think of giving an agent control of my computer it needs to meet 2 criteria. speed and intelligence. your demo looks quick but how intelligent is it with more complex usecases?

@nuwandavek @vikhyatk i've tried basically every small llm in existence and settled on qwen3.5 4B at q4_k_m there may be different prompts that make gemma work too but with my prompts gemma was unusably worse. qwen has really great general ability

@mayfer @vikhyatk which qwen is it? can you go even tinier with gemma? what’s the smallest stack where this can work reasonably well?

@alexpsouthwell @vikhyatk idk you tell me sounds interesting tho

@mayfer @vikhyatk can you preprocess by capturing the screen, looking at the parts of the image that are diff from the last and then identifying the content an action surfaces, then you have a text model of screen that the agent can interact with immediately. trade off speed / processing

@mdlahfir @yacineMTB @vikhyatk im using screenshots theyre more universal and tbh faster often

@mayfer @yacineMTB @vikhyatk I wonder whether it is using screenshot loop to predict the co-ordinates or using accessibility trees? If you want your LLMs to control using accessibility trees fully headless, you can use this: github.com/lahfir/agent-d…

@tombielecki @vikhyatk changes from active app and even url yes

@mayfer @vikhyatk is the prompt prefix static, or does it change based on what applications are focused?

@tombielecki @vikhyatk whisper-large-v3-turbo or whatever its called. it's hands down best especially with text prompt prefix to set the context. if you need faster due to hardware restrictions parakeet or apple speech recognition are ok

@mayfer @vikhyatk which whisper model are you using? I went into creating a next-action prediction model using hammerspoon and markov chains to help local STT understand what I might be trying to say (context for short utterances). Wonder if it might be useful here.

@m0ches @vikhyatk keyboard shortcuts and applescript yes, we use base mac mini for our tv. for local moondream you need 32gb memory

@mayfer @vikhyatk Wait, is this actually usable without needing a beefy machine attached?

@lyc_aon yeah when i said "low latency" i thought what i meant would come across

@mayfer Depends on how detailed/accurate you need the image analysis to be and how fast you need the loop. For live computer use like tasks local will prob win out, but for automatic detailed aesthetics / design / accurate text extraction, cloud will be ahead / faster for a while imo.

yeah this is super significant it's absurdly bad capital allocation to run low latency AI image processing on cloud. local computer use will win really hard

@joodalooped it helps but doesn't solve it until you make text unreadable

@mayfer image upload point honestly really good, although part of me is like "would low res work?"

the harness is GoatRemote goatremote.com it has an absurdly optimized qwen3.5 pipeline. the LLM call takes 300ms which determines what action to take based on the user's request imo forget traditional agent harnesses, they can't achieve this kind of latency, they're not built for it

@mayfer @vikhyatk What harness does it use to achieve 1s latency? In my tests, Desktop Control CLI achieves ~400-500ms for local perception, but you also need to add LLM call layency. See demo: x.com/yaroshevych/st…

@vikhyatk amazing to hear. every time latency is reduced it opens up new use cases

@mayfer more speedup coming soon btw, i stuck with the constraint of sticking with the same precision we trained the model on. but mac really needs heavier quantization... now that we know, can address it during training so there's no accuracy loss from PTQ

@m13v_ @vikhyatk that's a good optimization

@mayfer @vikhyatk fair on prefill, chromium-scale trees eat context fast. native apps with stable refs are where AX still wins, target the same element across multiple actions without re-grounding.