Jon
325 posts

Jon
@jonathan_conn
Pushing for local AI sovereignty among devs
Katılım Haziran 2022
271 Takip Edilen128 Takipçiler
Sabitlenmiş Tweet

@DuperVibes @TechMDAI I got mine used from FB marketplace, maybe something like that in your area
English

@TechMDAI @jonathan_conn Where can one get 4tb gb10 for $4k nowadays? No micro center near me I'm afraid.
English

@TechMDAI @with_gene2626 @MiaAI_lab @Tech2Wild @u1tra_instinct 70 tps would fly, would love to see it
I am just not sure how we get there without gutting more experts from the modal
Would love to hear @Tech2Wild and @MiaAI_lab take on it
Is it possible? or are we dreaming?
English

@with_gene2626 @MiaAI_lab Good to compare and if we can use the 3 spark set up from Miaai for glm 5.2 and use the extra spark for KV cache and context. Maybe we could see 70 Tk/s per second @MiaAI_lab @jonathan_conn @Tech2Wild @with_gene2626 @u1tra_instinct tra_instinct
English


@wicksplay Hell of a setup, congrats, can't wait to see what you do with it
Would love to see your GLM 5.2 tps
The spark community gets about 20-35tps
English

Got this monster of a machine up and running
CPU: AMD Ryzen Threadripper PRO 9975WX (4.0GHz)
RAM: 512 GB DDR5 5600 RDIMM
GPU: 4x Blackwell 6000 Pro Max Q (Will watercool at some point)
Current Model: huggingface.co/0xSero/GLM-5.2…
Let me know what models I should try 👇

English

@punit_arani @trashh_dev Thats sick! I thought about it
You should do a write up on your experience, would love to see more of the communities thoughts on it
English

@jonathan_conn @trashh_dev I ended up just trying it on serverless GPUs
I got the dopamine of self-hosting while seeing an extra $3K in my bank :)
English

@punit_arani @trashh_dev In all seriousness I do think one spark is worth it
Just for the learnings of understanding how to host local models
So many things fall out of that experience
If you can find a used one, I would pick one up. I was able to find them for about $3.5k usd
English

@jonathan_conn @trashh_dev I was on the verge of buying a DGX Spark until I did the math after sleeping on it fortunately
English

@punit_arani @trashh_dev Depends how late in the evening I do the math
English

Yes it does. What solidified the DGX spark as option is the increase in the 128 gb MacBook M5 max to 7k. Plus the raising cost of every gpu. The sparks at 4k is a steal for 128gb and 4TB of storage.
I was waiting on m5 ultra max studio but it got delayed. Imagine the cost of the 512 gb Mac Studio now?
English

Wanted to take a sec to shout-out the Local AI community
You guys have been great, excited to see the community grow
I arrogantly think all of us local first folks are ahead of the game
Jon@jonathan_conn
The nuke button Perfect for resetting after hitting OOM when trying new recipes
English

Can't say this enough
If you have a mac and work outside, get Vivid
Been using it for 2+ years now probably
getvivid.app
English










