Alfonso² Peterssen

@__tinygrad__ The operators serve hyper-specialized implementations of each model. How good is tinygrad at fusing high-level ops? Even with some advanced compiler magic, the hand-tuned kernels with nit-picked fusions are hard to beat. It's a pristine model blueprint vs. a tuned Franken-model.

Branching off of this is all the fun stuff. Megakernels, fast GGUF on the fly unpacker, function decorator for Python speed, KV cache swap to disk. This should stay ~500 lines, but outperform all the BS=1 LLM runners through the power of tinygrad. Kimi at 500 tok/s on MI350X?

We are looking to hire someone to improve our LLM runner, with USB GPU + high BS=1 tok/s it should be used a lot soon. The TODO list is in the Discord, but no bounties since that yields AI slop. The bottleneck today isn't writing code, it's filtering it. Show me you can do that.

My @GraalVM Native Image deep dive recording is already up: youtube.com/watch?v=1J6mbM… 🐰🚀 It includes the very public first demo of project Crema, Open World for Native Image, at 2:19:54 😅 Thank you, @Devoxx! All demos and notes are here: github.com/alina-yur/graa…

GPULlama3.java is out! Great effort by the @tornadovm team to bring GPU-enabled inference to the JVM

run an LLM with a supercharged engine powered by Java and GraalVM (ht @alina_yurenko ) youtube.com/shorts/7zSEa_B…

Looking forward to speaking tomorrow at @VoxxedCERN together with @TheMukel followed by delivering both a keynote and a regular talk at @VoxxedTicino on Friday! 🤩🎤#FunDaysAhead #AAP #Java

We just merged the current status of the upcoming JDWP support for @GraalVM Native Image! 🥳 This will soon provide developers with the same debugging experience they are used to in Java, but for native images! Stay tuned for more details. github.com/oracle/graal/p…

buff.ly/40KmT0t Graal compiler: +10% faster inference with the latest early access build. New features: batched prompt processing & AVX512 support.

As a result I can now use @DevoxxGenie with a pure Java Arm Inference engine running locally on my mac using Llama 3.2 🍏🔥

Modern @Java Project : a Spring Boot wrapper for Llama3.java from @TheMukel supporting OpenAI Chat Completion REST requests 🔥 github.com/stephanj/Llama… #OpenAI #SpringBoot

Earlier today was asked if Java AI integration improved yet, or that we'd still need to rely on Python or C bindings. Was happy to share github.com/mukel/llama3.j… by @TheMukel from the GraalVM team running native in Java without any dependencies and with superior performance!

@tjake For me your #Devoxx talk about github.com/tjake/Jlama and the one from @TheMukel about github.com/mukel/llama3.j… were the most relevant talks. Thanks for all the background information - I have learned a lot!

Just made the first-ever @DevoxxGenie LLM inference using ONLY @Java, powered by the awesome #Jlama project! ☕🔥 Huge thanks to @tjake for making it happen! 💪🏼#NoPython #JavaAI

@christzolov @vitalethomas @alina_yurenko I have a working prototype with function calling via LangChain4j. Vision is just a matter of implementing an additional component, the rest of the inference remains the same. I'll do my best to implement the missing encoder for vision soon-ish, starting with Llama, then Qwen.

@vitalethomas @TheMukel @alina_yurenko I like the idea, but last time I've checked this interface, it didn't support elaborate configuration, function calling nor the multimodality provided in Llama3.2. Has this changed or is it easy/possible to add support for it?

@diegoasua @bate5a55 @julien_c Why not? It runs Llama 3.2 1B at 40+ tokens/s on my laptop. It also supports GraalVM's Native Image with instant time-to-first-token.

@bate5a55 @julien_c you don't run a +1B model on CPU. Good luck with that, that's like tying a freight to a donkey. Will it move? Maybe. But also, don't do that

…but why? 😀

All details were presented at Devoxx Belgium! youtu.be/zgAMxC7lzkc PS: I'm the original author.

Developing and optimizing an LLM inference engine in pure Java: youtube.com/watch?v=zgAMxC… with @TheMukel #Devoxx #Java #LLM

Here's the #Devoxx Belgium recording, enjoy! youtube.com/watch?v=zgAMxC…

Contribute! 👇

jlama (by @tjake ) and llama3.java (by @TheMukel ) examples are working. See you at: "Pure Java Enterprise AI/LLM Integration (EAI 2.0)" @Devoxx (devoxx.be/talk/pure-java…) #java #llm #ai

.@TheMukel and @alina_yurenko talking about practical LLM inference in modern Java with Llama3.java and @GraalVM at @Devoxx! 🏎🎉 Try it out at github.com/mukel/llama3.j…