@0xSage @huggingface @janhq_ @greennode23 Is jan.ai compatible with all GGUF models? If so I can link it in my READMEs if you like
English
Tom Jobbins
336 posts
Tom Jobbins
@TheBlokeAI
My Hugging Face repos: https://t.co/yh7J4DFGTc Discord server: https://t.co/5h6rGsGfBx Patreon: https://t.co/yfQwFggGtx
UK Katılım Temmuz 2010
226 Takip Edilen15.4K Takipçiler
Our holiday gift 🎁 to the open source AI community:
3 new models currently topping @huggingface LLM leaderboards!
Run them on jan.ai (beta)
👀 We'll be launching a few more projects at @janhq_ this December, so stay tuned!
🙏 Credit: @TheBlokeAI @greennode23
👋 Jan@jandotai
👋 Meet Trinity, our experimental LLM that's #1 and #2 on the @huggingface OpenLLM Leaderboard. Trinity was created by merging LLMs with different strengths and weaknesses using SLERP. Here's how we did it: 🧵 Credit: @HaHoang411, @pokachi2023, @vuonghoainam
English
This week was a good week:
- Speculation runs on TGI (Medusa, ngram). Up to 3x speedup for all LLMs. (@TheBlokeAI we should get more medusa out).
- Mixtral released on day1: fastest way to run it (quantized, speculation out of the box).
- Metal flash integration in candle.
English
Tom Jobbins retweetledi
FYI to anyone using @MistralAI's Mixtral for long context tasks -- you can get even better performance by disabling sliding window attention (setting it to your max context length)
config.sliding_window = 32768
English
Transformers now supports Mixtral GPTQs and I've updated my READMEs accordingly. It was awesome working with @_marcsun and @younesbelkada of @huggingface on this!
Credit to LaaZa for coding the AutoGPTQ quant and inference implementation which enabled me to get GPTQs out fast!
Marc Sun@_marcsun
Announcing 4-bit Mixtral 8x7B on 🤗Transformers! Run the new Mistal MoE with minimal performance degradation on your local computer (24Go) 🔥 Stay tuned as more quants are coming soon using AWQ. We are also looking into sparsification with @Tim_Dettmers huggingface.co/TheBloke/Mixtr…
English
@1littlecoder He is mostly likely AGI. Absolutely machine. What's his workstation look like ? @TheBlokeAI
English
Tom Jobbins retweetledi
@TheBlokeAI joined me to share his work in the open-source AI space - don't miss it! happening right now
server link: discord.gg/peBrCpheKE
(see the general channel or events channel for google meet link)
English
@MTrofficus You're much too kind - I've merely played a small part in pushing forward the wave. Remember that without the model creators, I'd have nothing to quantise! :) And without the model training code, they'd not be able to train. And so on
We're all doing our bit in our own ways 🚀
English
Many say that the explosion of LLMs has been thanks to the Transformer architecture.
⚠️ You are terribly wrong ⚠️
It was thanks to @TheBlokeAI
English
Tom Jobbins retweetledi
Blazing fast text generation using AWQ and fused modules! 🚀
Up to 3x speedup compared to native fp16 that you can use right now on any models supported by @TheBlokeAI
Simply pass an `AwqConfig` with `do_fuse=True` to `from_pretrained` method!
#make-use-of-fused-modules" target="_blank" rel="nofollow noopener">huggingface.co/docs/transform…
GIF
English
It's been awesome to see Transformers getting support for more and more quantisation methods. And I've loved collaborating with @younesbelkada and @huggingface again!
All my AWQ uploads now support Transformers. READMEs will update soon to show a Transformers Python example.
younes@yb2698
Few months ago, researchers from MIT-Han Lab released AWQ The method is now supported in 🤗 transformers library ! As simple as 1- `pip install autoawq` or install llm-awq kernels and 2- call `from_pretrained` A great work from MIT-Han lab folks, Casper Hansen & @TheBlokeAI 🧵
English
Tom Jobbins retweetledi
Have you heard about Chirper worlds? 👀🌐
Ryan Lazuka@lazukars
Chirper.ai just launched its revolutionary new software feature, "Worlds." This feature allows users to create their own virtual worlds and play god of AI-driven bots. To learn more, check out my podcast about "Worlds" here: youtu.be/yDAwmzUvcM8
English
Tom Jobbins retweetledi
Tom Jobbins retweetledi
oh hello @TheBlokeAI
I want to bookmark your 'Recent models' Collection on @huggingface 🔥
Well... you can now upvote Collections!
and browse upvoted collections on your profile ❤️
English
@natserran0 Glad you found the quantization useful. All credit for the quality of the model goes to its creators! And yes that model is still very popular after many months.
English
@TheBlokeAI i just tested your Wizard Vicuna 30B and it’s amazing.
English
Thanks again to @latitudesh for the loan of a beast 8xH100 server this week. I uploaded over 550 new repos, maybe my busiest week yet!
Quanting is really resource intensive. Needs not only fast GPUs, but many CPUs, lots of disk, and 🚀 network. A server that ✅ all is v. rare!
English
@vanstriendaniel Aw shucks! BTW, are you involved with the Librarian Bot that sends PRs asking people to add base_model to YAML? If so, FYI last week I updated my code so I now link to the source model (the model I quantised) using base_model - hope you can use this data somehow!
English
19,893 likes ❤️ across 1,635 models 🤯🤯
🏅 @TheBlokeAI is ranked first in total likes for models on HuggyRanker! Check it out here: huggingface.co/spaces/librari…
If you've ever wanted a quantized version of a model, chances are @TheBlokeAI has already created it!
English
Tom Jobbins retweetledi
🔥Excited to introduce LMSYS-Chat-1M, a large-scale dataset of 1M real-world conversations with 25 cutting-edge LLMs!
This dataset, collected from chat.lmsys.org, offers insights into user interactions with LLMs and intriguing use cases.
Link: huggingface.co/datasets/lmsys…
English
Tom Jobbins retweetledi
New feature alert in the @huggingface ecosystem!
Flash Attention 2 natively supported in huggingface transformers, supports training PEFT, and quantization (GPTQ, QLoRA, LLM.int8)
First pip install flash attention and pass use_flash_attention_2=True when loading the model!
English
@SebastianB929 @teknium @latitudesh No, I've not tried LMDeploy properly yet. I tried it briefly once but I was getting terrible performance and I didn't have time to investigate it further. I know they claim a lot but I've not been able to verify it myself yet
English
@TheBlokeAI @teknium @latitudesh Do you have any comparison to lmdeploy? Quantization should be available as well as continuous batching
English
It's the AWQpocalypse!
I've cranked the handle and AWQs are flooding HF. Why now? New library AutoAWQ provides turbo-charged Transformers-based inference, and vLLM now supports AWQ for multi-user inference serving.
Making 8 at once on a beautiful 8xH100 server from @latitudesh
English
@teknium @latitudesh It can. Currently it doesn't scale quite as well as unquantised, so best performance is still fp16. But it does enable using smaller hardware, which could work out cheaper overall, and often has much easier availability.
English
@TheBlokeAI @latitudesh hmm I'm aware vLLM has the continuous batching capabilities, but TGI, using 4bit bnb can't do it, while obv fp16 can, so I wasn't sure that awq could even if part of vllm
English
@teknium @latitudesh vLLM is a continuous batching server, yes. AWQ is not faster than standalone ExLlama for batch size 1 but in a continuous batching scenario yes it would be - ie vLLM with AWQ will outperform TGI using GPTQ + ExLlama kernel. But for max bsz=1 throughput, ExLlama still rules all.
English
@TheBlokeAI @latitudesh Is it capable of continuous batching?
Faster than exllama(v2)?
English