RWKV

#RWKV is One Dev's Journey to Dethrone Transformers The largest RNN ever (up to 14B). Parallelizable. Fast inference & training. Quantizable. Low vram usage. 3+ years of hard work wiki.rwkv.com Created by @BlinkDL_AI Computation sponsored by @StabilityAI @AiEleuther

Wrapping up: #RWKV was created by @BlinkDL_AI as a project at @AIEleuther - and is now being hosted by @LFAIDataFdn The RWKV wiki can be found at: wiki.rwkv.com Our discord can be found at: discord.gg/bDSBUMeFpc Give the models a try, drop by our discord

Overall we are excited by the progress we look forward to - the step jump in RWKV7-based attention - and the upcoming converted 70B models Once properly tuned, and trained, it will serve as a full drop-in replacement for the vast majority of AI workloads today 🪿

Announcing a flock of RWKV models all arriving with apache2 licensing: available today First up, the strongest and largest linear model to date: QRWKV6-32B-Instruct-Preview Surpassing previous state space & RWKV models matching transformer performance with much lower inference

@danofer We will accept donations to our foundation in cash or GPU time ❤️ (the project is properly attributed in the legal txt file in the folder)

@RWKV_AI Kinda crazy if they're not also supporting/citing you

RWKV.cpp - is now being deployed to half a billion systems worldwide Making it one of the world's most widely deployed, truly open-source (apache2) AI solutions out there As it now ships with every Windows 11 system

🚀 RWKV.cpp AI system - is being deployed to 0.5 billion installs globally Making it one of the world's most deployed, truly open-source (apache2) AI solutions out there As it ships with every Windows 11 system today!! Our group's code install count, went from ~100k -> 0.5B

RWKV.cpp project can be found here: github.com/RWKV/rwkv.cpp Our RWKV wiki can be found at: wiki.rwkv.com Our discord can be found at: discord.gg/bDSBUMeFpc Give the models a try, drop by our discord, and have fun!

Energy cost - Is critical for a device that aims for long battery life! Variety of language support - is critical for shipping a product to the whole world (beyond English or Chinese) We will eagerly keep watch on this development blog.rwkv.com/p/rwkvcpp-ship…

Wrapping up: #RWKV was created by @BlinkDL_AI as a project at @AIEleuther - and is now being hosted by @LFAIDataFdn The RWKV wiki can be found at: wiki.rwkv.com Our discord can be found at: discord.gg/bDSBUMeFpc Give the model a try, drop by our discord

As previously covered, this model is an upcycled model from our v5 - with an additional 1.4T tokens trained, for the conversion process. The RWKV v6 Finch paper covering the architecture can be found here: arxiv.org/abs/2404.05892

The RWKV v6 Finch lines of models are here Scaling from 1.6B all the way to 14B Pushing the boundary for an Attention-free transformer, and Multi-lingual models. Cleanly licensedm Apache 2, under @linuxfoundation Find out more from the writeup here: blog.rwkv.com/p/rwkv-v6-finc…

Is AI's insatiable appetite for compute holding back innovation? I recently talked with Eugene Cheah about @RWKV_AI. Eugene and the open-source RWKV community are working to democratize AI by addressing some of the key limitations of transformers.