Aleph Alpha

Introducing Alpha-MoE: A fused megakernel for faster tensor parallel inference. With up to 200% speedups for MoE models in TP deployments. Optimized for Hopper. Built for sovereignty and scale. aleph-alpha.com/alpha-moe-a-me…

Inference isn’t just about speed. Our latest blog breaks down how hardware choices impact latency, throughput & cost when serving massive models like DeepSeek v3. Read the blog & full report: aleph-alpha.com/deepseek-infer…

Try them out, experiment with the architecture, and see how tokenizer-free models handle your language or domain. Find our blog post here: aleph-alpha.com/introducing-tf… Or read our model cards: huggingface.co/collections/Al…

We’re shipping both models with: Hugging Face inference for easy testing vLLM fork optimized for HAT – note this is still under active development Weights & inference code are available today under Open Aleph license for research and educational use.

Introducing two new tokenizer-free LLM checkpoints from our research lab: TFree-HAT 7B Built on our Hierarchical Autoregressive Transformer (HAT) architecture, these models achieve top-tier German and English performance while processing text on a UTF-8 byte level.

We signed it, sending a signal to the world. The EU’s AI Code of Practice isn’t just another piece of regulation. It’s a declaration. Of what AI can be when built with transparency, responsibility and sovereignty. Just like ours. aleph-alpha.com/a-signature-an…

Read our paper for more information: arxiv.org/abs/2505.00022

For #ICLR2025 we are unveiling a new, high-quality pretraining dataset for German LLMs. Shared to strengthen the open research community. Shaped by our belief in excellence and transparency. huggingface.co/datasets/Aleph…

We gathered here to m#ourn the tokenizer, whose time has passed. We released three 8B models – and said goodbye to an old friend. Find our models on Huggingface: huggingface.co/collections/Al… And read more in our paper: arxiv.org/abs/2501.10322 #RIPTokenizer

@jphme @kerstingAIML Thanks for your interest in our technology! It’s open for research and education to promote transparency and progress in the field. While commercial use is restricted, we fully support innovation and are always open to collaboration and dialogue. Let’s connect!

@Aleph__Alpha @kerstingAIML The "Open Aleph" license states you may use this (scraped) web dataset for "for any Non-Commercial and Non-Administrative purpose." Pretty Lame (and misleading license name) tbh!

🐍 #PyCon has kicked off! Swing by our booth to chat about your career aspirations and explore how we can collaborate 🤝 Let us know you're coming by filling out this form: forms.office.com/Pages/DesignPa…

Excited for ICLR 2025 in Singapore? Join our BoF Social (24 Apr, 12:30 p.m., Opal 103-104) on tokenizer-free, end-to-end architectures. Ready for insightful discussions and networking? Sign up here forms.office.com/e/UaffUVtyHx #ICLR2025 #AIResearch #EnterpriseAI #Tokenizers

🚀 Exciting Announcement from Davos: Aleph Alpha Unveils Tokenizer-Free LLMs! 🚀 We’re thrilled to announce a pioneering innovation that was unveiled yesterday at the World Economic Forum in Davos: Aleph Alpha has introduced a groundbreaking tokenizer-free (T-Free) LLM architecture, setting a new benchmark for sovereign AI solutions. 🔑 What makes this breakthrough so important? Many governments and enterprises have struggled to develop and master a sovereign generative AI strategy due to a lack of data, resources and expertise, forcing them to rely on established models that do not perform adequately on relevant languages and specialized knowledge. This results in a loss of power where value capture, but also ideological alignment, has to be left to a small group of LLM producers. By removing the tokenizer – a bottleneck to adaptability in traditional models – our new tokenizer-free AI architecture enables: ✅ High-quality multilingual capabilities for multiple languages and alphabets. ✅ Customization for specialized industries. ✅ Lower computational demands for fine-tuning, reducing costs and environmental impact. For Finnish, for example, this innovative AI architecture can deliver a 70 percent reduction in total training cost and carbon footprint compared to alternative options. This innovation democratizes access to AI, empowering governments and businesses to create solutions that are fully aligned with their cultural and linguistic specifications, specialized knowledge and terminology, while increasing efficiency and fairness in AI applications. 🌍 💡 What drove us to this development? As a European company rooted in a diverse linguistic and cultural ecosystem, we’ve prioritized sovereignty and multilingualism since day one. Years of research have culminated in the T-Free architecture, a true milestone in AI technology. 🤝 Collaborating with @AMD & Schwarz Digits In collaboration with AMD and Schwarz Digits, Aleph Alpha's new and innovative LLM architecture is enhanced with high performance computing and a sovereign cloud solution, creating a full-stack sovereign offering that combines AI software and hardware. 🔗 Press release: lnkd.in/eGAsPk9R 👉 Technical insights on our blog: lnkd.in/evse6YD9 👉 Our latest peer-reviewed Research Paper: lnkd.in/ewmJTQTf Let’s shape the future of trustworthy, sovereign AI solutions together.

@GCResearchTeam Here are the links for this release: Huggingface: huggingface.co/Aleph-Alpha/um… Paper: arxiv.org/abs/2407.17465… Codebase: github.com/Aleph-Alpha/sc…

New in Neural Network Parametrization Technique: Introducing Unit-Scaled Maximal Update Parametrization (u-μP). In partnership with @GCResearchTeam, u-μP merges μP and Unit Scaling to boost training stability & hyperparameter transfer across model sizes. Read more about the theory & practical tips for LLM training in our blog. 🔗aleph-alpha.com/in-awe-at-the-… #writtenbyalephalpha

Read our full paper here: arxiv.org/abs/2406.19223 Dive into the source code of T-Free: github.com/Aleph-Alpha/tr… Try out our interim research model checkpoints: huggingface.co/Aleph-Alpha/tf… huggingface.co/Aleph-Alpha/tf… (3/3)

Our innovation, T-Free, offers a novel approach to tokenization, boosting tokenizer fertility across various languages, and reducing the size of the embedding layer by up to 75% compared to traditional tokenizers. Early experiments with T-Free show promising results and could unlock new possibilities in LLMs, including: - Up to 50% reduction in training and inference costs - Improved semantic encoding of language - Enhanced performance in multilingual models (2/3)

Today we introduce T-Free, a new paradigm in language processing. Tokenization is one of the core building blocks of large language models (LLMs), transforming natural language into numeric representations for further processing. (1/3) 🔗 lnkd.in/eTi7kjuc #writtenbyalephalpha