SDAIA-KAUST AI

Meet OPEN SOURCE AND FREE SakanaAI/ The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. I have been running a lot of tests on this for quite a bit. Enjoy uncensored SCIENCE. github.com/SakanaAI/AI-Sc…

Calling all #STEM leaders! Applications for graduate studies at #KAUST are now open. This is your opportunity to advance your academic journey in a world-class research environment. Visit our website and apply now. #ApplyKAUST

I am excited to be giving a 4-hour tutorial on "Pretraining and Finetuning LLMs from the Ground Up" at the @SciPyConf conference in 5 days! This tutorial is aimed at coders interested in understanding the building blocks of large language models (LLMs), how LLMs work, and how to code them from the ground up in PyTorch. After grasping how everything fits together and how to pretrain an LLM, we will learn how to load pretrained weights and finetune LLMs using open-source libraries. I am currently putting the final touches on the code and will share it along with a reproducible environment soon (github.com/rasbt/LLM-work…). The (I hope not too ambitious) schedule is as follows: 1) Introduction to LLMs: An introduction to the workshop, covering LLMs, the topics being discussed, and setup instructions. 2) Understanding LLM Input Data: In this section, we will code the text input pipeline by implementing a text tokenizer and a custom PyTorch DataLoader for our LLM. 3) Coding an LLM Architecture: We will go over the individual building blocks of LLMs and assemble them in code. We won't cover all modules in meticulous detail but will focus on the bigger picture and how to assemble them into a GPT-like model. 4) Pretraining LLMs: We will cover the pretraining process of LLMs and implement the code to pretrain the model architecture we created. Since pretraining is expensive, we will only pretrain it on a small text sample available in the public domain so that the LLM is capable of generating some basic sentences. 5) Loading Pretrained Weights: Due to the lengthy and expensive nature of pretraining, we will load pretrained weights into our self-implemented architecture. We will introduce the LitGPT open-source library, which provides more sophisticated (but still readable) code for training and finetuning LLMs. We will learn how to load weights of pretrained LLMs (Llama, Phi, Gemma, Mistral) in LitGPT. 6) Finetuning LLMs: This section will introduce LLM finetuning techniques. We will prepare a small dataset for instruction finetuning, which we will then use to finetune an LLM in LitGPT. I know I say this every year, but I am really excited to be returning to my favorite conference once more! It's going to be my fifth SciPy this year, and I am thrilled to see it at a new location (Tacoma/Seattle) this time!

Thrilled to share that our research group at KAUST is making waves at #CVPR2024! We've had EIGHT papers accepted this year. Huge congrats to all the authors and our incredible collaborators! 🦾📚 #AI #ML #ComputerVision @CVPR Let’s connect in Seattle next week! 🇺🇸 @KaustResearch

Announcing the creation of *KAUST Center of Excellence in Generative AI*; official launch on July 1, 2024. Joint with @BernardSGhanem @SchmidhuberAI @bremen79 @TheSandyCoder and a few more KAUST colleagues. KAUST funding ($11m over 5 years) + industrial funding. We are looking for further industry partners! @KaustResearch @cemseKAUST @KAUST_News @AI_KAUST @kaustMLhub

Today we’re releasing OpenEQA — the Open-Vocabulary Embodied Question Answering Benchmark. It measures an AI agent’s understanding of physical environments by probing it with open vocabulary questions like “Where did I leave my badge?” More details ➡️ go.fb.me/7vq6hm All of today’s state-of-art vision+language models (VLMs) fall well short of human performance. In fact, for questions that require spatial understanding, today’s VLMs are nearly “blind” – access to visual content provides only minor improvements over language-only models. We hope that OpenEQA motivates additional research into helping AI understand and communicate about the world it sees.

Okay I did a first quick pass of naive CUDA kernels for the forward pass of GPT-2 and pushed everything to one file in llm.c, Still only ~1000 lines of code: github.com/karpathy/llm.c… Current per iteration timings on my Lambda box <3 A100 40GB PCIe, B=4, T=1024: - llm.c: 111ms - PyTorch: 180ms - +torch.compile: 86ms - +fp32 tensor cores: 26ms So there is a gap to close! Come hack, make fast :)

Our #GPTSwarm models Large Language Model Agents and swarms thereof as computational graphs reflecting the hierarchical nature of intelligence. Graph optimization automatically improves nodes and edges. arxiv.org/abs/2402.16823 github.com/metauto-ai/GPT… gptswarm.org

Absolutely agreed with @MingchenZhuge. It is about time to bring optimization and learning to the top level of LLM stack: agents.

In a vibrant celebration of our community's diversity of 119 nationalities, #KAUST hosted the annual Parade of Nations 2024, reflecting unity and cultural richness. 🌍✨ #KAUSTParadeOfNations #SaudiArabia #DestinationKAUST

#سدايا تفتح باب التسجيل في معسكرات تعلم الآلة (ML) بمرحلتها الرابعة. للتسجيل: sdaia.tuwaiq.edu.sa/MLBootcamp#

@suchenzang Suppose instead of selling the data you sold API access to the embedded data or API access to a RAG application that has been fine-tuned in the data and created a smart contract that implemented some revenue sharing mechanism between the University and the authors?

so i guess this is a thing now universities running ads to resell students' data for training llms 💰💰💰

When it comes to XGBoost: 1) Only number 3 after CatBoost and LightGBM 2) Unsuitable for uncertainty quantification. #xgboost

@predict_addict Would be hilarious if it turns out that Chebyshev polynomials are all you need!

Are activation functions in deep learning overrated? Perhaps Hadamard product is all one needs? A new paper 'MULTILINEAR OPERATOR NETWORKS' suggests so. 'Despite the remarkable capabilities of deep neural networks in image recognition, the dependence on activation functions remains a largely unexplored area and has yet to be eliminated. On the other hand, Polynomial Networks is a class of models that does not require activation functions, but have yet to perform on par with mod- ern architectures. In this work, we aim close this gap and propose MONet, which relies solely on multilinear operators. The core layer of MONet, called Mu-Layer, captures multiplicative interactions of the elements of the input token. MONet captures high-degree interactions of the input elements and we demonstrate the efficacy of our approach on a series of image recognition and scientific computing benchmarks. The proposed model outperforms prior polynomial networks and performs on par with modern architectures. We believe that MONet can inspire further research on models that use entirely multilinear operations.' #computervision #deeplearning

@peter_richtarik According to LLaVa 1.6 using @ollama your random photo of KAUST is... ...a nighttime shot of a building on a campus, bathed in dramatic lighting. Shadows from palm trees add to the atmospheric vibe. A moonlit sky completes the scene.

Random photo of KAUST.

Check out their code, trained models, and generated data at: github.com/hammoudhasan/S… URL: arxiv.org/abs/2402.01832… #SynthCLIP #FullySyntheticTraining #CLIP #NaturalLanguageProcessing #MachineLearning 5/5

With training at scale, SynthCLIP achieves performance comparable to CLIP models trained on real datasets! 🎉 They also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images! 4/5

🚀 Exciting news! 🔍 @hammh0a and team just dropped a game-changing research paper - SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training? 🤯 1/5