SDAIA-KAUST AI

1.4K posts

SDAIA-KAUST AI banner
SDAIA-KAUST AI

SDAIA-KAUST AI

@SDAIA_KAUST_AI

SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence

Thuwal, Makkah, Saudi Arabia Katılım Mayıs 2022
572 Takip Edilen1.3K Takipçiler
SDAIA-KAUST AI retweetledi
Brian Roemmele
Brian Roemmele@BrianRoemmele·
Meet OPEN SOURCE AND FREE SakanaAI/ The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. I have been running a lot of tests on this for quite a bit. Enjoy uncensored SCIENCE. github.com/SakanaAI/AI-Sc…
English
15
111
525
38.2K
SDAIA-KAUST AI retweetledi
KAUST
KAUST@KAUST_News·
Calling all #STEM leaders! Applications for graduate studies at #KAUST are now open. This is your opportunity to advance your academic journey in a world-class research environment. Visit our website and apply now. #ApplyKAUST
English
4
43
287
698K
SDAIA-KAUST AI retweetledi
Sebastian Raschka
Sebastian Raschka@rasbt·
I am excited to be giving a 4-hour tutorial on "Pretraining and Finetuning LLMs from the Ground Up" at the @SciPyConf conference in 5 days! This tutorial is aimed at coders interested in understanding the building blocks of large language models (LLMs), how LLMs work, and how to code them from the ground up in PyTorch. After grasping how everything fits together and how to pretrain an LLM, we will learn how to load pretrained weights and finetune LLMs using open-source libraries. I am currently putting the final touches on the code and will share it along with a reproducible environment soon (github.com/rasbt/LLM-work…). The (I hope not too ambitious) schedule is as follows: 1) Introduction to LLMs: An introduction to the workshop, covering LLMs, the topics being discussed, and setup instructions. 2) Understanding LLM Input Data: In this section, we will code the text input pipeline by implementing a text tokenizer and a custom PyTorch DataLoader for our LLM. 3) Coding an LLM Architecture: We will go over the individual building blocks of LLMs and assemble them in code. We won't cover all modules in meticulous detail but will focus on the bigger picture and how to assemble them into a GPT-like model. 4) Pretraining LLMs: We will cover the pretraining process of LLMs and implement the code to pretrain the model architecture we created. Since pretraining is expensive, we will only pretrain it on a small text sample available in the public domain so that the LLM is capable of generating some basic sentences. 5) Loading Pretrained Weights: Due to the lengthy and expensive nature of pretraining, we will load pretrained weights into our self-implemented architecture. We will introduce the LitGPT open-source library, which provides more sophisticated (but still readable) code for training and finetuning LLMs. We will learn how to load weights of pretrained LLMs (Llama, Phi, Gemma, Mistral) in LitGPT. 6) Finetuning LLMs: This section will introduce LLM finetuning techniques. We will prepare a small dataset for instruction finetuning, which we will then use to finetune an LLM in LitGPT. I know I say this every year, but I am really excited to be returning to my favorite conference once more! It's going to be my fifth SciPy this year, and I am thrilled to see it at a new location (Tacoma/Seattle) this time!
Sebastian Raschka tweet media
English
32
171
1.2K
62.6K
SDAIA-KAUST AI retweetledi
AI at Meta
AI at Meta@AIatMeta·
Today we’re releasing OpenEQA — the Open-Vocabulary Embodied Question Answering Benchmark. It measures an AI agent’s understanding of physical environments by probing it with open vocabulary questions like “Where did I leave my badge?” More details ➡️ go.fb.me/7vq6hm All of today’s state-of-art vision+language models (VLMs) fall well short of human performance. In fact, for questions that require spatial understanding, today’s VLMs are nearly “blind” – access to visual content provides only minor improvements over language-only models. We hope that OpenEQA motivates additional research into helping AI understand and communicate about the world it sees.
English
40
248
1.2K
406.8K
SDAIA-KAUST AI retweetledi
Andrej Karpathy
Andrej Karpathy@karpathy·
Okay I did a first quick pass of naive CUDA kernels for the forward pass of GPT-2 and pushed everything to one file in llm.c, Still only ~1000 lines of code: github.com/karpathy/llm.c… Current per iteration timings on my Lambda box <3 A100 40GB PCIe, B=4, T=1024: - llm.c: 111ms - PyTorch: 180ms - +torch.compile: 86ms - +fp32 tensor cores: 26ms So there is a gap to close! Come hack, make fast :)
Andrej Karpathy tweet media
English
73
316
3.7K
415.9K
SDAIA-KAUST AI retweetledi
Dmitrii Khizbullin
Dmitrii Khizbullin@dmitrii_tech·
Absolutely agreed with @MingchenZhuge. It is about time to bring optimization and learning to the top level of LLM stack: agents.
Mingchen Zhuge@MingchenZhuge

@SchmidhuberAI Using graphs to build agents is very important this year! Rethinking LLM-based agents: 'Agent = LLM + Memory + Plan + Tool + ...' is 'technically correct' but limits the creativities in this field. Now, we can involve many AI techniques with 'GPTSwarm'-like thinking. 😃

English
0
1
5
678
SDAIA-KAUST AI retweetledi
SDAIA
SDAIA@SDAIA_SA·
#سدايا تفتح باب التسجيل في معسكرات تعلم الآلة (ML) بمرحلتها الرابعة. للتسجيل: sdaia.tuwaiq.edu.sa/MLBootcamp#
SDAIA tweet media
العربية
15
67
253
114.1K
SDAIA-KAUST AI
SDAIA-KAUST AI@SDAIA_KAUST_AI·
@suchenzang Suppose instead of selling the data you sold API access to the embedded data or API access to a RAG application that has been fine-tuned in the data and created a smart contract that implemented some revenue sharing mechanism between the University and the authors?
English
0
0
0
873
Susan Zhang
Susan Zhang@suchenzang·
so i guess this is a thing now universities running ads to resell students' data for training llms 💰💰💰
Susan Zhang tweet media
English
100
470
2.8K
849.5K
SDAIA-KAUST AI retweetledi
Valeriy M., PhD, MBA, CQF
Valeriy M., PhD, MBA, CQF@predict_addict·
When it comes to XGBoost: 1) Only number 3 after CatBoost and LightGBM 2) Unsuitable for uncertainty quantification. #xgboost
Valeriy M., PhD, MBA, CQF tweet media
English
0
1
9
1.8K
SDAIA-KAUST AI
SDAIA-KAUST AI@SDAIA_KAUST_AI·
@predict_addict Would be hilarious if it turns out that Chebyshev polynomials are all you need!
Thuwal, Kingdom of Saudi Arabia 🇸🇦 English
0
0
5
312
Valeriy M., PhD, MBA, CQF
Valeriy M., PhD, MBA, CQF@predict_addict·
Are activation functions in deep learning overrated? Perhaps Hadamard product is all one needs? A new paper 'MULTILINEAR OPERATOR NETWORKS' suggests so. 'Despite the remarkable capabilities of deep neural networks in image recognition, the dependence on activation functions remains a largely unexplored area and has yet to be eliminated. On the other hand, Polynomial Networks is a class of models that does not require activation functions, but have yet to perform on par with mod- ern architectures. In this work, we aim close this gap and propose MONet, which relies solely on multilinear operators. The core layer of MONet, called Mu-Layer, captures multiplicative interactions of the elements of the input token. MONet captures high-degree interactions of the input elements and we demonstrate the efficacy of our approach on a series of image recognition and scientific computing benchmarks. The proposed model outperforms prior polynomial networks and performs on par with modern architectures. We believe that MONet can inspire further research on models that use entirely multilinear operations.' #computervision #deeplearning
Valeriy M., PhD, MBA, CQF tweet media
English
2
11
73
7.9K
SDAIA-KAUST AI
SDAIA-KAUST AI@SDAIA_KAUST_AI·
@peter_richtarik According to LLaVa 1.6 using @ollama your random photo of KAUST is... ...a nighttime shot of a building on a campus, bathed in dramatic lighting. Shadows from palm trees add to the atmospheric vibe. A moonlit sky completes the scene.
English
1
0
3
219
Peter Richtarik
Peter Richtarik@peter_richtarik·
Random photo of KAUST.
Peter Richtarik tweet media
English
2
0
12
3K
SDAIA-KAUST AI
SDAIA-KAUST AI@SDAIA_KAUST_AI·
With training at scale, SynthCLIP achieves performance comparable to CLIP models trained on real datasets! 🎉 They also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images! 4/5
English
2
0
2
272
SDAIA-KAUST AI
SDAIA-KAUST AI@SDAIA_KAUST_AI·
🚀 Exciting news! 🔍 @hammh0a and team just dropped a game-changing research paper - SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training? 🤯 1/5
Hasan Hammoud@hammh0a

🚀✨ SynthCLIP is out! 🎉 In the past few months, I've been working on an end-to-end synthetic pipeline for CLIP training. Today, the paper is out on Arxiv, and our 30M samples synthetic-image pairs dataset is available on @huggingface ! Thread. 🧵 Paper: arxiv.org/abs/2402.01832

English
1
1
2
726