Diego Alejandro Idfc
42.7K posts

Diego Alejandro Idfc
@BR4D112
I'm just looking to share a lot of software content || deep learning || system design || databases
Katılım Eylül 2020
1.7K Takip Edilen167 Takipçiler
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi

@FCB_nenn Los del barsa y el Madrid almenos nos podemos reír del cholo
Español
Diego Alejandro Idfc retweetledi

What is a token? (basics for understanding AI)
1. Models can't process raw text as is, so they start with tokens
2. Tokens are small units into which text is broken before being fed into a model
3. A token can be a whole word, part of a word, punctuation, or even a space
4. Common words are often 1 token, while longer or rarer words can be split into smaller pieces like encod + ing
5. Roughly, one token ≈ 4 characters, or about ¾ of a word. But it depends on the language and the tokenizer
6. Firstly, text is tokenized into tokens, which are mapped to IDs, then turned into vectors, and only then the model begins to process them
7. The context window of a model is counted in tokens, so they determine how much a model can “remember” in one pass
8. And finally, tokens are the pricing unit of generative AI. When we use AI, we pay per token
This is just the tip of the iceberg. But it gets much more interesting under the hood. Here’s a breakdown: turingpost.com/p/token

English

@MadridistaMexa Focus en que florentino no les pegue una cachetada en cada fin de partido
Español

Focus en qué? En nuestro partidazo contra el Alavés?😍🙌🏻
Real Madrid C.F. 🇬🇧🇺🇸@realmadriden
👀 Focus 👀
Español
Diego Alejandro Idfc retweetledi

Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi

#Mathematics for #MachineLearning and #Artificialintelligence — Download this 412-page PDF: mml-book.github.io/book/mml-book.…
—————
#AI #DeepLearning #BigData #DataScience #Statistics #DataScientists #Calculus #LinearAlgebra

English
Diego Alejandro Idfc retweetledi

COBRIX TENÍA HAMBRE Y VANESA LE LLEVÓ CAFÉ CON CHOCOTORTA ☕🍰
ES EL MEJOR JUGADOR DE LA HISTORIA 🐐🪄
TODO 412 RENDIDO ANTE EL NINE 👏🛐
LACOBRAAA Out of Context@OOC_LACOBRAAA
Te queremos mucho Davito, toda la comunidad te manda fuerzas ❤️
Español
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi
Diego Alejandro Idfc retweetledi



















