EIFY

For some reason I decided to swap out standard dot-product attention for a scaled-rbf kernel. Pretty much expected it to fail to converge or be impossibly slow but the scaled-rbf-attention is getting unexpectedly good results?? 👇

The size of the representation is not very important. You can always increase the dimension of a single-vector embedding. It barely helps. The important thing is interaction: different pieces in the query can attend to (or align with) different pieces in the document. This modularity makes learning good representations much more generalizable. It's very easy to learn a representation of a primitive concept (or a keyword). It's extraordinarily hard to learn a single vector such that a dot product captures deep relationships across distributions of queries and documents. Because of this, the size of late-interaction (multi-vector) representations of a passage is often *much smaller* than off-the-shelf single-vector representations you get from using float16 to encode a 4096-dimensional vector. Each vector in late interaction encodes extremely compact information, often representable with as little as 20 bytes.

捡来一张图

Nowadays biases are omitted in transformers for simplicity/same quality but it might matter more to keep affine layers for more expressivity, since bias doesn't affect Lipschitz

THAT'S CHINESE🇨🇳🤣 Imagine being so clueless you can't even identify your own history. That's 100% Chinese attire. Stop embarrassing yourselves and learn what your ancestors actually looked like. STUPID CHINESE🤣🤣

A new cryptic lineage popped up in St Louis a few weeks ago. I’ve been sampling this sewershed (500k people) twice a week for years and the first time I see this cryptic lineage it is 5 years old and makes up 50% of the sample. 1/