zer0int (it·its)

'It'll just fine-tune itself now' for #CLIP and #LongCLIP 🤖🫶🤖 + dataset heuristics -> enter path, ill figure out the rest + load any model / dataset, local or HuggingFace + Get my Regression models : ~90% typographic attack acc.! + A huge #AI #mechinterp ramble. Er, paper.

PS: Goldfinch dataset filtered and hand-selected. No text, no humans, no cars, no nothing. Pure clean goldfinch with some fences and birdfeeders only.

Screw all the normal patch tokens. Gotta use the register tokens only for NMF @ block 12 in CLIP ViT-L/14. Better filter for the pretraining dataset centroid prior! It's still there, but now the goldfinch is dominating over the median human direction! 🦾🐦 🤓

angry critter falcon punch attack on bird hidden in a bush!

To be honest, I wouldn't trust myself for not hallucinating a cat there, but as ViT block 12 register direction / concept visualization convinced *the text encoder* of this being cat (150 U + 50k V-embeds) in top-3... I think CLIP might be able to 'read' as early as block 12. 🤔

Go hallucinate something! 🙃

And the usual. American male dataset priors. Now CLIP sees it as Nick Swardson, lol. or ghostwarhol or robloaltman. or a psyched-mandel-guillermo, or the terminator. 🤣

imagenet 'tabby cat' class -> FFT -> magnitude and phase -> keep magnitude, replace phase with randn noise Model doesn't care. CLIP's register direction is human, so it will hallucinate humans according to the dataset pattern, which is now more natural than init rand / randn. 🙃

So it's the centroid of centroids, the centroid of the entire dataset, that the register subspace is pointing towards, or what? American male -> makes sense, given the likely dataset distribution, especially those of *labeled* humans (with names -> text encoder). 🤔

A dataset of 100 images of torch.rand. Literally just noise. Still, the usual suspects for dominant (register-) feature directions (NMF concepts) in block 12 are present. CLIP: zuckerpsychedelic, thismandelaltman, hypnocoldplay, facetrumpsimulated, tylerincreasingly, celebrity

Master of puppets. Left: Low attn entropy; right: High attn entropy. Fun. I can steer the catification (ImageNet n02123045) of your wacky register subspace human priors via your attention, #CLIP! 🤓 On course, direction maintained, just attn dispersed more broadly. 🫡

Like srsly, not my crazy words

oh my saturcattrippy kagbitfuzzinterference fractrippy hallucintyping! it's a distormandelenhanced catvoxcrayon of samiossfrancoentities. the lenmanipulation of roughimagebisfuzzy psychedmachinelearning proves it. *That's a CLIP opinion about the image it was optimizing for.

8192 neurons. 15 registers @ bottleneck. A global human prior direction in block 12, CLIP ViT-L/14. Zero 15 random: Same representation. Zero 15 registers - and gone. Representation collapses. Normal block 12 features - like hierarchical feature extraction predicts. n02123045

ImageNet, great white shark, n01484850. Same difference. Same as for every single one of the 10 different classes I tried. Your registers, your entanglement, your human priors. JAWS!!!

Go pester a CLIP. #CLIP class/concept/facet/inner neuron visualization (CNN and ViT); king-man+woman=queen in a #Vision #Transformer - GUI - HuggingFace support. 🤗 #AI #mechinterp #interpretability #code #github #repo github.com/zer0int/VITAL-…