Colin Wei

Nothing beats the adrenaline rush of debugging numerics on a job as big as GPT4.5. It was an honor to work on this project with such talented teammates!

OpenAI is nothing without its people

💜

Curious about why contrastive learning produces representations useful for downstream tasks? Check out @jhaochenz, @ColinWei11, and @tengyuma's theoretical explanation in our latest blog post: ai.stanford.edu/blog/understan… Based on the NeurIPS 2021 oral paper: arxiv.org/abs/2106.04156

Pretraining is ≈SoTA for domain adaptation: just do contrastive learning on *all* unlabeled data + finetune on source labels. Features are NOT domain-invariant, but disentangle class & domain info to enable transfer. Theory & exps: arxiv.org/abs/2204.00570 arxiv.org/abs/2204.02683

Curious about why self-training with unlabeled data can magically improve a classifier’s performance? Check out the theoretical explanation in the following blog post: ai.stanford.edu/blog/understan… from @ColinWei11, @jhaochenz, and @tengyuma

Why can GPT3 magically learn tasks? It just reads a few examples, without any parameter updates or explicitly being trained to learn. We prove that this in-context learning can emerge from modeling long-range coherence in the pretraining data! arxiv.org/abs/2111.02080 (1/n)

RNNs and transformers with *infinite*-precision are universal approximators---but are they statistically learnable? We show that even finite-precision transformers can express any Turing machine, and they are learnable with polynomial samples. arxiv.org/abs/2107.13163

Pretrained language models are trained with losses quite different from downstream tasks; why can they help? Why does prompt tuning work so well? We analyze them under generative assumptions of the language (HMMs or more realistic variants) arxiv.org/abs/2106.09226

Our ICML paper (arxiv.org/abs/2102.11203) provided theoretical evidence on the effectiveness of consistency-based methods like AdaMatch, FixMatch, etc. with empirical results for the adaptation of FixMatch on domain adaptation tasks : p