Ren Wang

KiVA (Kid-inspired Visual Analogies) Challenge Test Phase is NOW LIVE (Sep 1–Oct 6)! Can your model reason like a child? Can it beat adults? 🥇 $1,000 🥈 $500 for 2 runner-ups Join/submit: eval.ai/web/challenges… @AlisonGopnik @shiryginosar @kate_saenko_ @anisanmajhi @mqraitem

Excited to share our latest work on audio-visual soundscape stylization at #ECCV2024! In this work, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene.

Functional Reward Encodings (FRE) provide a simple, unsupervised approach to offline zero-shot RL: 1. Learn a robust latent space of reward functions, then 2. Train a generalist agent on *random functions*, such that new tasks naturally map to solutions. kvfrans.com/fre/

Our new multimodal generative model can learn in-context from both images and text. One more step towards a unified computer vision tasks solver! Kudos to @Jerry_XU_Jiarui @_amirbar @jw2yang4ai @JianfengGao0217 @trevordarrell and @xiaolonw

Guidance on top of diffusion models can now be used to drag and manipulate images, create pose-conditioned images, and so much more! Check out Readout Guidance: readout-guidance.github.io Work w/ @trevordarrell, @oliver_wang2, @danbgoldman, @holynski_. More in thread 🧵.

Test-Time Training on Video Streams paper page: huggingface.co/papers/2307.05… Prior work has established test-time training (TTT) as a general framework to further improve a trained model at test time. Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders. We extend TTT to the streaming setting, where multiple test instances - video frames in our case - arrive in temporal order. Our extension is online TTT: The current model is initialized from the previous model, then trained on the current frame and a small window of frames immediately before. Online TTT significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets. The relative improvement is 45% and 66% for instance and panoptic segmentation. Surprisingly, online TTT also outperforms its offline variant that accesses more information, training on all frames from the entire test video regardless of temporal order. This differs from previous findings using synthetic videos. We conceptualize locality as the advantage of online over offline TTT. We analyze the role of locality with ablations and a theory based on bias-variance trade-off.

Test-Time Training on Video Streams: Every frame in a test video can be used for training, even without a ground truth label. When deploying your model in a video stream, train it online with self-supervised learning (e.g. MAE). video-ttt.github.io github.com/renwang435/vid…

@Azumanga Yes that's a middle ground I can definitely get behind.

@ren_wang1 However, I would like to be proved wrong here (although I'm not sure how to do that without risking harm -- offer reviewers the option to be deanonymised?)

This latest instance of #ICLR2023 drama really makes me question why reviewers remain anonymous on OpenReview after the review process is over. Authors don't; why should reviewers get to spew vitriolic nonsense under the cloak of permanent anonymity? x.com/CSProfKGD/stat…

@Azumanga And call me utilitarian, but I feel the benefits > risks. On one hand you have nebulous characterizations of potential career blackballing; on the other, you have concrete instances of the mental toll subpar and exceedingly outrageous reviews have on (especially junior) authors

@ren_wang1 The problem is there are plenty of senior people who would use their power to punish people who gave them bad reviews. I would strongly discourage PhD students, postdocs or junior academics from ever reviewing at a conference which deanonymised reviewers.

@Azumanga I disagree with the "plenty of" qualifier. I have no doubt that there is a small subset of "senior people" who would retaliate against negative reviews. But I think the emphasis here is on *small*. I think we all largely hold true negative reviews != bad reviewers.

And maybe even more so than exposing bad reviewers, I'd like to be able go up to good reviewers at a conference and thank them.

Deanonymizing reviewers at some point would go a long way towards improving review and discussion quality. This would be similar to some of the motivation behind publicizing all submitted ICLR papers; I'm less liable as an author to submit something half-baked that way.

Bonus: I will be presenting another work on correspondence learning (Wednesday at 13:30 in Hall E). If you're interested in self-supervised correspondence, definitely take a look at @gao_young's thread here: x.com/gao_young/stat…

7/7: A special thanks also to @olivierhenaff and the rest of his team at DeepMind for their work DetCon (arxiv.org/abs/2103.10957), which made this subsequent work possible. Incidentally, they also have a concurrent work of a very similar nature at ECCV: x.com/olivierhenaff/…

1/7: Excited to share our ECCV work on a new scalable paradigm for self-supervised learning in-the-wild! We iterate ad infinitum between unsupervised generation of segmentation masks and contrastive learning which leverages those masks, all throughout improving representations.