Adam W. Harley

We made a @gradio demo for AllTracker! AllTracker is the current state-of-the-art for general-purpose point tracking. The demo gives a good sense of the accuracy---try your own videos and see for yourself! 🔗 Demo: huggingface.co/spaces/aharley… 💻 Code: github.com/aharley/alltra…

New #NVIDIA Paper We introduce Motive, a motion-centric, gradient-based data attribution method that traces which training videos help or hurt video generation. By isolating temporal dynamics from static appearance, Motive identifies which training videos shape motion in video generation. 🔗 research.nvidia.com/labs/sil/proje… 1/10

Introducing ShapeR, a method for robust conditional 3D shape generation from casually captured sequences. ShapeR leverages a rectified flow transformer conditioned on per-object multimodal data to turn casual image sequences into full metric scene reconstructions. Project Page: facebookresearch.github.io/ShapeR Paper: arxiv.org/abs/2601.11514 Links to code and huggingface below ⬇️

📢 New Paper PointSt3R: Point Tracking through 3D Grounded Correspondence Can point tracking be re-formulated as pairwise frame correspondence solely? We fine-tuning MASt3R with dynamic correspondences and a visibility loss and achieve competitive point tracking results 1/3

Generative Point Tracking with Flow Matching My latest project with @AdamWHarley @CSProfKGD @DerekRenderling @chrisjpal Project page: mtesfaldet.net/genpt_projpage/ Paper: arxiv.org/abs/2510.20951 Code: github.com/tesfaldet/genpt

Happy 80th birthday to the legend Takeo Kanade! 🐐 Thanks @katerinafrag for sharing!

@CSProfKGD @2prime_PKU Must be a different Adam!

@2prime_PKU Are you a co-author @AdamWHarley? 😂

Anyone knows adam?

Yup! But I wouldn't say "single shot" -- AllTracker, inheriting from CoTracker and PIPs and RAFT, is an iterative approach. In practice, we iterate 4x to get the final answer (and this sums to 11 FPS at 576x1024).

Alternate sampling rates.

The visualizer in the AllTracker repo is now GPU-based (much faster), with new options to improve clarity. Very satisfying to see 3D shape "pop out" from the tracking. github.com/aharley/alltra…

@cam_sentinel @_akhaliq @Gradio Real-time version of this should be doable very soon. On-device is possible, but depends on the device of course. The model shown here is only 16M parameters (66mb), but needs a GPU.

@AdamWHarley @_akhaliq @Gradio Impressive advance-how do you see point tracking optimizing for real-time, on-device inference, especially where cloud connectivity is intermittent or data locality is critical?

We made a @gradio demo for AllTracker! AllTracker is the current state-of-the-art for general-purpose point tracking. The demo gives a good sense of the accuracy---try your own videos and see for yourself! 🔗 Demo: huggingface.co/spaces/aharley… 💻 Code: github.com/aharley/alltra…

@georgtrof @Gradio Well, it's great failure case for me to stare at, so thanks for sharing! I think point tracking should eventually be the backbone of pose estimation.

@AdamWHarley @Gradio I think it looks better now. There are still minor issues but it's more a pose estimation problem than tracking.

@georgtrof @Gradio I was thinking static ROI yes (e.g., just crop the video with ffmpeg). And yes, blur and self-occlusion make it harder for the model.

@AdamWHarley @Gradio The camera comes with a fixed focus so cannot move too close. Maybe too many issues here - blurry, dynamic plus self occlusions. Or maybe I shouldn't be tracking the object and just use a static roi?

@mkocab_ @andrew_n_carr @andrew_n_carr I bet you are right that you can classify real/fake from here, but @mkocab_ is right that the visually apparent "cracking" artifact in the background is pretty common overall. This comes from subpixel errors, and maybe subpixel ambiguity in general.

@andrew_n_carr Those carving artifacts don't only happen with generated videos. They occur in real videos too. Here are results from real videos showing the same patterns. This is a failure of the point tracker. Assuming you used AllTracker, @AdamWHarley could probably speak to this better.

synthetically generated videos have fewer and fewer visible artifacts. however, they still have substantial noise. one interesting way to detect this noise is to use dense pixel tracking. here are some veo 3 videos with each pixel tracked over time! look at that pattern

@georgtrof @Gradio Not too bad! A cheap way to improve results here might be to just move the camera closer, or crop and resize the region of interest.

@AdamWHarley @Gradio Nice work, thanks. The model is slightly confused in my case. But it still looks great on deformable surfaces.

Tricky sample for AllTracker. It has never seen water at training time, so I'm glad it knows to quickly discard the tracks there... The bird that travels right-to-left (behind the wings most of the time) could be handled better.

Preprint of (not) today: Harley et al., "AllTracker: Efficient Dense Point Tracking at High Resolution" -- alltracker.github.io Efficient architecture to track all points/pixels in real-time. Matches multiple frames in real time at high resolution via RNN-based refinement.

Happy to report that AllTracker was accepted to #ICCV2025! The twists and turns and methodical experimentation here took at least 12 months in all. Super hard project, though in retrospect our solution is pretty simple. code: github.com/aharley/alltra… paper: arxiv.org/abs/2506.07310

AllTracker is open source and the paper is on arXiv: 🌐 Project page: alltracker.github.io 💻 Code: github.com/aharley/alltra… 📄 Paper: arxiv.org/abs/2506.07310

AllTracker works by estimating long-range optical flow between a query frame and every other frame in a video, incorporating temporal context in a sliding window. Along with flow, it estimates visibility and confidence, so unreliable tracks can be easily filtered out.

AllTracker: Efficient Dense Point Tracking at High Resolution If you're using any point tracker in any project, this is likely a drop-in upgrade—improving speed, accuracy, and density, all at once.