Shreyas Kapur

My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n

@ChungMinKim Incredible work ✨ This is so cool!!

Excited to introduce PyRoki ("Python Robot Kinematics"): easier IK, trajectory optimization, motion retargeting... with an open-source toolkit on both CPU and GPU

wow

@martinsit nice

another Vibe Draw demo! we are LAUNCHING SOON, comment "DRAW" for a link to our waitlist!!!

@JeremyNguyenPhD I'm trying to figure out a good way to share this, since running GPUs is pretty expensive. Though my plans were more so to turn this from a demo to a more polished toy/game 😊

@shreyaskapur Can we play around with it? Or any plans to share code?

I've been waiting 10 years to make this.

@xf1280 I experimented with a bunch of image to 3D models. In this video I'm using fast3d mainly because of very low latency, though in my experiments other models like hunyuan3d and trellis gave better quality meshes.

@shreyaskapur What img to 3d tool you are using? Seems it predicts specular and roughness too

@alexanderchen Thanks Alex! The "system" prompt I wrote specifies that the model should largely follow the user sketch, but if it's a particularly bad sketch, the model is allowed to be creative. It would be so cool to include that more as a slider for user control ✨

This is so cool! Interesting how sometimes Gemini fits your contour very closely but other times it interprets it more loosely like the chess piece. Seems like a nice flexible behavior in this case. I know what you mean ... I've had ideas like "If only one day I could x ..." that are now possible, like this one: x.com/alexanderchen/…

@a_lidayan all credits to you haha

@shreyaskapur potato! :)

@omegablitz_ yesss, that's exactly what I was going for!!

@shreyaskapur scribblenauts on steroids

@sergeykarayev I think @ndea may be Hobbit coded :)

Anthropic is elf-coded, OpenAI is orc-coded, xAI is dwarf-coded, and Google DeepMind is human-coded. This leaves an opportunity for a hobbit-coded research lab.

I wrote up the full results on my blog, shreyaskapur.com/blogs/lateral/ alongside example outputs from models. (2/2)

Can LLMs do lateral thinking puzzles? I tested a bunch of language models on questions from @lateralcast and the #OnlyConnect gameshow! (1/2) 🧵

LMs can generalize to implications of facts they are finetuned on. But what mechanisms enable this, and how are these mechanisms learned in pretraining? We develop conceptual and empirical tools for studying these qns. 🧵

Come check out my tree diffusion poster at the system 2 reasoning at scale workshop at NeurIPS!

Can interpretability help defend LLMs? We find we can reshape activations while preserving a model’s behavior. This lets us attack latent-space defenses, from SAEs and probes to Circuit Breakers. We can attack so precisely that we make a harmfulness probe output this QR code. 🧵

I'll be at NeurIPS, let me know if you want to catch up or chat about program synthesis, world models, neurosymbolic, search, probabilistic programming, or mourning the loss of King Da Ka.

I am currently holding my dad's cryopreserved brain tumor samples in hopes of creating a personalized vaccine for immunotherapy. However, there are some critical and time-sensitive questions in the attached post: x.com/tejasdkulkarni… This is time-sensitive so would appreciate any DMs/RTs.

@EmilevanKrieken I think it has a lot of synergies with GFlowNets (which we mention in the paper) and one of our baseline methods (REPL Flow) is a mix between Ellis et. al. reimagined as a GFlowNet.

@shreyaskapur Hey Shreyas! This is super cool work, congratulations. I wonder if this approach is not more similar to sth like GFlowNets than to Diffusion. I don't think you can do k-step (de)noising. Rather, it seems more like you perform an 'action' at each step, like in GFN.

My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n

@EmilevanKrieken In our current mutation scheme, the expression can get longer or shorter at roughly the same probability, so not sure about the limiting distribution. Anecdotally we noticed that if we noise the program some number of times, the programs resemble just random programs.