Simon Stepputtis

Thrilled to join @virginia_tech as an assistant professor in @VirginiaTech_ME this fall! At the TEA lab (tealab.ai), we’ll explore hybrid AI systems for efficient and adaptive agents and robots 🤖 Thank you to everyone who has supported me along the way!

If you’re interested in robot learning, hybrid AI systems, or AI/ML for autonomous systems, I’ll be recruiting students soon and would love to connect!

Thrilled to join @virginia_tech as an assistant professor in @VirginiaTech_ME this fall! At the TEA lab (tealab.ai), we’ll explore hybrid AI systems for efficient and adaptive agents and robots 🤖 Thank you to everyone who has supported me along the way!

I am excited to announce that we are accepting applications for the Pathways@RSS 2025 Fellowship! 📌 Fellowship for BSc, MSc & early PhD students 📌 Supports first-time RSS / Conference attendees 📌 Attend & thrive at #RSS2025 🗓️ Find out more and apply by March 21st: sites.google.com/lehigh.edu/pat…

@joecampb Thank you for your kind words! It is great working with you too!

Having worked with @SimonStepputtis for more than 7 years, I can say that he is an incredible colleague. He is exceptionally talented and collegial and, importantly, is a great mentor for students. If your department is looking to hire in ML + Robotics, don't miss this chance!

📣 Thrilled to announce that I'm on the job market for Fall 2025 faculty positions! I am currently a postdoc @CarnegieMellon @CMU_Robotics. 🔍 My research is dedicated to developing robots that can intelligently reason about their environments and the humans within them. By considering the affordances, attributes, and relationships of objects in the environment, my work enables robots to efficiently learn behaviors and generalize them to novel settings 🚀 ⭐️ My focus is on developing neurosymbolic models that combine the expressivity of deep neural networks with the reasoning abilities of symbolic AI. 🤖 I have previously applied my work to applications such as in-home assistance, manufacturing, and beyond. 🌐 Learn more about my work at simonstepputtis.com 🔭 If you think I might be a good fit for a position at your institution, please don't hesitate to reach out!

🚀 Excited to share our latest work: extending RLAIF to work well with small language models which may produce incorrect rankings! 🚀 Catch our poster session at #EMNLP in Miami next Thursday! 📄 Paper: arxiv.org/pdf/2410.17389 🎥 Presentation: youtu.be/jKKUqB-nf7Y

🚀 Excited to share our paper ShapeGrasp: Zero-Shot Object Manipulation with LLMs through Geometric Decomposition at #IROS2024 (Session ThCT3.3)! 🎉 Amazing work by @SamuelLi826114 on leveraging geometric decomposition and LLMs for zero-shot task-oriented grasping. The LLM dynamically generates hypotheses on what each geometry is for and selects the best one for the task! 🦾🤖 Explore more at: shapegrasp.github.io

Have you ever wondered how robots can proactively support a human’s task? 🤖🧑‍🍳 A key aspect of proactive user support is fast action anticipation. ⭐ Our method uses external domain knowledge to reduce response time and selects the most helpful, non-disruptive task. ⏱️🥗 Below, our system quickly predicts that the user wants to make a dressing and adds further ingredients for the dressing. 🤖🧂 In our new paper, we present an approach for action anticipation from short video contexts in a collaborative kitchen task, preparing salads. 🥗 Through our neuro-symbolic approach, our method quickly and accurately selects helpful actions! 📜 Paper: ieeexplore.ieee.org/document/10582… 🌎 Website: sarthak268.github.io/NeSCA/

@liuziyi93 Really nice paper! We found similar things when we were exploring how well LLMs were able to identify player roles in Avalon Paper and Dataset: aclanthology.org/2023.findings-…

🤔Are LLMs capable of demonstrating social intelligence in a complex interactive game environment? Check out our recent work: arxiv.org/abs/2406.12203 We introduce ‘InterIntent’ to study social intelligence via intention understanding in the Avalon game. [1/x]

@joecampb @PurdueCS Congratulations! Exiting news to see you move to a professor position!

I am very excited to share that I will be joining @PurdueCS as an Assistant Professor in Fall 2024!! Many thanks to my friends and colleagues for their support during this process. More updates coming in the near future.

I'm excited to share our lates work, Sigma, providing an efficient approach to multimodal segmentation that can overcome RGB-only shortcomings like low-light or overexposure issues by reasoning over thermal and depth information. At the core of our multimodal method, sigma achieves global receptive field coverage with linear complexity by utilizing recent advances in Mamba! Please take a look at our paper and code! Paper: arxiv.org/abs/2404.04256 Code: github.com/zifuwan/Sigma

Ever wanted to know how out-of-distribution data will affect a deep neural network's accuracy? @yuzhe_lu is at #NeurIPS2023 this week presenting our paper which shows how to predict model accuracy given only unlabeled data from the target dist. Paper: arxiv.org/abs/2305.15640

In our latest EMNLP 2023 Findings paper, we delve into the intricate realm of Avalon: The Resistance, a complex social deduction game that requires a nuanced understanding of long-horizon player interactions, their strategic discussions, as well as deception and persuasion amongst them. When predicting hidden player roles, we find that LLMs struggle identifying and tracking deceptive user behavior. To address this challenging setting, we introduce a new NLU dataset of 20 human-played games grounded in the game's state information. With labeled utterances for deception and persuasion, we hope that our dataset opens up future development in understanding deceptive behavior, as well as many other research avenues! Our paper will be presented at the Novel Ideas in Learning-to-Learn through Interaction Workshop at @emnlpmeeting today (December 7th)! Full Paper: arxiv.org/abs/2311.05720… Website: sstepput.github.io/Avalon-NLU/

Please check out our ICCV CVEU workshop paper Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos in which we utilize symbolic domain knowledge to alter a transformer's attention mechanism to improved future action prediction. Link: arxiv.org/abs/2309.05943

Please check out our new paper at the IROS-HmRI workshop!

Today, we are presenting our work "Sample-Efficient Learning of Novel Visual Concepts" at @CoLLAs_Conf 2023. Check out our poster this afternoon at poster 12! Project Website: sarthak268.github.io/sample-efficie…

What if transfer learning in RL could selectively transfer only beneficial knowledge? What if it was interpretable and we knew exactly what was transferred? I have a poster today on exactly this topic at CoLLAs. Check it out: arxiv.org/pdf/2306.12314… #CoLLAs2023

I am excited to announce our talk at #CoLLAs2023 for our paper titled "Sample-Efficient Learning of Novel Visual Concepts"! With our neuro-symbolic architecture, we quickly learn about novel objects, as well as their attributes and affordances. Website: sarthak268.github.io/sample-efficie…

Our workshop on Articulate Robots is happening right now at #RSS2023 in Room 306A and online! More information: sites.google.com/andrew.cmu.edu…