Haijun Xia

🎥 New Talk: “Generative, Malleable, and Personal User Interfaces”. This talk describes our work toward the long-held vision in the field of human-computer interaction. Talk Link: youtu.be/MbWgRuM-7X8 We are committed to making it as real as we can.

Excited to see this exploration from @GoogleLabs! My lab has been exploring Generative & Malleable Interfaces for a while. Check out this project we did last year youtu.be/X3cf1U1dFGQ, and its research publication at ACM SIGCHI 2025 dl.acm.org/doi/10.1145/37…

@geoffreylitt @YiningCao3 @peilingjiang Glad you enjoyed the work! Both questions are related to the notion of X-on-demand. Formality, interpretability, or something else. A challenge of malleability is how to layer the different levels of abstraction so it's easy to unwrap, like the ladder of skills in your notion.

If you're thinking about AI-generated UIs, recommend checking out JELLY by @YiningCao3, @peilingjiang, and @HaijunXia. My favorite kind of work: both a compelling system/demo AND a bigger idea that people can build on! talk video: youtube.com/watch?v=X3cf1U… paper: dl.acm.org/doi/10.1145/37… tldr: vibe-coded UIs aren't ideal for users generating software, because it's hard to steer the generation and keep things consistent. They propose solving this by first generating a more structured model of the user's needs, including a data schema that the user can see/edit. Then UIs get generated based on this schema, but it feels more like fluidly composing premade widgets in a task-specific way than building a new "application". Reminds me of @alexobenauer's work on an itemized OS and @jasonyuan's Mercury concept, as well as the Embark system that I worked on. The demos feel compelling and magical, but there's also enough technical meat to see how this is actually feasible today with LLMs. Really cool. Things I'm not so sure about: - I like formality on demand: super unstructured representations (text, drawings) and only adding structure when needed. It seems like Jelly jumps straight to rigid relational models. Good fit for some tasks but not all. I wonder about fitting in less-structured bits and then structuring on-the-fly with LLMs. (As a mitigating factor: the fact that you can edit the schema live on the fly does help a lot, blurring the line between using and creating the software. And structure is really useful for things like different views of the same info) - I'm curious how much the exposed schema ends up really being useful to users for understanding. Their own user study found the majority of users just relied on the UI rather than the schema. Feels like there's a lot more work to do here to achieve deeper interpretability. The challenge of "how do you tell users what software does without showing code" is endlessly deep...

@msbernst Congratulations Michael! Your work has been truly inspiring!

Thank you to everyone for your energy and enthusiasm in joining this adventure with me so far!

@imawasif Not sure how much consistency is desired. Each website has its unique design and content, and I would prefer the websites to continue to have their styles in their chatbots, instead of all through a single voice?

@HaijunXia Loss of consistency may be a bigger concern, as each AI agent seems to be "chatting on behalf of" different modules. Who do you think should be responsible for coordinating the semantic boundaries of these agents in the long run?

Another thought: why do I want to chat through Dia about a website. If you want me to chat, I want to chat with the website itself, directly.

In the next few years, the integration of AI will make interfaces absolutely ugly. I’m already seeing 2–3 AI buttons or chatboxes crammed into a tiny area on screen. One thing I know for sure -- more is coming!

Please RT! We're seeking data analysts using AI in their workflows. We would love to observe your process and learn what has been exciting or challenging. Your insights will help us build better tools. If you are willing to sit down with us (on zoom) for a couple of hours (compensated), please fill in this form! forms.gle/8o3Y1KbZHwn97x…

Web Browsing Reimagined. We propose 𝗕𝗿𝗼𝘄𝘀𝗶𝗻𝗴 𝗮𝘁 𝗦𝗰𝗮𝗹𝗲 with #Orca. Orca supercharges your web browsing with AI. View, navigate, and operate as many tabs as you want, all at the same time. More at hci.ucsd.edu/orca

A step towards Generative, Malleable, and Personal user interface led by my student @BryanDMin !

@MatthewWSiu @yoshikischmitz Yep. We want to set up a good, foundational environment for humans and agents.

@HaijunXia @yoshikischmitz A malleable computing environment would also help with agentic workflows Computer-use agents could first specify an information environment that is more suitable for doing the task while still automating certain repetitive, time-demanding tasks for the end-user

🎥 New Talk: “Generative, Malleable, and Personal User Interfaces”. This talk describes our work toward the long-held vision in the field of human-computer interaction. Talk Link: youtu.be/MbWgRuM-7X8 We are committed to making it as real as we can.

@MatthewWSiu @yoshikischmitz Yes. @yoshikischmitz's prototype is definitely relevant and has been discussed a few times during our lab meetings.

This was a great talk, I liked how you framed the types of AI supporting activities at the start Your work with Bryan on Malleable Overview-Detail Interfaces reminds me of this prototype @yoshikischmitz did back in the day (pre-LLMs)... we are getting closer! x.com/yoshikischmitz…

📢CHI'25 workshop: Tools for Thought: Research and Design for Understanding, Protecting, and Augmenting Human Cognition with Generative AI 🌐aka.ms/toolsforthough… 🗓️Submission deadline: Feb 13, 2025 AoE

💼 I'm on the job market for tenure-track faculty positions or industry research scientist roles, focusing on HCI, Human-AI interaction, Creativity Support, and Educational Technology. Please reach out if hiring or aware of relevant opportunities! RT appreciated! 🧵 (1/n)

I am on the job market, seeking tenure-track or industry research positions starting in 2025. My research combines human-computer interaction and robotics—please visit karthikmahadevan.ca for updated publications and CV. Feel free to reach out if interested. RT appreciated!

@plopesresearch @acm_chi The decision on mine has been reverted, and I am hearing other ones too!

@HaijunXia Wow. I'm completely shocked. @acm_chi I'm assuming this is just some automatic filter/ pcs setting that was incorrectly set? We can't really be judging papers based on the length of the title, right? P.s. a few of those short titles are mine apparently, so yeah, I'm biased :)

"Insight Nugget" is absolutely spot on. More so as I just get one after sitting down for 3 hours sketching-reading-battling with distractions. Miss the good old days when I can almost do this on a daily basis.

Come check out a new form of augmented presentation, done by my student Rima (@YiningCao3) and wonderful colleagues at Adobe!

Dev is going to present at Rm 324, in the session of Supporting Programmers and Learners A this morning after the keynote. A glimpse of how graphics can be integrated with programs. 🏅 #CHI2024