Toh Jing Hua

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: arena.xlang.ai Leaderboard (tentative): arena.xlang.ai/leaderboard Blog: arena.xlang.ai/blog/computer-… Data & Code (coming soon): github.com/xlang-ai/compu… ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n

We finally have an official `nvidia-smi` for TPU 🎉 Simply install it with `pip install tpu-info`

OSWorld has been accepted by NeurIPS 2024 D&B track! 🎺✌️ Again, graceful thanks to all of our collaborators for their invaluable contributions to the project: @_zdy023, @chenjx210734, @xiaochuanlee, @SihengZhao, @RuishengC49326, @nikushii_, @ChengZhoujun, @dongchan, @fangyu_lei, @taoooo917, @yihengxu_, @shuyanzhxyc, @silviocinguetta, @CaimingXiong, @hllo_wrld, @taoyds; and @sidawxyz, @ptshaw2,@ChenHenryWu,@pengchengyin,@ShunyuYao12,@xhluca,@sivareddyg,@ruoxi_cc ,@LukeZettlemoyer, @ZhiyuanZeng_, @_TobiasLee, @zywu_hku, Chengyou Jia for their helpful feedback on this work!! Also thanks to contributors who help with improving this ecosystem and trust it, good and still behind~ Let's go over this video again!!!

#1 trending github repo right now looks INSANE Single image to live stream deep fake. Look at that quality!! It's called Deep-Live-Cam (link in replies)

comic page in under 5 minutes (realtime!!!). all you have to do is: - set up the panels - type in ur story prompt - choose best variations (retry if necessary) - adjust text layers - adjust crops of images - adjust style (optional)

Working idea but I've noticed a bunch of archetypes of AI researchers & engineers in my career. 😂 Here are some of them: 1. Carry: Hero-level person capable of making unprecedented (alone or in a small group). Either in terms of modeling, infra or making impact in general. Very rare entity and often hands on front-line fighter. 2. Support: Decent execution and can boost the effectiveness of teammates (or other carries). Can take on assigned tasks independently, does a lot of role support, running side experiments or writing code to do so, polishing papers, making beautiful plots etc. 3. Magicians: Researchers that may not have the best coding ability (e.g., frontline grinding) but very good at coming up with 10x research ideas that can be a game changer. 4. Warriors: Infra-class people who may not have good research ideas but write very solid code. Can't train models but build beautiful infra. 5. Tanks: People not doing actual work (damage) but somehow tanking other problems and responsibilities. 6. Feeders: People who try to help, but can't do much due to skill issue. 7. AFK: People who don't care and just coast. 8. Leechers: People who hover around projects hoping to farm/leech XP (credit) from others by just being around or just talking and talking (and not doing). Not hard categories but most people are mixtures I guess. For instance, a mix of the above (e.g., 70% warrior 30% mage). I think this is an interesting way to think of roles in executing research. Can you map people you know into these archetypes? 😂 Disclaimer: just for fun don't be offended :), mostly gaming terms inspired. I find my friends use these terms a lot "oh this guy is a carry", "that guy is a support" etc.

Audio Mamba Bidirectional State Space Model for Audio Representation Learning Transformers have rapidly become the preferred choice for audio classification, surpassing methods based on CNNs. However, Audio Spectrogram Transformers (ASTs) exhibit quadratic scaling

OSWorld Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing

🚀Multimodal agents is on rise in 2024! But even building app/domain-specific agent env is hard😰. Our real computer OSWorld env allows you to define agent tasks about arbitrary apps on diff. OS w.o crafting new envs. 🧐Benchmarked #VLMs on 369 OSWorld tasks: #GPT4V >> #Claude3

🤔Can we assess agents across various apps & OS w.o. crafting new envs? OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS. + annotated 369 real-world computer tasks 👇os-world.github.io

Today we’re releasing OpenEQA — the Open-Vocabulary Embodied Question Answering Benchmark. It measures an AI agent’s understanding of physical environments by probing it with open vocabulary questions like “Where did I leave my badge?” More details ➡️ go.fb.me/7vq6hm All of today’s state-of-art vision+language models (VLMs) fall well short of human performance. In fact, for questions that require spatial understanding, today’s VLMs are nearly “blind” – access to visual content provides only minor improvements over language-only models. We hope that OpenEQA motivates additional research into helping AI understand and communicate about the world it sees.

Will there be significant architectural updates to NLP models in the next two years? It seems that the current extensive amount of model parameters is impeding the exploration of new model architectures.

Introducing Gemini 1.0, our most capable and general AI model yet. Built natively to be multimodal, it’s the first step in our Gemini-era of models. Gemini is optimized in three sizes - Ultra, Pro, and Nano Gemini Ultra’s performance exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks. With a score of 90.0%, Gemini Ultra is the first model to outperform human experts on MMLU. blog.google/technology/ai/…

Excited to present Pix2Act! An agent that can interact with GUIs using the same conceptual interface that humans commonly use — via pixel-based screenshots and generic keyboard and mouse actions -- arxiv.org/abs/2306.00245 (1/4)

How far can we go with vision alone? Excited to reveal our Large Vision Model! Trained with 420B tokens, effective scalability, and enabling new avenues in vision tasks! (1/N) Kudos to @younggeng @Karttikeya_m @_amirbar, @YuilleAlan Trevor Darrell @JitendraMalikCV Alyosha Efros!

With collaborators @Google we're announcing 💫 ZipLora 💫! Merging LoRAs has been a big thing in the community, but tuning can be an onerous process. ZipLora allows us to easily combine any subject LoRA with any style LoRA! Easy to reimplement 🥳 link: ziplora.github.io

Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation paper page: huggingface.co/papers/2311.08… To maintain user trust, large language models (LLMs) should signal low confidence on examples where they are incorrect, instead of misleading the user. The standard approach of estimating confidence is to use the softmax probabilities of these models, but as of November 2023, state-of-the-art LLMs such as GPT-4 and Claude-v1.3 do not provide access to these probabilities. We first study eliciting confidence linguistically -- asking an LLM for its confidence in its answer -- which performs reasonably (80.5% AUC on GPT-4 averaged across 12 question-answering datasets -- 7% above a random baseline) but leaves room for improvement. We then explore using a surrogate confidence model -- using a model where we do have probabilities to evaluate the original model's confidence in a given question. Surprisingly, even though these probabilities come from a different and often weaker model, this method leads to higher AUC than linguistic confidences on 9 out of 12 datasets. Our best method composing linguistic confidences and surrogate model probabilities gives state-of-the-art confidence estimates on all 12 datasets (84.6% average AUC on GPT-4).

“What should I work on?” is a question we hear more & more often from NLP students, during a time when the media rhetoric is that “it’s been all solved” Turns out there are many NLP research areas rich for exploration—here is our answer from 20+ students arxiv.org/abs/2305.12544