Sabitlenmiş Tweet
fig
57 posts
fig
@figbrains
hyperintelligence for humanity
San Francisco 🌁 Katılım Ocak 2024
2 Takip Edilen109 Takipçiler
fig retweetledi
New knowledge pours in so fast at @figbrains that vibe-organizing isn’t cutting it anymore… a good problem to have 🚀
We set up a simple @NotionHQ structure this week. Anyone have tips worth stealing? With a blank slate, this feels like the perfect moment to go AI-native.
English
fig retweetledi
There are several useful lessons in this, but imo most interesting version of the future is:
if open models become so ubiquitous and foundational that they are invisible, AI systems themselves will start to look very different. constellations of increasing capabilities.
there's significant opportunity (and optimism) in that future!
thanks for sharing @Alfred_Lin
Alfred Lin@Alfred_Lin
English
fig retweetledi
The results weren’t what we expected going into this project... but the insights we uncovered are even more valuable than we’d hoped.
Dive into our latest research below. Great work, team 🚀
harsh✌️@HarshSikka
Would training a computer-control model on data built to target its systematic weaknesses actually make it better? We built new browser-use models to find out - and our research revealed surprising results. Sharing new Models, Dataset, Paper, and Data Pipeline below. This work was done by the @figbrains team in collaboration with @ManifoldRG
English
fig retweetledi
We’re excited to share the results of another collaboration with the team at @figbrains! GUI-Perturbed is part of GOLEM, Manifold’s long-term research program focused on building AI systems that can perceive, reason, adapt, and act across any input and output modality.
harsh✌️@HarshSikka
Would training a computer-control model on data built to target its systematic weaknesses actually make it better? We built new browser-use models to find out - and our research revealed surprising results. Sharing new Models, Dataset, Paper, and Data Pipeline below. This work was done by the @figbrains team in collaboration with @ManifoldRG
English
@HarshSikka This is amazing work Harsh!! The models are memorizing absolute pixel positions! Rather than building a spatial model or a semantic model of the page (and this is something scaling will NOT solve)! I also love how your work separates the different failures that benchmarks blur together:
• robustness (does net accuracy hold?)
• consistency (do individual predictions stay stable?)
Most evals miss that entirely because they average these out.
This is the type of science and engineering we need to make AI reliable.
Great work @figbrains and @ManifoldRG!!
English
fig retweetledi
counterintuitive but important. Fine-tuning on failure cases degraded performance, more data made it worse. So bottleneck is the recipe and base model, not the source. You can't patch spatial robustness onto a model that never learned it. Excellent work @figbrains and @ManifoldRG
harsh✌️@HarshSikka
Would training a computer-control model on data built to target its systematic weaknesses actually make it better? We built new browser-use models to find out - and our research revealed surprising results. Sharing new Models, Dataset, Paper, and Data Pipeline below. This work was done by the @figbrains team in collaboration with @ManifoldRG
English
fig retweetledi
Would training a computer-control model on data built to target its systematic weaknesses actually make it better?
We built new browser-use models to find out - and our research revealed surprising results.
Sharing new Models, Dataset, Paper, and Data Pipeline below.
This work was done by the @figbrains team in collaboration with @ManifoldRG
GIF
English
fig retweetledi
Yesterday, @figbrains released GUI-Perturbed, a new dataset of realistic website changes that browser-use / Computer Control models really struggled with.
If you were trying to improve these models, do you think fine-tuning them on this data would help, or hurt them?
English
fig retweetledi
Dataset here: huggingface.co/datasets/figai…
This work is a small iteration building on tremendous research from folks like @xwang_lk, @TianbaoX, @PangWeiKoh, @lateinteraction, @ZhiruoW, @Adamlu28 and many others.
What an exciting time to be working in this field!
English
fig retweetledi
GUI-Perturbed is open source. Use it to evaluate your own models.
Pipeline: github.com/ManifoldRG/GUI…
Technical Report: fig.inc/blog/gui-pertu…
Results Viewer: huggingface.co/spaces/figai/G…
English
fig retweetledi
fig retweetledi
The team @figbrains, along with our friends @manifoldrg, took three of the best computer-use models and, surprisingly, broke all of them with very simple perturbations like changing zoom or colors.
Read on to understand our research, including a new SoTA Evaluation Dataset for Browser-use models + a new kind of interactive data sandbox!
GIF
English
fig retweetledi
At @figbrains, we’re testing frontier models (Fable, Kimi, etc) on simple web tasks that should be solvable.
They failed in ways that wouldn't stump a human (we think)
Results coming in a few days, but we want to see how good humans are:
Which change causes the most failures?
English
fig retweetledi
GUI-DR confirms an intuition we at @figbrains have had for a while: today’s computer control models often overfit to specific interfaces rather than learning the underlying task. Systematic GUI perturbations significantly reduce model performance. Read more below!
harsh✌️@HarshSikka
Computer Control models can score 90%+ on standard benchmarks, but will fail when you set page zoom to 70%. We're built GUI-DR, an OS pipeline that can restyle, reposition, and remove DOM elements on real webpages to reveal model weaknesses that fixed-scene benchmarks miss.
English
fig retweetledi
It was great working with @figbrains on GUI-DR!
We applied domain randomization from robotics to vary visual scenes and instructions, exposing fragile model behaviors like confusing the browser search bar with the formula bar in Google Sheets.
harsh✌️@HarshSikka
Computer Control models can score 90%+ on standard benchmarks, but will fail when you set page zoom to 70%. We're built GUI-DR, an OS pipeline that can restyle, reposition, and remove DOM elements on real webpages to reveal model weaknesses that fixed-scene benchmarks miss.
English
fig retweetledi
The Software Control research team at Manifold has been working on advancing new frontiers in long horizon computer control & grounding with @figbrains
Check out some of our early research below, with more to come soon!
harsh✌️@HarshSikka
Computer Control models can score 90%+ on standard benchmarks, but will fail when you set page zoom to 70%. We're built GUI-DR, an OS pipeline that can restyle, reposition, and remove DOM elements on real webpages to reveal model weaknesses that fixed-scene benchmarks miss.
English
GUI-DR applies domain randomization from robotics, varying visual scenes and instructions along controlled axes to expose fragile model behaviors.
harsh✌️@HarshSikka
Computer Control models can score 90%+ on standard benchmarks, but will fail when you set page zoom to 70%. We're built GUI-DR, an OS pipeline that can restyle, reposition, and remove DOM elements on real webpages to reveal model weaknesses that fixed-scene benchmarks miss.
English
Fig wants to directly support researchers working on foundationally new takes on frontier models - targeting hard problems like long horizon multi-environent action.
Reach out to contact @ fig . inc if you're working on these or related areas.
Pranav Guruprasad@pranavguru13
This week at #CVPR2026 we presented MultiNet v1.0 at the MMFM workshop. It is a benchmark built around a question most evaluations skip: what happens to a multimodal model when you take it out of the one domain it was trained for and ask it to handle everything at once?
English
fig retweetledi
Loved @pliang279’s #CVPR2026 talk on AI modalities beyond vision/language: touch, smell, etc. The vision-tactile retrieval work reinforces that good representations make hard-to-observe signals queryable. We’re applying a similar lens to trajectories at @figbrains. More soon!
English