Mecado

Amazing working with @MikushRab and @huggingface on this! CADGenBench is the most comprehensive AI-for-Mechanical-Design benchmark out there right now, and it tests 2 important things: Can a model understand 2D representations of 3D? Can it understand unlabeled geometry?

Fable is the new leader on CADGenBench! Still long way to go:

@claudeai @MikushRab You know what to do

Introducing Claude Fable 5: a Mythos-class model that we’ve made safe for general use. Its capabilities exceed those of any model we’ve ever made generally available.

@BrierRat Excited to see how you do on this benchmark! huggingface.co/spaces/Hugging…

3.923 lbs, correctly answered by Claude CAD and build123d! This was a screenshot, sorta non-standard, drawing. Base-Claude is pretty bad out of the box. Here's hoping the hand-holding I did and the resulting skills file helps future drawing interpretations

@sonichNik There should be a benchmark for this... huggingface.co/spaces/Hugging…

new side quest: trying every AI CAD startup/tool I can find to see if any of them are actually good enough. drop names below

@calebfahlgren Many more to come! Anything you'd like to see in our next one?

This is super cool! Benchmark measuring how well agents can use CAD to build a valid, geometrically correct 3D part! We need more benchmarks that model real world tasks like this!

@2cantsam Benchmarks on functional engine mechanisms coming soon :)

This is great. Can’t even model a fully functional engine is the new too many fingers.

What cannot be measured cannot be improved! As @mecadoinc shows in our work with @huggingface, there's a whole lot of work to do to create AI systems that really work for mechanical design. And these are just single parts - no requirements or assembly context!

@MikushRab Can LLMs understand a drawing? Can they infer design intent? (The answer is... maybe not yet)

Introducing CADGenBench: measure how well AI systems produce engineering-grade 3D parts! While current models can generate 3D parts, they are far from precise enough to build functional parts. We built a benchmark to systematically measure their capabilities on two tasks: 1. Generation from an engineering drawing of a part 2. Editing: given an existing STEP file and a requested change The benchmark is tool-agnostic. It makes no assumptions about how you build the model. You can vary the LLM, and you can vary the environment. Use build123d, Onshape, Autodesk, or a model without an LLM entirely. We open sourced the scoring engine and a reference baseline on top of build123d. A collaboration between Hugging Face and @mecadoinc! Submission space: huggingface.co/spaces/Hugging… Code repository: github.com/huggingface/ca…