We (Emochi, with 10M+ users) stopped using benchmark scores for production decisions about 6 months ago.
The pattern was consistent: models ranking top on benchmarks underperformed in real usage. Models ranked lower drove better retention.
A few notes on what we learned: