@mrnasrinasir Totally! For this very reason, all of our evals come are within Notion, and we make a point of testing model reasoning in messy context settings.
English
eli(as)
19 posts
@earlierism curious how these stack up inside tools like Notion where the context window gets messy. been testing different models on client databases and.. structured output quality varies way more than benchmarks suggest
English
We’ve often been asked about how we eval each new model at Notion, since we’re one of the few apps deploying both frontier lab models and leading open source models for general knowledge work.
Read on for what we share with the model providers after we evaluate their models.
Notion@NotionHQ
GPT-5.5 is now in Notion 🫡
English
@aNotioneer Good catch! That's indeed part of our larger list of metrics.
Glad you appreciate the report. I'll try to share one tailored to CAs soon :D
English
@earlierism Would % of tool errors also be quite an important metric there or am I missing something? Either way, please keep sharing these results, they’re a great reference point when choosing models for custom agents!
English
@STUDI0BRITTANY Good question! It depends on where we'll ship the model.
E.g.
- on "Auto," price and latency come to the fore,
- on Custom Agent, accuracy parity,
- on Personal Agent, the ceiling of the model's intelligence, to understand what it unlocks for our user
English
@earlierism i’m curious as to which dimension you weight hardest internally before ship: accuracy parity, error floor, or unit economics.
feels like the answer changes which provider wins the slot?
English
@earlierism This is the level of detail I love - pls keep em coming
English
@sarahmsachs @OpenAI what version of GPT 5.5 is this? medium/high/xhigh?
English
On Notion's knowledge work benchmark, GPT 5.5 is 33% faster, uses half the tokens (so half the price), and scores slightly higher than Opus 4.7. @OpenAI has declared themselves the winners, this week, in the frontier knowledge work arena.
eli(as)@earlierism
We’ve often been asked about how we eval each new model at Notion, since we’re one of the few apps deploying both frontier lab models and leading open source models for general knowledge work. Read on for what we share with the model providers after we evaluate their models.
English
eli(as) retweetledi
eli(as) retweetledi
eli(as) retweetledi
From Michelangelo’s “Prometheus Devoured” to Botticelli’s “Birth of an Inbox” witness the renaissance of email.
✍️: @_as_eli
English
eli(as) retweetledi
eli(as) retweetledi
eli(as) retweetledi
eli(as) retweetledi
eli(as) retweetledi
New fave way to announce a a product release: COMICS
As of today, you can turn anything you track in @NotionHQ into a chart. Elias Rimer, a recent grad who landed on our AI team, turned the news into a strip💥
English