Simon FL

We just published OfficeQA Pro - a set of 133 challenging questions from the original OfficeQA benchmark. Even the best frontier agents still struggle on OfficeQA Pro with common issues stemming from errors in parsing, retrieval, and visual reasoning.

Most AI benchmarks test reasoning in isolation. Real enterprise tasks require grounded reasoning: 1️⃣ Find the right documents 2️⃣ Extract the right values 3️⃣ Perform analyses OfficeQA Pro evaluates this end-to-end. Frontier agents still score <50%. 🧵Paper & details below!

Today we’re releasing OfficeQA — a new benchmark for end-to-end grounded reasoning that reflects the real work enterprises need AI agents to do. More details below 👇

Why is it "two buck chuck" and not "two bucks chuck"? Is this like the "maple leafs" vs "maple leaves"? English plurals are insane

Since joining @databricks, our research team has been hard at work on Agent Bricks, a new product that helps enterprises develop state-of-the-art domain-specific agents. We are now releasing a research blog about Agent Learning from Human Feedback (ALHF) databricks.com/blog/agent-lea…

RLVR isn't just for math and coding! At @databricks, it's impacting products and users across domains. One example: SQL Q&A. We hit the top of the BIRD single-model single-generation leaderboard with our standard TAO+RLVR recipe - the one rolling out in our Agent Bricks product.

Hey @minimax_ai, I'm trying to serve M1-80k on vLLM. Your docs say "a server with 8 H800s can process inputs up to 2 million tokens" but then recommend --max_model_len 4096. What settings did you use for 2M tokens? I'm trying this on 8 H100s.

@harryh I sadly have no experience to offer then.

@simonfl AC not PTAC. All a central system for the whole apartment. Not window based.

AC guy sent me a $1,400 bill for some maintenance. Says a new transformer (no idea what kind exactly) was $500 and a new pump was $600. Do spare parts for an AC really cost this much? Feels a bit like I'm getting ripped off.

@clementmiao @ffx You had me at "split keyboard on your pants"

@ffx no i want it to be prominent, with very bright rgb leds, and it makes loud laser sounds for each key press

I don't have one yet, but seems like the biggest issue with the Vision Pro for productivity on the go is the bad virtual keyboard when you're not connected to your mbp. I can see split keyboard on your pants being popular within certain groups as a solution.

DONDA is the new FAANG Deepmind Open AI Nvidia Databricks Anthropic

@what_mara_said same

All I want to do, is relax, in the sky, in a very comfortable hot air ballon, and eat swedish fish

@harryh My bad, I must have misconstrued something I heard!

@simonfl I don't think that was my mindset? I've just never had the stability in my life to potentially buy before.

I was about 1% apart from a seller on an apartment negotiation and he table flipped, pulled his last offer, and is apparently delisting the apartment. He's gonna ride it out with his low mortgage and see what happens. Sign of the times I guess.

@zackdotcomputer @etrade Be ready for that 1 cent to be transferred to Morgan Stanley soon!

About 10 years ago I closed my @etrade account. They messed up the math and left it with $0.01 after closing. A decade later they're still sending monthly statements for this penny, but I can't tell them to just keep it because "only active accounts can call customer service." 🤦🏻‍♂️

First day on the job at @databricks and we're already making some big moves. Exciting times ahead!

I feel like today's war on cars is so much tamer than it was 80 years ago

@harryh Sidney is much better at this, except it gets confused easily, I think because of the other people who will show up on your LinkedIn profile, it thought I currently had @leok 's job . I assume it's the same problem with ChatGPT

TIL that I was once the VP Eng at Digg.

A source has leaked to me a transcript of a recent phone call between @elonmusk and @kanyewest.

Now this is AI thought leadership at its finest. A+. No notes

@TheKhandyman is you taking notes on a criminal fucking conspiracy?

Having older colleagues is great because they understand all of my Wire references immediately