Mikel Bober-Irizar

Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵

@Dorialexander I agree, still super impressive for the size!

@mikb0b Not fully but correctly inferring it’s medical domain and trying to connect dots from there. In this extreme size range I’m surprised.

The threshold for consistent English/query understanding is now 3M parameters.

@maxgmcg bird application

twitter 👍

@stupidtechtakes @ScottAndRivers registrars.nominet.uk/uk-namespace/g… I think they added it in response to GDPR

@ScottAndRivers no whois privacy iirc

call me a broke or whatever but i have to afford rent this month, this is what we have to go with for now

@bilaltwovec high quality lighting in general is super underrated

an underrated feature of a city's subway system is having the nice 2700K led lighting instead of the standard harsh white florescent tubes its such an upgrade

@willccbb I need to post more, but I was quite proud of this one: anokas.substack.com/p/llms-struggl…

who are the best up-and-coming technical bloggers on here nowadays? if this is maybe you, feel free to reply w your fav recent post :)

@LouisKnightWebb Thought this was real for a split second and got way too excited

@PiotrZelasko This is awesome! For English specifically, would you recommend parakeet v2 or v3?

Which model is best for what? nvidia/parakeet-tdt-0.6b-v3: blazing fast and accurate ASR inference with PnC and timestamps nvidia/canary-1b-v2: top accuracy with fast inference, ASR and translation, with PnC and timestamps Both are commercially-friendly licensed: CC-BY-4.0

You asked for it, and we listened. MULTILINGUAL Canary v2 and Parakeet v3!! 🌏 25 European languages 🏆 SotA on Multilingual Open ASR Leaderboard 🔥 600x and 2000x faster than real-time 🕰️ Timestamps! 🗣️ Speech translation (Canary) 🃏 Granary: all data is open, train it yourself!

@love_soze_ would love a word for this

growing up i was always bothered that there was no idiomatic expression in english for “on the pareto frontier of most-A and most-B”

@JakeArkinstall @kalomaze spoilers! but yes, tons of fun. I've not actually listened to the audiobook but I've been thinking about it

@mikb0b @kalomaze The audiobook is also fantastic. I dont even like fiction all that much, just listened to it because I was looking for my next listen and the movie trailer looked good. Never thought I'd get so attached to a fictional 5 legged alien spider thing.

for some reason i want to read something on the plane. what should i go for

@DavidSHolz completely agree with this - not to mention cost of each call going way up (even if the list price goes down)

crazy how wait times increased with thinking LLMs and how the experience itself feels net neutral. I desire the response more, but all the *flow-of-the-thing* is gone. the extension of the mind has become an excellent assistant. feeling profoundly not-me. a missed opportunity.

@shitpost9000 @twofifteenam @billyhumblebrag single vent means that it's only blowing hot air out of the apartment, which means negative pressure => sucks an equal amount of outside air back into the apartment. way less efficient but weirdly every portable unit seems to have this flaw

@twofifteenam @billyhumblebrag what does this mean why would there be more than one vent

Americucks coping and seething as i seamlessly install 12 000 BTU of cooling into my European apartment

@microsoft_worm the urge to make a highly engineered custom binary format in every project

growing up and realizing most bespoke file formats/datastores are just zip files or sqlite tables and not highly engineered custom binary formats was my own personal “there is no santa clause”

@keirbradwell love the colours in these

a good thing about making highly amateur photography a hobby is that it forces you to go on more walks

@theo @GenePark i absolutely love the outer wilds promotion grind, a great cause

@GenePark DM me a screenshot when you hit the credits and I’ll throw $500 at a charity of your choice 🫡

My 5 favorite video games of all time. my fight against recency bias is keeping Donkey Kong Bananza off the top 5, we'll see.

@emh203 @LoganDark @lauriewired Wouldn't fixed point make this problem much worse?

@LoganDark @lauriewired yea, fixed point is really simple. Not sure why you would do volume any other way

Fading out audio is one of the most CPU-intensive tasks you can possibly do! Values that get close (but not quite) zero, hit an underflow gap known as "Subnormal" range. It’s a mathematical conundrum so tricky, both x86 and ARM made special CPU instructions just to handle it!

@secemp9 this is kinda neat - hard part might be generating the right question surface for the (avoiding lots of low-quality or similar questions, getting all the weird quirks that happen in programming) or you could just train it on stackoverflow :D

another idea I had: train a model on just books/docs, make it solid at explaining, analogies, reasoning on code (but it would be bad at generating code of course). then use that as a teacher for a new model that would learn programming, stackoverflow-style. basically: - generate a question, get the teacher to answer - turn the whole thing into a feedback loop - distill that into a dataset or use it for RL - or merge the book-thinker and the code-learner, or train a third model on the distilled output. feels like a decent pipeline but I haven’t seen much done like this afaik

@tenderizzation that’s when you write a shell script to ssh into all of them for you (again, rather than reading the docs)

launching the training run by manually sshing into each node and starting the script because you cba to figure out how the cluster management software works

@secemp9 It's hidden, but it runs really well on CPU with github.com/k2-fsa/sherpa-…

no support for ctranslate2 so I need to use either my gpu or add support for it myself...can't believe this is my life

I couldn't believe whisper was SOTA and then found out there is actually a better model from nvidia (WER around 6 vs 9-10 for whisper)