Miha Jenko

@EmilWallner I used ClearML in the past, it's battle-tested: github.com/clearml/clearml

✨Introducing GPU Benchmark Tool A simple tool to test the health and performance of your NVIDIA GPU and see its performance. Let's be honest, FurMark and 3DMark suck 😂 and they are not really testing AI performance.

Summary: Huawei trained a fairly big AI model entirely on their own hardware, with no Western GPUs. On the one hand, this shouldn’t be shocking to anyone. On the other hand, I know lots of people who claim that this couldn’t happen, so I thought it was worth reposting. (I expect a lot of people in the West will be even more shocked when the Chinese catch up on chip process technology.)

You should use Signal. Seriously. There are other encrypted messaging apps out there, but I don’t have as much faith in their longevity. In particular I have major concerns about the sustainability of for-profit apps in our new “AI” world.

. @sh_reya 's paper confirms what I see in practice 1) Automated evals don't work (without semi-manual human alignment) 2) Most tools don't provide this alignment 3) Automated evals add mostly noise 4) You can only write good evals by looking at data and reacting to failures

@LeonDerczynski Researches can fall into the trapping offered by step-wise learning robustness. Since the algorithms are robust to lone errors in the training data, we develop habits to follow up on the training error whilst failing to apply the same rigor on test data too.

Why are NLP benchmarks consistently so poor quality? MT-Bench' resason, math, and coding eval instances were wrong in *25%* of cases Utter disaster. Easy fix: Look at your data with your eyes and be prepared to reject it

@mtrc We are also going to need lossless video-audio on the internet, because these days it's hard to trust a low-res video with bleepy audio to not be AI-generated.

dude's got it bang on. the one thing we're definitely not going to need, in a world increasingly full of computers, is people who understand computers.

Hype around AI in medicine often ignores two key risks: - patterns of automation contribute to centralization of power - medical knowledge is limited by the systemic refusal to trust patient expertise 1/ my new post: rachel.fast.ai/posts/2024-02-…

today i found out that this one australian guy has been toiling away making incredibly detailed Neural Circuit Diagrams with the vibe of a 1950s issue of Popular Mechanics, but content fit for the 2020s behold. the Transformer

@lvwerra This only works for you because you have some context already.

Why waste time say lot word when few word do trick? The ultimate ChatGPT prompt for productivity boosts! (ref: youtube.com/watch?v=VvPaEs…)

@EmilWallner @patriciobcs Only for certain regions 😔

@patriciobcs it’s under search > made for you

@BlancheMinerva Feels like a Miro board to me.

Does anyone have recommended tools / templates for creating some of the standard diagrams in transformer papers like the ones below?

A fact worth highlighting: NVIDIA is making its own *CPU*, and will increasingly excel at it. To max out GPU's performance, building CPU in-house is an inevitable path. Below is GH200, the first superchip that includes all home-grown components: CPU (Grace), GPU (Hopper), and NVLink Chip-2-Chip (C2C) interconnect (delivers up to 900 GB/s total bandwidth). On GH200, CPU and GPU can access each other's memory freely and concurrently. Whitepaper: resources.nvidia.com/en-us-grace-cp…

I don't talk much about this - I obtained one of the first FDA approvals in ML + radiology and it informs much of how I think about AI systems and their impact on the world. If you're a pure technologist, you should read the following: There's so much to unpack for both why Geoff was wrong, and why his future predictions should not be taken seriously either. Geoff made a classic error that technologists often make, which is to observe a particular behavior (identifying some subset of radiology scans correctly) against some task (identifying hemorrhage on CT head scans correctly), and then to extrapolate based on that task alone. The reality is that reducing any job, especially a wildly complex job that requires a decade of training, to a handful of tasks is quite absurd. Here's a bunch of stuff you wouldn't know about radiologists unless you built an AI company WITH them instead of opining about their job disappearing from an ivory tower. (1) Radiologists are NOT performing 2d pattern recognition - they have a 3d world model of the brain and its physical dynamics in their head. The motion and behavior of their brain to various traumas informs their prediction of hemorrhage determination. (2) Radiologists have a whole host of grounded models to make determinations, and actually, one of the most important first order determination they make is whether there is anything notably wrong with a brain structure that "feels" off. As a result, classifiers aren’t actually performing the same task even as radiologists. (3) Radiologists, because they have a grounded brain model, only need to see a single example of a rare and obscure condition to both remember it and identify it in the future. This long tail of rare conditions to avoid missing is a large part of their training, and no one has any clue how to make a model that acts similar in this way. (4) There’s so many ways to make Radiologist lives easier instead of just replacing them, it doesn’t even make sense to try. I interviewed and hired 25 radiologists, whose primary and chief complaint was that they had to reboot their computers several times a day. (5) A large part of the radiologist job is communicating their findings with physicians, so if you are thinking about automating them away you also need to understand the complex interactions between them and different clinics, which often are unique. (6) Every hospital is a snowflake, data is held under lock and key, so your algorithm might not work in a bunch of hospitals. Worse, the imagenet datasets have such wildly different feature sets they don’t do much for pretraining for you. (7) Have you ever tried to make anything in healthcare? The entire system is optimized to avoid introducing any harm to patients - explaining the ramifications of that would take an entire book, but suffice to say even if you had an algorithm that could automate away radiologists I don’t even know if you could create a viable adoption strategy in the US regulatory environment. (8) The reality is that for every application, the amount of specific and UNKNOWABLE domain knowledge is immense. LONG STORY SHORT: thinkers have a pattern where they are so divorced from implementation details that applications seem trivial, when in reality, the small details are exactly where value accrues. Should you be worried about GPT5 being used to automate vulnerability detection on websites before they’re patched? Maybe. Should you be worried GPT5 is going to interact with SOCIAL systems and destroy our society single-handedly? No absolutely not.

@davisblalock d) The conservative position has not been equally reflected in internet data as the liberal position. The scrapped data, not the content filter, skews toward the status quo of the available data.

c) Even with a balanced pool of labelers, there's a liberal vs conservative divide in labeling practices. This divide could come either from cultural differences (e.g., I've rarely heard conservatives use the word "harmful") or bias in the labelers' instructions. [10/11]

Imagine a world where keyboards only let you type sentences that the keyboard manufacturer agrees with. Or where spellcheck and autocorrect work if you're arguing for one side of a debate, but not the other. That's the world we're building with AI services like ChatGPT. [1/11]

@f4micom @Ianternn Passphrases are out.

@Ianternn that’s quite reassuring but considering loads of people use unsafe password that are present in dictionaries it still can provide for a quite useful resource for potential malicious actors

github.com/ggerganov/kbd-… this tool lets you extract text from an audio recording of keyboard strokes, right now, for free i am not making this shit up, you can potentially steal a password from an audio recording in an office

As I was privately babbling out my own diatribe on the FLI letter yesterday, I was honored to be pinged by @emilymbender and @timnitGebru about joining forces to put out coherent thoughts. Excited to be able to share it today. dair-institute.org/blog/letter-st…

@matthew_d_green This legislature might not make it in, but it was enough for R&D departments to pay attention and invest.

@matthew_d_green That's what AI hype does. Pumps up the media and drives legislators and lobbyists into a frenzy. The industry and the academia supports it, of course, because mandates translate to contracts and grants.

The EU’s “chat control” legislation is the most alarming proposal I’ve ever read. Taken in context, it is essentially a design for the most powerful text and image-based mass surveillance system the free world has ever seen.

Someone mistook the @huggingface logo on my hoodie for Shell's. Too good not to share.