Lydia (in SF)

sounds good! re (1), i was thinking to send an LM across LW/AF collating completed replications like alignmentforum.org/posts/fKuugaxt… lesswrong.com/posts/udDLgexm… lesswrong.com/posts/pCMmLiBc… alignmentforum.org/posts/vBcsAw4r… and then iirc also i think people at AISST / MAIA have done informal replication clubs? so might have code to contribute (also finishing thread here so ppl w knowledge can jump in later)

Really appreciate the suggestions Lydia! These are all things we have planned. 1. No. Well, we have one from Zephy, but we are new and haven’t done many! Or do you perhaps mean collating replications in general? 2. Not written down, we’re working on this. We anticipate to work this out with our fellows once we finalise them. 3. Yes! We will put out a form soon, and your suggesting of a board on our website seems great. Let’s move to dms! I’ve asked Zephy to upgrade his X version if that’s the issue 😁

I’m co-founding Second Look Research with @zephaniahbroe and we are accepting summer fellowship applications for 2026! Fellows will come to the University of Chicago, complete 2-3 replications over 10 weeks (June 15-August 22), and work with external advisors. Fellows will receive a stipend of $10,000 and we will to cover housing and meals. secondlookresearch.com

we need a verb for 'get as much of the upside with as little of the downside as possible'. i will be using 'upsidize'

would be interested! studied RAP years ago in mathphil, since then been running/attending ML/bostrom/carlsmith reading groups, and just returned to epistemology, where reductionism / antireductionism matter again @Miles_M_K has a great post on psychological connectedness mkodama.org/content/psycho…

unironically this is our book club loll let us know if you wanna read about personal identity with a bunch of ai nerds

@smolotnikov @ZanaBucinca @JacobSteinhardt this is great! when can we trial Priori?

I prototyped the early versions of Priori last year and presented at Cosmos & FIRE's AI x Truth-seeking Symposium in December. Since then, multi-agent orchestrators and agentic AI have come far, and some lower-level choices need less auditing than they used to. If anything, that has made the underlying question more urgent: as systems grow more capable, human judgment needs to become easier and more worthwhile to express. Thanks to @cosmos_inst and @TheFIREorg for supporting this work, and to @sebkrier @glukianoff @jon_rauch @AndrewMayne @PhilippKoralus for the thoughtful feedback at the first symposium. I will have more work on this direction coming soon :)

If AI is going to participate in our thinking, we need to build systems that keep humans in charge of the important choices. I built Priori, an interface for human-AI collaboration, toward that goal last year.

@smolotnikov @cosmos_inst @Brendan_McCord @lawhsw @Houda_nait exciting! it's the way!

@cosmos_inst @Brendan_McCord @lawhsw @Houda_nait Waiting until everything feels finished can mean waiting too long. So I'm going to try sharing more of what I'm working on. Follow if you're interested in human-AI collaboration, scalable oversight, and human autonomy :)

It was wonderful to spend time with so many thoughtful people reflecting seriously on the type of world they are building toward at the @cosmos_inst Symposium. None of us are philosopher-builders by default, but it is a worthy aspiration. @Brendan_McCord @lawhsw @Houda_nait @BenGoldhaber @avaa411 @zweinberg @MarroSamuele @allisondman @lishiyori @jliemandt @IvanVendrov @PhilippKoralus @AlexKormokose @chalmermagne @FreeSpeech_AI @luke_drago_ @rossmatican @pwang

@sebngriego @AcerFur discuss online so they can find you

@AcerFur I'm always worried that I'm working on the same big project as someone else

time to take on a big project

@nathancgy4 based based based

This was the first plotted figure during our first attention residual run. I would've drawn it even if the loss was worse than the baseline. This by itself is something insightful and interpretable. Besides a better performance, Attnres gives a better "understanding" about the trained model :)

@cxgonzalez interested if this is sf! would be wonderful to revisit RAP

guy at a party asked if i wanted to join a Reasons and Persons reading group. what even is this city man

@agniv_s @zhengdongwang ++ the content of what i’ve learned/read/solved has made me who i am

The main cases where I think it doesn't apply as much are when you're studying outside of the USA (in the UK, where your degree choices are more fixed), and when your subject requires more longtermism and asks for more breadth (mathematics). However, I believe there are good lessons there anyway.

I'm consistently amazed by just how correct Bluebooking for Happiness from @zhengdongwang is when thinking about American colleges. Should be required reading for basically any motivated college student.

@weeyev sure

@LydNot what if it's more like 30 papers to understand 20 different things

hi! does anyone have...curated lists of significant papers they like? e.g. '30 papers to understand x'. compiling a list of lists. the papers need to be information-dense.

@celestepoasts maybe ceealar.org?

where do I go for next 2 months, planning to do random euro travels

@celestepoasts oxford for a bit of late april / early may

yep! ran a reading group on this for some time. and then rachel instigated ilya list for ai safety. i think we need more ilya lists for diff fields curation by a 'high-taste' individual is important feature of the ilya list. ofc we need better systems for filtering/sorting the vast amount of research increasingly generated perhaps spending a bit of time filtering/sorting historical research may be instructive

@LydNot The Ilya list is the famous one. open.substack.com/pub/theahura/p…

@belindmo @xai exciting! :)

I'm starting a new startup! it's called Long Horizon Research. our first product is Sundial, an AI workspace for humans + agents to self-improve by creating skills together. We are hosting a hackathon tmrw with @AGIHouse @xai, come through

@StefanFSchubert hm, that's weird

There are now more tech workers going from the US to Europe than the other way around

@celestepoasts WHOAAA :D

SF called they want celeste from celeste-land back

@luke_drago_ yes, yes & yes

should we drop some more stuff?

@panickssery does gwern take notes

how to become Gwern "I will spend 10 hours a day reading papers, reading books, reading comments, reading posts, reading everything ... And then maybe I’ll write something in the remaining 30 minutes perhaps on an average day."