Good Science Project

A new way to keep up with the Good Science Project activities: goodscience.substack.com/p/welcome-to-t…

One thing about philanthropy that has bothered me for about a decade is how inefficient, bespoke, and ad hoc it is, especially when it comes to high-net-worth individuals and family offices. Lots of people who have more money than they could ever spend on themselves, and who fully intend to get into philanthropy (such as Giving Pledge signatories), nonetheless don’t want to create a large foundation with a full-time staff of several dozen folks. But what’s the alternative? In some cases, they hire one or two people (if that much!) to write a few big checks here and there. nytimes.com/2024/03/10/us/… Check out this multi-billion foundation with no employees, projects.propublica.org/nonprofits/org… In other cases, they rely on consulting firms like Bridgespan, Rockefeller Philanthropy Advisors, and the like. nytimes.com/2021/11/15/bus… But in both cases, there is a significant mismatch problem on both the demand and supply side (i.e., the demand for high-impact grantmaking opportunities and the supply of good proposals). Indeed, a common phenomenon is for folks to shift philanthropic dollars to donor-advised funds, apparently because they can’t think of what else to do. Why is there such a mismatch problem? * On the demand side, the philanthropic advisors or the family office staff are often limited by a lack of subject-matter expertise in all possible philanthropic areas, and/or the time to investigate and do due diligence. * On the supply side, non-profits and academics who might have wonderful and ambitious ideas have no easy way to figure out which family offices might be interested or how to contact them. Even at large consulting firms, there are usually no public RFPs that might give a clue as to what their clients are interested in, as well as how to apply for funding. The result: a high degree of mismatch between the non-profits/academics who have great ideas and the philanthropists who might want to fund them. It’s way too hard for both sides to find each other. Put another way, there’s often a high degree of mismatch between someone’s philanthropic ambitions and their ability to actually have an impact. That’s why I’m so glad to see the new initiative launched today: Renaissance Philanthropy, involving @tkalil2050, @kumaragarg, @parthion, @ronitkanwar, and more. It goes without saying (although I’ll say it anyway) that with Tom, Kumar, etc., there’s a deep bench of expertise as well as extensive social and professional networks. Renaissance should therefore help solve the huge mismatch problem for potential philanthropists who are interested in science, technology, and innovation. Bravo!

A recent piece: Why Are We Screwing Over Researchers Who Make Innovative Discoveries? The piece delves into the effect of the Bayh-Dole Act of 1980, which basically allowed universities (rather than government) to patent discoveries made with federal funds. But what about researchers themselves?

@ImperialSci @imperialcollege @WCIDLondon Wrong Good Science Project (this is the one in the US)!

🔬 Join the @GoodSciProject's #FridayForum at @imperialcollege! 📅 8 March, 12.20-14.00 📍 Michael Uren Hub, @WCIDLondon In this session, scientists will discuss & debate science at different scales: small ⚛️ to huge 📡, short-term ⏰ to long-term ⏳ imperial.ac.uk/events/172426/…

New piece at the Good Science Project, on a recent academic paper about delays/gaps in NIH funding. We need to have mechanisms for bridging gaps in funding. Otherwise, scientists are left to scramble for resources, and often have to fire people. goodscience.substack.com/p/the-value-of…

There are few things more important to our civilization than understanding how to better do R&D. Thankfully, @eric_is_weird has dedicated himself to studying this question. As a result, he's become the foremost scholar and historian of 19th and 20th century R&D labs. 1/🧵

# Lessons from history’s greatest R&D labs: A historical analysis of what the earliest electrical and great applied R&D labs can teach Answer.AI, and potential pitfalls. Note from Jeremy: This is a guest post from @eric_is_weird, cross-posted from FreakTakes (freaktakes.com/p/lessons-answ…), his blog, to Twitter. I've never posted a long-form article to Twitter directly, so this is an experiment! Eric is the foremost historian of 19th and 20th century R&D labs. I thought I was fairly well informed when it comes to the history of these labs, but after talking to Eric, I quickly realised I'm a rank amateur by comparison! Eric's knowledge of the history of modern research and development is unparalled, and I found his insights into why some organizations were dramatically more effective than others to be utterly compelling. Therefore, I asked him for his totally honest assessment of our plans for Answer.AI, based both on our written plans and a number of in-depth conversations we had together. I wanted to know Eric's absolute unvarnished honest thoughts -- if he thought we were going off on the wrong track, based on his analysis of what's worked and what hasn't in the past, then that would be vital feedback for our young company! As it turns out, Eric discovered that Answer.AI is set up in a way that parallels many of the best practices of the top R&D labs over the last 200 years. This isn't a coincidence -- Eric Ries and I created this company based on our own experience and studies of effective research and development -- but it's very encouraging to see that a top expert like Eric Gilliam feels the same way. # Lessons from history’s greatest R&D labs, by Eric Gilliam Jeremy Howard (former President and Chief Scientist of Kaggle) and Eric Ries (creator of The Lean Startup movement and Long Term Stock Exchange) have teamed up to found a new applied R&D lab: Answer.AI. When speaking with Jeremy, he made it clear that many details of Answer.AI's structure are still being worked out. Only announced a month ago, the org is still in its early development stages. But the founders have conviction on certain principles. The most prominent of them is one extremely relevant to my regular readers: The founders seem to be particularly inspired by Edison’s Menlo Park Lab and the early days of commercial electric research. In the piece, I’ll briefly examine the (working) plans for the lab and do some historical analysis, detailing: 1. What the earliest electrical R&D labs can teach Answer.AI 2. Useful rules-of-thumb from other historically great applied R&D labs 3. Potential pitfalls to keep in mind as they move forward ## Answer.AI in a Nutshell Jeremy’s blog post announcing Answer.AI makes it clear that the org is, to a large degree, inspired by the field of electricity’s path of progress in the 1800s. He believes the current state of the AI field is similar to the state of the electricity field between the work of Michael Faraday and Edison’s lighting projects. This was an era in which new electrical findings were being pieced together, but few had made any progress in turning the potential of electricity into great applications. I don’t find this comparison crazy. So far, I don’t believe AI has come close to the level of breakthrough that electricity proved to be. Electricity brought the sunlight indoors for a negligible cost and powers so many of our modern conveniences— refrigeration, TVs, central heating, etc. That’s a high bar. However, given that human ingenuity created the breakthrough that was electricity and each of those applications, it is surely worth considering that AI could grow to be the most impactful field of them all. Whether AI does reach that level of promise, to me, is a question of human ingenuity. So, I have no issue with Jeremy comparing the AI field to the electrical field c. 1830 to 1910. With that elephant out of the way, let’s briefly examine what sets Answer.AI apart from AI labs like OpenAI and Anthropic. From a funding perspective, Answer.AI seems much, much cheaper. The founders have initially raised USD10 million. This stands in stark contrast to the gargantuan initial rounds of OpenAI and Anthropic. Also, Answer.AI's research agenda is more application-centric. The following excerpt from Jeremy’s blog post highlights what he thinks differentiates the lab’s approach: > At Answer.AI we are not working on building AGI. Instead, our interest is in effectively using the models that already exist. Figuring out what practically useful applications can be built on top of the foundation models that already exist is a huge undertaking, and I believe it is receiving insufficient attention. > > My view is that the right way to build Answer.AI’s R&D capabilities is by bringing together a very small number of curious, enthusiastic, technically brilliant generalists. Having huge teams of specialists creates an enormous amount of organizational friction and complexity. But with the help of modern AI tools I’ve seen that it’s possible for a single generalist with a strong understanding of the foundations to create effective solutions to challenging problems, using unfamiliar languages, tools, and libraries (indeed I’ve done this myself many times!) I think people will be very surprised to discover what a small team of nimble, creative, open-minded people can accomplish. > > At Answer.AI we will be doing genuinely original research into questions such as how to best fine-tune smaller models to make them as practical as possible, and how to reduce the constraints that currently hold back people from using AI more widely. We’re interested in solving things that may be too small for the big labs to care about — but our view is that it’s the collection of these small things matter a great deal in practice. It would be unfair to say that an application-centric research agenda is necessarily less ambitious than AGI. Those biased toward basic research might say so, but I don’t think that opinion is very historically-informed. Edison himself was application-centric above all else. His deep belief in market signals is fascinating when juxtaposed with the market indifference of many great academic physicists. In the book From Know-How to Nowhere (amzn.to/3HoUUbW), a history of American learning-by-doing, Elting Morison described the interesting nature of Edison’s motivations: > If the means by which he [Edison] brought off his extraordinary efforts are not wholly clear, neither is the cause for his obsessive labors. No diver into nature's deepest mysteries carrying next to nothing for the advancement of knowledge and even less for the world's goods, he would become absorbed in making something work well enough to make money. The test in the marketplace was for him, apparently, the moment of truth for his experiments. Edison built his god-like reputation by dreaming in specific applications. He kept market, resource, and manufacturing constraints in mind from the earliest stages of his projects. Edison dreamed practical, realizable dreams. And when the limitations of component technologies stood in the way of his dreams, he often had the talent to invent new components or improve existing materials. Edison’s biggest dream, the light bulb, mandated that  Edison solve a much broader set of problems. The following excerpts from my Works in Progress piece on Edison (worksinprogress.co/issue/thomas-e…) paint a clear picture of his ambitious but practical dreams: > After Edison’s bulb patent was approved in January 1880, he immediately filed another for a ‘System of Electrical Distribution’. Filing for these so close together was no coincidence. To Edison, it was never just a bulb project. It was a technical business venture on a possibly unprecedented scale. Edison wanted to light up homes all over the world, starting with lower Manhattan. > > Bringing the project from dream to mass-market reality would require solving over a hundred technical problems. His was a new bulb that needed to be powered by a generator that did not yet exist at the start of the project, strung up in houses that had no electricity, connected via underground street wiring that was only hypothetical, and hooked up to a power station that had never existed before. > > Yet, at the end of two years’ time, Edison would do it. And, just as importantly, the entire venture was profitable by the end of the project’s sixth year. Edison was clearly doing a different kind of dreaming than those who do basic research. His lighting work embodies what extreme ambition looks like in application-centric research. Answer.AI making this kind of ambitious, applied work their North Star is an extremely interesting goal. This goal has the potential to give Answer.AI a comparative advantage in the growing space of for-profit AI labs. For example, the most ambitious aspects of OpenAI are considered to be in its research, not its work on applications. Answer.AI’s particular setup can also set it apart from AI startups and academic labs. New AI startups do some research on how to commercialize new AI models in new ways, but they generally have short runways. In this kind of environment, only specific types of research projects can be pursued. Academic labs — for many reasons covered elsewhere on this Substack (such as in the ARPA series (freaktakes.com/s/arpa-playbook)) — don't have the right combination of incentives, experience, and staffing to build new technologies in most problem areas. The main incentive of the profession, in a simplified form, is producing many paper studies that get cited many times. Answer.AI has the chance to let its alternative focus lead it to areas under-explored by academics, companies with brief timelines to hit revenue benchmarks, and more AGI-focused R&D labs. Legally, Answer.AI is a company. But in practice, it might hover somewhere between a lab and a normal “profit-maximizing firm” — as was the case with Edison's lab. The founders seem perfectly content to pursue high-risk projects that might lead to failures or lack of revenue for quite a while. In saying this, I do not mean to imply they are content to light money on fire doing research with no chance of a return. Rather, they hope to fund a body of research projects that ideally have positive ROI in the long term. They are just not overly concerned with short-term revenue creation. (Making the pursuit of research agendas like this easier is actually one of the founding goals of Ries’ Long Term Stock Exchange — which I address later.) There is apparently no pressure to produce a product that can hit software VC-style revenue goals within 12-24 months, or anything similar. This is good. Seeking to satisfy these types of metrics does not traditionally permit a company to act like a truly ambitious R&D lab. I’m not saying it can’t happen — DeepMind seems to have made it work in its early years — but it does require pushing against investor pressure quite strongly. The VC money raised for Answer.AI has left the founders with enough voting shares that investors can’t veto founders’ decisions. Additionally, Howard says the company’s investors understand what they are trying to build is, first and foremost, a lab. This is a great step towards building an organization focused on building very useful, very new things rather than the most profitable thing possible — which often comes with bounded technical novelty. Interestingly, Answer.AI will also keep a small headcount. Jeremy built Fastmail up to one million accounts with only three full-time employees. He hopes to keep the Answer.AI team exceptionally talented and “ruthlessly small” in a similar way; he believes keeping teams small is important to building new, technically complex things. Now that I've outlined some important pieces of Answer.AI’s vision, I'll dive into the historical analysis. In the first section, I detail lessons that Answer.AI can draw from both Edison’s Menlo Park laboratory and the Early GE Research Laboratory. In the following section, I'll share useful lessons from other historically great industrial R&D labs. Lastly, I’ll highlight the bureaucratic details that explain why the operational models of the great industrial R&D labs have not been replicated often. ## Learning from the First Electrical R&D Labs I find it exciting that Edison’s Menlo Park lab is a North Star for Answer.AI. I covered Edison’s work in several pieces because I think evergreen lessons can be drawn from his work. But I think a more complete way to incorporate lessons from the 1870-1920 electrical space is to draw on the work of both Edison’s Menlo Park Lab and the young GE Research Lab. The latter operated as a more traditional industrial R&D lab. GE Research’s history holds many lessons to help steer Answer.AI’s problem selection and work on its standard projects. However, exceptionally ambitious projects may draw more heavily on the lessons of Edison’s lab. (As a note, while Edison General Electric was one of the two companies that merged to become GE — along with Thomson-Houston Electric — Edison had essentially nothing to do with the formation of the iconic GE Research Laboratory.) Different types of projects characterized the work of the two electrical labs. When it came to electrical work, for years, Edison’s lab and mental efforts were focused on doing everything necessary to bring a single, revolutionary product to market. On the other hand, GE Research usually had many separate courses of research underway at once. These projects all sought to improve the science and production of existing lighting systems, but they were otherwise often unrelated to each other. Additionally, GE’s work could be categorized as more traditional “applied research.” The lab was not actively looking to create a field of technology from scratch as Edison did. GE Research's projects were often novel and ambitious, but in a different way than Edison's. Later, I will explore the types of novelty the GE Research Lab pursued. First, I’ll give the reader a more fine-grained idea of how Edison’s lighting project actually operated. ### Lessons from Edison’s Work on Electricity Edison’s lighting work provides great management lessons for those looking to direct a large chunk of a lab’s efforts toward a single, big idea. Edison’s major contribution to the field of electricity was not inventing each of the components in his lighting system, but in turning a mass of disparate gadgets, scientific principles, and academic misconceptions into a world-changing system. The burden of doing “night science” — as Francois Jacob refers to it (genomebiology.biomedcentral.com/articles/10.11…) — largely fell on Edison. In the late 1870s, nobody knew much about electricity yet. The existing academic literature had more holes than answers, and many of its so-called “answers” turned out to be wrong or misleading. From this shaky starting point, Edison proceeded. He combined his unique mix of attributes and experience to deliver a world-changing system. These included: knowledge of several adjacent scientific fields, deep knowledge in then-overlooked experimental areas, market knowledge, manufacturing knowledge, and the ability to adequately operate a small research team. In large part, Edison created his lab as a way to scale himself. As a result, to understand how his lab operated, one needs to know how Edison himself carried out his explorations. Edison was one of the more stubborn experimentalists of all time. He spent most of his waking hours carrying out one experiment or another. While he did pore over scientific literature, for him, nothing was settled until he proved it for himself at the lab bench. I write in my Works in Progress piece: > Edison respected scientific theory, but he respected experience far more. In Edison’s era of academia as well as today’s, many professors had a certain preference for theory or ‘the literature’ over hands-on improvement. Because of this Edison did not care much for professors. He was even known to go on long diatribes, during which he had assistants open up textbooks, locate scientific statements that he knew to be untrue from experience, and quickly rig up lab demonstrations to disprove them. ‘Professor This or That will controvert [dispute with reasoning] you out of the books, and prove out of the books that it can’t be so, though you have it right in the hollow of your hand and could break his spectacles with it.’ Contained in his head was a database of countless experiments and results that made it seem as if his “intuition” was far beyond his contemporaries. This left him with an unparalleled skillset and body of knowledge. If anyone could feel comfortable pursuing a project that others had previously failed at, it was Edison. Edison’s confidence in his skills was never more on display than when he chose to pursue his lighting work. Many in the scientific establishment knew electric bulb lighting was technically possible, but claimed they had proven that it could never be economical. Edison disagreed. On top of Edison’s admirable approach to experimentation, he brought a high level of practicality to his process. He knew his inventions needed to make commercial sense in order to make it out of the lab. So, even in early courses of experimentation, he kept factors like manufacturability in mind. He wouldn’t commit much time to something that didn’t make commercial sense. With that being said, Edison wanted to change the world with his technologies more than he wanted to get rich. So, the practical factors he paid aggressive attention to were primarily treated as constraints. He did not optimize for profitability, but he knew his ideas needed to be profitable. Nobody who wanted to optimize for profit would have pursued lighting in the way Edison did. The technical risks were too great. Edison was able to imagine an ambitious system that required many technical advances. It was so futuristic that maybe only he was capable of coming up with it. But just as impressively, he was able to do it profitably and on schedule. His dogged commitment to experimentation seems to be largely responsible for this. Edison and “the boys” constantly experimented on every piece of the process to improve and learn more about all the sub-systems in Edison’s grand system. They wanted to know how every piece of every sub-system performed in all conditions. I’ll share just two excerpts from my Works in Progress piece as examples. The first is from Edmund Morris' biography of Edison. It recounts how thoroughly Edison and his trusted aid, William Batchelor, were in carrying out round after round of filament experiments: > For week after week the two men cut, planed, and carbonized filaments from every fibrous substance they could get — hickory, holly, maple, and rosewood splints; sassafras pith; monkey bast; ginger root; pomegranate peel; fragrant strips of eucalyptus and cinnamon bark; milkweed; palm fronds; spruce; tarred cotton; baywood; cedar; flax; coconut coir; jute boiled in maple syrup; manila hemp twined and papered and soaked in olive oil. Edison rejected more than six thousand specimens of varying integrity, as they all warped or split… > > In the dog days, as heat beat down on straw hats and rattan parasols, the idea of bamboo suggested itself to him. Nothing in nature grew straighter and stronger than this pipelike grass, so easy to slice from the culm and to bend, with its silicous epidermis taking the strain of internal compression. It had the additional virtue, ideal for his purpose, of being highly resistant to the voltaic force. When he carbonized a few loops sliced off the outside edge of a fan, they registered 188 ohms cold, and one glowed as bright as 44 candles in vacuo. This approach went far beyond bulb filaments. The following excerpt describes the work of one of Edison’s lead mechanics in turning the Menlo Park yard into a 1/3 scale model of what they would later install in Lower Manhattan. I write: > [Kruesi, Edison’s mechanic] along with a group of engineers and a team of six diggers, turned the excess land of the lab in Menlo Park, New Jersey…into a one-third-scale model of Edison’s first lighting district in lower Manhattan. This team tested and re-tested the electricity delivery system, digging up Menlo Park’s red clay to lay and re-lay an experimental conduit system. The team carried out countless tests to ensure that they found materials to efficiently carry the electric current while also keeping the delicate materials safe from water and ever-present New York City rats. The entire process was marked by the classic trial-and-error of the Edisonian process. The first subterranean conducting lines and electrical boxes the group laid were completely ruined by two weeks of rain — despite being coated with coal tar and protected with extra wood. While the diggers dug up the failed attempt so the damage could be examined, Kruesi and a young researcher…studied and tirelessly tested unbelievable numbers of chemical combinations — making full use of the laboratory library and chemical room — until, finally, a blend of ‘refined Trinidad asphaltum boiled in oxidized linseed oil with paraffin and a little beeswax’ was found that protected the electrical current from rain and rats. > Edison built his own style of dogged experimentation into the culture of his lab. Since the lab was meant to scale Edison, this makes perfect sense; he was a man with far more ideas than hands. So, he hired more hands. Edison did not search far and wide to hire the world’s best research minds, and many of those he employed did not even have scientific backgrounds. This didn’t matter much to Edison because most of them were employed to undertake courses of research that he had directed them to pursue. A couple of his Menlo Park employees had advanced scientific degrees, but far more did not. For the most part, the lab and its activities were steered by Edison and his ideas. As a result, the productivity of his lab followed wherever his attention went. After some time working on a project area, Edison would often grow antsy and wish to move on to the next thing — he craved novelty. The lab’s resources and extra hands would move with him. As we’ll see in the next section, this stands in stark contrast to how the GE Research Lab recruited and chose problems. Menlo Park's electrical activities provide a great management playbook for what it looks like to direct a lab’s efforts toward a single, major system. If Answer.AI does not want to go all-in on one thing, it can still find a way to apply this playbook to a certain focused team of employees while leaving the others to tinker around with exploration-stage ideas. In Edison’s less-focused experimentation periods, his lab served as more of an “invention factory,” doing this sort of fiddling. Additionally, Edison's preference for application and commitment to experimentation over theory in a young area of science can surely provide Answer.AI some inspiration. Of course, Edison did some things better than others. Edison’s most easily-spottable “deficiency” is that his lab was largely dependent on him. Without him and his big ideas, the lab would have probably ground to a halt. While Edison’s technical vision, practicality, and experimental approach are absolutely worthy of emulation, the lessons of GE Research should probably be added into the mix as well. GE operated as more of a prototypical industrial R&D lab with an approach quite suited to the fact that the science of electricity was beginning to mature in the early 1900s.

New essay: "Texas Gas Stations, NIH-Sponsored Post-Docs, and the Foolishness of Artificially Subsidizing Demand." goodscience.substack.com/p/texas-gas-st…

New column on why NIH-sponsored post-docs would be better off working for a Texas gas station: goodscience.substack.com/p/texas-gas-st…

New podcast interview with Monya Baker at Issues in Science and Technology: issues.org/episode-40-bet… podcasts.google.com/feed/aHR0cHM6L… open.spotify.com/show/2NRvvlDtB…

@ImperialMed @ImperialInfect Sorry, that was a different "Good Science Project," not the US version!

Why is 'doubt' crucial in science? How can we support the kind of science that takes intellectual risks – and takes time? Dr Nana-Marie Lemm from @ImperialInfect reflects on the recent Day of Doubt conference organised by the @GoodSciProject. 👇 #more-4291" target="_blank" rel="nofollow noopener">blogs.imperial.ac.uk/imperial-medic…

Nature picked up on some quotes from my newsletter about the NIH nomination hearing: nature.com/articles/d4158…

🚨 New Science Journalism Fellowship! The Johns Hopkins Science Writing program and @GoodSciProject are awarding $5K reporting grants for articles that reveal flaws in science policy, practice, or funding. Grateful for RTs and help spreading the word. advanced.jhu.edu/good-science/

For science journalists: The Good Science Project is joining with Johns Hopkins to sponsor a set of awards for reporting on science policy issues (e.g., funding high-risk, high-reward research). advanced.jhu.edu/good-science/

@Topaz20211 @Jikkyleaks @WildColonialGal @VPrasadMDMPH The other time I came across him was a thread from last year about how vegan burgers are made from the blood of aborted fetuses: twitter.com/Jikkyleaks/sta… Insane.

@Jikkyleaks I believe @VPrasadMDMPH may not be accessible to you? 🤔 vinayprasadmdmph.substack.com/p/do-not-repor…

@Topaz20211 @Jikkyleaks @WildColonialGal @VPrasadMDMPH Who says "Jikky" is a whistleblower? From what I've seen, he is either a lunatic living under a bridge and sending tweets while begging for dollars on a street corner, or else is a pharma stooge who is paid to make anti-vax people look like lunatics living under a bridge, etc.

@Topaz20211 @Jikkyleaks @WildColonialGal @VPrasadMDMPH All I know is that he flat out lied about where I work, baselessly accused psychologists of hiring a private investigator, and said things about Gino and Data Colada that sound like a schizophrenic ranting on a street corner.

@Jikkyleaks @WildColonialGal @VPrasadMDMPH Vinay is legit. No reason for him to spend time on anonymous trolls.

@WildColonialGal @VPrasadMDMPH No he blocked me for some reason I think it was because I challenged him on a claim which presumably he now agrees was false

It's Saturday night, and I'll probably post about this again, but I just wrote up well over 20k words on my personal history as a metascience venture capitalist (so to speak).

Good article featuring @patrickc, @tylercowen, @BrianNosek, @SchmidtFutures, and more. philanthropy.com/article/quick-…