Lisa Dolan

@josephsemrai @contextsuite @josephsemrai congratulations! Try it out !! This is time for creativity and problem solving, not rote office work @contextsuite

Meet Context, the first AI office suite. Humanity spends 2.5 trillion hours a year on office work. Context can one-shot most of it. Welcome to the era of vibe-working. Sign up today or tag @contextsuite with a prompt.

So... “vibe coding” and this isn’t a Xeet about any particular person, whether a one-person company will be worth a billion dollars, or any specific toolset. I’m a child of the 80s and grew up coding, and I’m not per se anti-vibe-coding. The real question is: What's analogous? If you think vibe coding (which is people with little coding/technical knowledge being able to produce serviceable code that will usually run) is a “breakthrough,” and I don’t, then my question is my usual question: What must be true, or what must I believe, for that to be credible? I can easily believe the idea that vibe coding is like gaming. In a videogame, I can enjoy it even if I’m not among the best at it, and I can pretend I have powers (flying, kung-fu, coding) that I don’t have in the real world or have at a far lower level (running, jumping, driving) in the real world. My friend, who runs the AI stuff at a global investment firm, argues that it’s more like giving people cars; they can drive around and do stuff more efficiently even if they don’t know how the cars work and even if they aren’t very good drivers. This I can also believe, though the analogy is imperfect. But some of the narratives from AI-centric VCs I find unbelievable. My nit to pick is with the “but imagine when the customer is so close to the creative locus” or something like this… I’ve seen this in multiple “vibe coding” decks at this point and it doesn’t match with any of my priors. The premise here, which is anecdotally seductive but analytically inadequate, is that it would be great if people who were closer to the desired result designed the thing. This is what I somewhat crudely call the “don’t trust the skinny chef” logic that people generally overapply. I know of no evidence that people who have very keen vision are the best at engineering 8K televisions (though they might best appreciate them). Most minivans are not designed by moms (I’ll note Mimi Vandermolen, a woman and mom, did great work on the Taurus at Ford, but didn’t design the Aerostar or Windstar). The concept that there are many holes in the market that are going to be filled by non-coders who are suddenly going to develop important things that return venture-capital-scale returns seems not only first-glance unbelievable but incredibly unlikely, given these people will compete with real engineers. The right analogy, I think, is that if you look at areas of Europe that had the largest gains in literacy in the nineteenth century, like Prussian proto-Germany and industrializing England, we do not see a boom in memorable, durable literature and poetry from these places. Who was the Shakespeare of Manchester in 1840? Nobody cares. I’m all for giving humans new tools. In fact, most of progress of our species is attributable to wise allocations of capital and labor in tandem. But I don’t think “vibe coding” tools are particularly special and I think the current wave of VC excitement both misunderstands and overpromises.

It seems increasingly likely that the real reason we never got Opus 3.5 was surprisingly simple; serving it was just too expensive for Anthropic to justify.

Congratulations One of the most remarkable teams weve seen - ever. AI will not take out labor, it may be quite the opposite - more expensive, valuable labor. Mercor will lead this change, and be a part of optimizing output by MATCHING the right talent-at the right time/place-to outcomes. @mercor_ai @BrendanFoody , @adarsh_exe @suryamidha @Link_Ventures

Mercor is solving talent allocation in the AI economy. The difference between greatness and failure is the right person being in the right place at the right time. Putting them there is the hardest unsolved problem in capitalism. We’re excited to announce our $100M Series B at a $2B valuation from @felicis, @generalcatalyst, @benchmark, DST and @MenloVentures.

> oh god, not another wrapped > wait wtf - how does it know that? introducing your 2024 x(oul) wrapped we reconstruct a parallel version of you from your timeline & reveal what you've become on this hell site this year. see your xoul 👁️: xoul.ai/wrapped

Power consumption might be the biggest bottleneck for AI for the rest of this decade. “But wires are getting so crowded that some prospective data center customers—which request far more power than other users—are being told they may have to wait until the next decade to get the power they are seeking. Others are receiving less power than they expected.”

I used the chatbot arena data from @lmsysorg to create a visualization of LLM’s Elo rating changes. You can see: 1. The gap between various companies/open source projects is narrowing. 2. The major players are gradually becoming the various big tech companies.

LLaMA-3 is a prime example of why training a good LLM is almost entirely about data quality… TL;DR. Meta released LLaMA-3-8B/70B today and 95% of the technical info we have so far is related to data quality: - 15T tokens of pretraining data - More code during pretraining (leads to better reasoning capabilities) - More efficient tokenizer with larger vocabulary - Super sophisticated (including LLM components) data quality filtering - Extensive empirical analysis of data mixture - Focus on quality filtering of post training data (for SFT/RLHF/DPO) All of the cool stuff in this report is related to how to curate data effectively for pre/post-training! This really shows that data curation/filtering is the most difficult and impactful aspect of training foundation models. (1) Model architecture: Only 5 sentences are provided about the model architecture, which simply state that LLaMa-3 uses a standard decoder-only architecture with grouped query attention to improve inference efficiency (and a longer 8K context). It’s pretty clear that model architectures are becoming standardized, and most of the research focus is going into constructing datasets. In fact, the main architecture modification made by LLaMA-3 is a more efficient tokenizer! “Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance.” - from LLaMA-3 blog (2) Better tokenizer: LLaMA-3 comes with a custom tokenizer with a vocabulary of 128K tokens (LLaMA-2 had a vocabulary of 32K tokens). This tokenizer is more token efficient (i.e., fewer tokens are necessary to encode the same piece of text relative to LLaMA-2), which makes inference more efficient. Authors also note that the new tokenizer improves performance! In other words, making sure that we are encoding the model’s input data correctly is super important. (3) Massive pretraining corpus: LLaMa-3 is pretrained over 15T tokens of text (5% non-English), which is a 7X improvement over LLaMA-2 and even larger than the 12T pretraining corpus of DBRX. The pretraining corpus also has 4X more code relative to LLaMA-2 (this was a big criticism of LLaMA-2). With this in mind, it’s not a surprise that LLaMA-3 has strong reasoning/code capabilities—several papers have correlated pretraining on code to better downstream reasoning in LLMs. “We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3.” - from LLaMA-3 blog (4) FIltering pretraining data: Few concrete details are provided on the filtering process for the pretraining corpus of LLaMA-3, but it’s clear that a lot of filtering is done. These filters include heuristic filters, NSFW filters, semantic deduplication, and text classifiers to predict data quality. Plus, authors note that LLaMA-2 is very good at detecting text quality, so they use these models in the filtering process (see above). Authors also mention that they do extensive empirical analysis to figure out the correct data mixture (DBRX also mentions this is hugely important). (5) Overtraining: Chinchilla proposed the compute optimal training regime for LLMs, but recent work indicates that pretty much everyone overtrains their LLMs relative to the compute-optimal ratio. LLaMA-3 is pretrained on two orders of magnitude more data (for the 8B model) beyond the compute-optimal ratio, and we still see log-linear improvements. Sure, we could train a larger model on fewer tokens and achieve similar performance while spending less on training compute. But, this doesn’t consider inference costs! We almost always will pay for more training compute if it means we can deploy a smaller model with the same performance. “The quality of the prompts that are used in SFT and the preference rankings that are used in PPO and DPO has an outsized influence on the performance of aligned models.” - from LLaMA-3 blog (6) Post training data quality: Even beyond pretraining, data quality is pivotal for LLaMA-3! The model is aligned with a combination of SFT, rejection sampling, PPO, and DPO. During alignment, authors claim that the quality of supervised/preference data is super important. In fact, the biggest quality improvements in LLaMA-3 came from curating this data and performing multiple rounds of quality assurance on humans annotations!

"So much advantage in life comes from being willing to look like a failure in the short term." — @shaneparrish (Found in his Sunday Newsletter)

Today, we're announcing $3.6M in funding and the launch of our platform, which uses AI to assess and match talent with companies. Our round is led by @generalcatalyst and includes participation from @ScottDSandell (CEO of NEA), @Soma_Capital, @Link_Ventures, and others.

I'm attending "BetterHealth" NOVEL DRUGS CURING DISEASE: Transformative Drug Discovery . Would you like to attend? lnkd.in/eGiTJq-q

Where will there be sustainable value creation in the early innings of a platform shift? After family conversations over July 4th, it is clear how top of mind this question is for everyone. Not only folks in our industry The foundational models which m…lnkd.in/ereMCC2m

As we head into the next Imagination in Action event, here is a recap from the last event Link Ventures supported It’s an awesome interview of Sam Altman by our fearless leader and founder, David Blundin! #generativeai #thought…lnkd.in/erqkuH_n lnkd.in/eugfzBRN

Excited for next week’s event by Imagination in Action! What a lineup starting at 8:30am… To apply to attend click here: lnkd.in/e7wSD-3e Link Ventures #thoughtleaders #artificialintelligence #startupadvice lnkd.in/eq7ygqDM

What awesome takeaways from @snowmaker I love Jared’s thoughts… my key takeaways from him: -“A high quality team is 2-4 smart people who know each other well (and despite that, can work with one another), such bonds and perfect timing opportunities are…lnkd.in/eCMBzngz

AI Takeaways from the week prior: As early-stage private #investors, we are seeing the trend toward smaller, privacy compliant models vs. larger models, and the power of creators and specialized crowd sourced data The large #tech companies will naturall…lnkd.in/ey2A8jXe

It was great to see the reporting on Sam Altman’s recent Israel trip   A few thoughts after the recaps:   1.)  It is really important for global cooperation on AI and the rules of the game. I would envision large enterprises leadin…lnkd.in/eWZVGKQq lnkd.in/eiC4YEBk

Seeking long-term value in the LLM ecosystem? Check out my latest piece in Medium to find the answers By collaborating with, and supporting outstanding teams, we can unlock incredible opportunities If you’re a founder working…lnkd.in/eFx43abe lnkd.in/eUsJKBY8