Rayco HM

@EDemirNBA No kidding. Knicks, Oklahoma, Phili, GSW.... $24M value on $2M contract. Crazy growth trajectory. Fix shooting from disaster to average would put him into Draymond trajectory. 2 1st round picks would be a steal. 1-1 swap is just robbery.

I hope the New York Knicks will explore a trade for Oso Ighodaro, who's owed $2.2M this year and $2.4M next season. Clearly a defensive plus and a interior playmaking hub. Maybe a Miles McBride 1-on-1 swap where'd the Knicks add a second as they're the clear buyers here.

Creating and Sustaining a Time Crystal: sciencedaily.com/releases/2026/…

LA looked overwhelmed tactically w/PHX throwing the whole book at em. Season high ice rate, 2nd lowest hedge/blitz rate, PHX jumping go-to OOB sets, custom coverages vs top sets, the exact iso/post help LA stinks against, plus a custom Luka coverage pushing him to his weakness.

Vaya por delante que estoy a favor de gravar el capital improductivo como idea, pero no veo cómo puede haber suficientes viviendas vacías en España *allá donde existe la demanda actual que tensiona los precios*. Un hilo con datos 👇

Global Chip Investment 👇 bloomberg.com/news/features/…

Want to serve the LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system 🔥 🗞️ Paper - "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization" 📌 The existing problem - LLMs are seeing growing use for applications such as document analysis and summarization which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in ultra-low precisions, such as sub-4-bit. 📌 KVQuant from this paper addresses this problem by incorporating novel methods for quantizing cached KV activations, including: 👉 (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; 👉 (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; 👉 (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; 👉 (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges; and 👉 (v) Q-Norm, where we normalize quantization centroids in order to mitigate distribution shift, providing additional benefits for 2-bit quantization. 📌 By applying this method to the LLaMA, LLaMA-2, and Mistral models, the paper achieves <0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. ---- On a related note, a recent paper "Activation Beacon" was about extending context length, but "KVQuant" the technique proposed in this paper is about making context more compact in memory. "KVQuant" is not for extending context beyond trained-in limits. It's only about making KV cache more compact by quantizing it. Quantization cannot prevent catastrophic loss of coherence when 8K context model goes beyond 8K. And from page-6 of "KVQuant" paper, for context extension they used Longlora and Lm-infinite. Also memory footprint is not the only concern. Transformers slow down with growing context size because each crank requires looking back at all preceding tokens. Over long ranges it becomes just too slow. Beacons paper tries to address that too.

Wow! Qwen released Qwen2-Math - a 1.5B, 7B & 72B models - beats GPT4o, Claude 3.5 on AIME 24/ AMC 23 🔥 > 84 (72B), 75 (7B), 69.4 (1.5B) on MATH > 72B SoTA in Olympiad Bench, College Math, MMLU STEM > Release both base and instruct models > Apache 2.0 license for 1.5B & 7B, 72B released under Qianwen license > Based on the same Qwen 2 Architecture > Pretrained further on Math specific data (they don't describe what) + synthetic data generated by Qwen 2 > Construct the SFT data with RM + rejection sampling > Perform GRPO after SFT > They decontaminate pre-training plus instruct datasets with exact match and 13-gram dedupe > Integrated with Transformers! 🤗 Kudos to Qwen team on yet another stellar release 🐐

I just finished reading the paper, Mixture of A Million Experts, which is probably one of the most intriguing papers I have read. Mixture of Experts (MoE) is not a new concept. This paper pushes that concept to the extreme by scaling the number of experts in MoE to a million. It sounds a bit impractical, but is it? Here is a quick summary: 1. Introduction - Scaling the number of parameters in a model, the amount of training data, or the computational budget has proven to be an effective yet simple technique for improving model performance. - The feedforward (FFW) layers in standard transformer architectures, responsible for storing "factual knowledge", incur a linear increase in computational costs and activation memory as the hidden layer width grows. - Mixture of Experts (MoE) has become a good alternative for decoupling the computational cost and the parameter count. - Recent works have found that using more experts (high granularity) increases performance. When the number of training tokens is compute-optimal, MoEs consistently outperform dense models in terms of FLOP efficiency. - Another reason to have many experts is lifelong learning. It has been proved that by simply adding new experts and regularizing them properly, MoE models can adapt to continuous data streams. 2. Proposed Method Introduces the Parameter Efficient Expert Retrieval (PEER) layer, which is a MoE architecture using product keys in the router and single-neuron MLPs as experts. A PEER layer consists of three components: a pool of N experts E, a corresponding set of N product keys K, and a query network q. Here is how it works: 1. Given a query x, retrieve a subset of k experts whose corresponding product keys have the highest inner products with the query q(x). 2. Apply an activation function like sigmoid or softmax to obtain the router scores for these retrieved top k experts. 3. Compute the output by linearly combining the expert outputs weighted by the router scores. 3. Product-Key Retrieval - If you have a large number of experts, each with a dimensionality d, then computing the top k indices can be very expensive (O(Nd), N being >= 1e6 in this case). - The authors apply the product key retrieval technique used in the "Large Memory Layers with Product Keys" paper. Instead of using N independent d-dimensional experts as keys, use two independent subsets, each with dimensionality d/2, to form the keys. - Similarly, split the query vector q(x) into two sub-queries and then apply the top-k operation to the inner products between the sub-queries and sub-keys. - This Cartesian product structure of K allows us to find the top k experts efficiently, reducing the complexity from O(Nd) to O((√N + k^2)d) If you are thinking why this makes sense, consider doing napkin math by taking an example. Say, N=10000(which are your keys/experts K), d = 512 (dimensionality of query vector, and each key), and you are searching for the top 10 values from the inner product of the query, and the keys. 4. Parameter Efficient Experts and Multi-Head Retrieval - In a PEER layer, every expert e is a hidden layer with a single neuron as shown below in equation 6. - This is different from the existing MoEs as the hidden layer is often set to the original size of the FFN layer. - To adjust the reduced expressiveness in the PEER layer, the authors use multi-head retrieval, similar to the multi-head attention mechanism in transformers and the multi-head memory in PKMs. - h independent query networks instead of one, each computes its query and retrieves a separate set of k experts. However, different heads share the same pool of experts with the same set of product keys. The outputs of these h heads are simply summed up. If one expert is retrieved (k = 1) per head, using a PEER layer with h heads is the same as using one expert with h hidden neurons. 5. The million-dollar question: Why a large number of small experts? Look at the scaling law for MoE in equation 9 in the above figure where L is the final test loss, a, b, g, γ, α, β are constants, D is the total number of training tokens, and the granularity G is the number of active experts. G = P (active) / P (expert) ; where P = total number of parameters P(active) = number of active parameters per token P (expert) = size of single expert To improve model performance, we need to scale up P, D, and G. On the other hand, we need to limit P (active) because the computational and memory costs are primarily determined by the active parameters during training and inference. Now, N = P / P (expert) = P * G / P (active) This implies that we should decrease the size of each expert, and increase the number of experts N. Also: P(expert) = (2*d_model + 1) * d_expert P(active) = (2*d_model +1) * d_active; where d_model = hidden dimension of the transformer d_expert = number of hidden neurons used in one expert d_active = total number of hidden neurons activated per token For PEER, d_expert is 1, hence the granularity of PEER is always hk where h is the number of heads, and k is the number of experts retrieved per head 6. Results

We just released a new climate emulator to explore the application of Stratospheric Aerosol Injection (SAI) to mitigate global warming! SAI uses reflective particles in the atmosphere to reflect sunlight and thereby cool Earth’s surface. Our emulator lets you explore how different ways to apply SAI might affect average global temperature. Please check out the emulator at planetparasol.ai. SAI is a promising direction, but we still need more research to better understand its impact and potential implementation. Big thanks to collaborators @jeremy_irvin16 @DanVisioni Ben Kravitz @dakotagruener @chrisroadmap and @DWatsonParris

llama.cpp is now in Homebrew Core 🍺

The Intel 386 processor (1985) was a key member of the x86 line, moving to 32 bits. It has a bunch of on-chip registers, implemented with compact, highly-optimized circuitry. Let's look at the circuit, called T8, that implements some of these registers. 1/11

Coding with AI is getting insane.

This is huge: Llama-v2 is open source, with a license that authorizes commercial use! This is going to change the landscape of the LLM market. Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers Pretrained and fine-tuned models are available with 7B, 13B and 70B parameters. Llama-2 website: ai.meta.com/llama/ Llama-2 paper: ai.meta.com/research/publi… A number of personalities from industry and academia have endorsed our open source approach: about.fb.com/news/2023/07/l…

No one ever responded, but I found about as good as it can get. A definitive list of computer architecture papers of the last 50 years. Cc @jonmasters @anfatima21 sites.coecis.cornell.edu/isca50retrospe…

This looks to be an extremely valuable and useful opportunity - I hope compute providers get behind this effort. Orca is the closest thing I've seen to closing the closed language model capability gap.

Last year, @BarackObama, @MichelleObama and I launched the Voyager Scholarship to help college students pursue careers in public service. Now, we are accepting applications for our second class. I hope you'll apply by March 22 voyager-scholarship.obama.org

Geoff Hinton’s “capsules” neural net is tantalizing - another way to do AI with less data goo.gl/eqbFXz

Intel will ship processors with integrated AMD graphics and memory arstechnica.com/gadgets/2017/1… by @drpizza

Impressive to see such fast photo retouching - done by embedding some AI on the phone. goo.gl/mQyZPz

All tax revenue as % of 2014 GDP Denmark 51% France 45% Sweden 43% Germany 36% OECD avg 34% UK 33% Turkey 29% US 26%