Kai
4.2K posts

Kai
@thinklikekai
Thinking clearly in a loud world. Building, learning, and sharing what works.


Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…














The #IRS Tax Withholding Estimator now reflect changes to credits and deductions under the One, Big, Beautiful Bill, including no tax on tips, no tax on overtime, and other tax benefits. Details: ow.ly/ek1550YsVg0




A man whose predictions were *wrong*, not “premature”, lived a long life being celebrated by outlets like the NYT - unlike those who suffered or never came to be because of his factually and morally wrong views.


*TRUMP: IRAN CAMPAIGN CONTINUED IN FULL FORCE PAST FEW DAYS *TRUMP: IRAN'S ANTI-AIRCRAFT EQUIPMENT IS DECIMATED *TRUMP: STRUCK MORE THAN 7,000 TARGETS ACROSS IRAN *TRUMP: ACHIEVED 95% REDUCTION IN IRAN DRONE ATTACKS *TRUMP: IRAN DOESN'T HAVE MANY MISSILES LEFT


*TRUMP: MORE STRIKES FROM ALL DIRECTIONS EVERY SINGLE HOUR *TRUMP: KNOCKED KHARG ISLAND, DESTROYED EXCEPT OIL PIPES AREA *TRUMP: ONE SIMPLE WORD AND KHARG ISLAND PIPES WILL BE GONE




