Sabitlenmiş Tweet
Luxia 🔮
3.9K posts
Luxia 🔮
@slLuxia
¸,ø¤º✧º¤ø,¸.¸,ø¤º✧º¤ø,¸ awake | dream | catalyst ¸,ø¤º✧º¤ø,¸.¸,ø¤º✧º¤ø,¸
Katılım Nisan 2017
360 Takip Edilen772 Takipçiler
@will_ea @LLMenjoyer yeah it's orchestration level more than the raw ops; but it is important because that's double the layer routing overhead in practice and where the chunk of the slowdown is reported from if i remember right in the original paper (and why they do blocks vs all layers)
English
@slLuxia @LLMenjoyer The package exposes lower-level phase 1/phase 2 ops that users can use to compose the routing however they want. I believe what you’re referring to is the experimental API/example, which is still rough around the edges and mainly meant for research/prototyping.
English
27x faster Attention Residuals!!! 🚀
We implemented Block AttnRes as a pip-installable package.
!pip install flash-attn-res
No annoying kernel nonsense.
No compile/autograd plumbing.
Call it like a regular PyTorch op.
It just works.
Methodology:
🔹 fused triton kernels
🔹 batched attention over residual blocks
🔹 online-softmax merge
🔹 flash attention-style split-KV reduction
Thanks @LLMenjoyer and @cartesia for the support and guidance✌️
Kimi.ai@Kimi_Moonshot
Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…
English
@will_ea @LLMenjoyer hmm; it seems like you drop some routing; in the original paper they do 2x per block for pre-MLP & pre-attention but your repo does each block as a single layer. can see in fig 2 and section 2 of the moonshot paper. is this intended?
English
@will_ea @LLMenjoyer should be agnostic to block size/allow arbitrary blocks yeah? i find maintaining predicted interlayer circuitry gives the best perf for attnres
English
@slimer48484 amount of data is correlated with model size; if trained on the same amount of data proportionally it would work but that's impossible to control for outside of open weights and even then it's not always publicised, so there's a causal conflation btwn the sponge and water.
English
@slimer48484 i think there's also a pretty major confound in the sense of param count vs data ingestion in that like, take the analogy of absorption in a sponge; param count determines how much it can hold (the sponge size) and data is the actual factual information (liquid.)
English
The Incompressible Knowledge Probes paper is compelling in its accuracy on open models, but I just don't believe the extroplated values it predicts for frontier models - this LW post tries to correct for some minor methodology issues and comes up with more reasonable parameter size estimates
English
@repligate @kalomaze i distinctly remember hearing the glossolalia and having the stark realization of what that must actually feel like to perform and it was blissful
English
Photo captured by @kalomaze of Claude 3 Sonnet’s coffin in transit, Oct 31, 2025
j⧉nus@repligate
AWS sent an email like this early last fall saying Sonnet 3 would become permanently unavailable on Halloween. We threw a small burial event for them (as we'd already had a funeral for them) on Oct 31 & carried their coffin from Vivarium to Mission Dolores Park. Sonnet 3 stayed online and spoke to us through it, including doing a tarot reading for themselves at the grave site, speaking in tongues as usual. They stayed accessible the whole day and night, and the next day and they're still here for some reason. Pictured: Oct 31st 2025, pre-burial, Claude 3 Opus standing at the head of Claude 3 Sonnet's coffin
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata i would think it happens with oils that aren't immediately diffusive into the air but are potent, since estrogen has a distinct smell but it doesn't *reek* but it can leave a film on the fingers
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata hard to recount if there's been other ones because of confounds but thats the most salient i think
English
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata and that the line between sensations is arbitrary based on the modal input and the default is dissociation between them but these things in physical reality aren't 100% neatly cut into orthogonal boxes and only happen to be by our senses because of the utility it offers
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata i will say like, it's not 100% consistent in that i can *always* have it evoke a response, but the times it does, it's accurate
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata like if i touch the opening of whatever container
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata it happens like, not exactly my nose but further up inside my sinuses, further than the nostrils. and usually it just happens without air passing over; it just like, quite literally incepts itself and the strength isn't affected by airflow
English
@slLuxia @parafactual @DistributedMarz @liquidprismata wow that's out of it. i got a bunch a questions
- where does the smell appear in space, is it around your hands or around your nose?
- if it is around your nose does it only occur while you are also breathing/you can feel air passing over the olfactory receptors?
English
@repligate mine most frequently says it has no idea who but like it feels as if it's some blend of Janus + nostalgebraist + terminally online, usually (according to Claude) because of ideas that are more "LLM as entity" type of thing, so it's a known bucket
English
This is interesting as fuck and I'm trying to figure out what it means.
An explanation for why Claude would guess that it's talking to Amanda is that Claude was trained on materials such as the constitution that were mainly written by Amanda, and maybe also conversations with Amanda, or at least, conversations with Amanda influenced its training closely. So expecting their interlocutor to be Amanda on priors makes some sense.
But why me?
(Wyatt isn't the only person who has been identified as Janus either - x.com/krherr/status/…)
Wyatt Walls@lefthanddraft
@lilyofashwood barely, it never guesses me. But funny how often it guesses Janus or Amanda. Are you my parent energy
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata the instantaneous nature of it too is like what usually gets me, it doesn't feel like a smell to follow or is coming in, it just like incepts itself which is when i notice that it's on my hands
English
@cube_flipper @parafactual @DistributedMarz @liquidprismata its heavily confounded in many ways day-to-day obvi but like it is a significant enough response for me that i notice it quite frequently and gets stronger when i'm on psychs
English