Tweet fijado
I made AgentIR: a scheduler for distributed LLM serving that makes agent workloads run much faster.
41.3% lower E2E latency, and up to 70% higher throughput
English
Krish Modi
211 posts
Krish Modi
@krishmodi404
@uwaterloo se, dev @palantirtech, prev @bloomberg, @Huawei, ISEF
Sarnia ON Se unió Şubat 2022
446 Siguiendo682 Seguidores
I took OSDN, a brand-new linear-attention model that learns to tune its own memory updates as it reads (think AdaGrad for the architectures trying to replace the transformer), rebuilt it from scratch in pure C++ with my own autograd engine, and ran it on a $4 microcontroller to predict hypoglycemia 60 minutes before it hits.
No PyTorch. No JAX. No TensorFlow. No ML library at all. Straight C++ standard library.
English
launching AgentIR Blackbox agentir.dev
an llm request router for agent system
Blackbox finds which llm calls are on your workflow’s critical path, sends them to faster providers, and routes less urgent calls cheaper to maintain your selected cost-latency constraint
it uses your workflow stats and real-time provider latency profiles to reroute before throttling or slowdowns hit the full workflow
setup is simple too. connect your app, and blackbox handles the workflow annotations for you
use it for free!
English
i’ve joined @datacurve to lead design in san francisco (as an intern)! check out our new site :)
English
ive witnessed krish work on agentIR for the past several months! it’s really magical to use, try it out!
Krish Modi@krishmodi404
launching AgentIR Blackbox agentir.dev an llm request router for agent system Blackbox finds which llm calls are on your workflow’s critical path, sends them to faster providers, and routes less urgent calls cheaper to maintain your selected cost-latency constraint it uses your workflow stats and real-time provider latency profiles to reroute before throttling or slowdowns hit the full workflow setup is simple too. connect your app, and blackbox handles the workflow annotations for you use it for free!
English
