Vijay
4.9K posts

Vijay
@Vijay
Doting dad 👶, passionate about marketing and product development through the art of storytelling , AI Automation #Drone riders (trisonics) threat detection ROS










📢 WebGPU is now officially supported across Chrome, Edge, Firefox, and Safari → goo.gle/3MmN7Rj Access high-performance 3D graphics and AI capabilities right in the browser with this major milestone.


LiteLLM HAS BEEN COMPROMISED, DO NOT UPDATE. We just discovered that LiteLLM pypi release 1.82.8. It has been compromised, it contains litellm_init.pth with base64 encoded instructions to send all the credentials it can find to remote server + self-replicate. link below



Lately, I have been trying to beat Google. Ok not really xD I have been building a search engine over niche web data and it's been one of the most fun projects I've ever worked on. The idea of querying the internet exactly the way I want, at scale, has always fascinated me. Especially what I could do with it: look at 1,000 companies' job postings and find hiring trends using natural language, map competitive landscapes including the lesser known players, find alternative data for investing. Or just a personalized corner of the web, curated for me. It's been an interesting journey. Started out with 200 workers on localhost, rewrote the crawler from Python to Rust to get up to 1,600 workers in parallel. Moved it to the cloud. Hit challenges that made the whole Rust rewrite useless xD Then I woke up to an abuse report from a French website with the words "European Commission", "appeal", "court" in it. For a second I genuinely thought I was about to have legal proceedings against me. Phew. Nevertheless it's been the most fun project I've worked on. It's surprising how you can cover a lot of information starting from a modest number of seed URLs. I'll be sharing everything I learned in the open. Code will be open sourced soon and the search engine will be live too. If you're curious how to build a distributed web crawler that covers 10M+ pages under $200/month, do read my blogpost. Blogpost Link: lnkd.in/g7MsaXYA











