Spider Cloud 🕷️

You can now use Spider in @llama_index as a web reader! Crawl/scrape urls and format the HTML into LLM ready markdown! Spider is the fastest web crawler built for AI Agents and LLMs. h/t @WilliamEspegren for the PR

Before writing cold email copy: Jina → homepage Spider → full crawl Claygent → pricing/careers Built With → tech stack Google News → triggers LinkedIn → profiles All synthesized into ONE research column. Then we write.

Dify v1.8.0 is live. This release makes it easier to refine prompts, fix code, and manage workflows. Refine or repair right inside LLM and Code nodes with an agent. Prompt and Code - Prompt Optimization: use {{last_run}} with your ideal outputs to quickly refine prompts and keep iterations under control. - Code Fix: auto repair captures {{current_code}} and {{error_message}} to generate corrected versions so you spend less time on manual debugging. - Version Management: every optimization and fix is saved as a version so you can compare and roll back any time. Workflow and agent upgrades - Multi model credentials: configure and switch between multiple keys for the same provider or a custom model. - MCP with OAuth: connect to MCP servers with OAuth, including token expiry control and callback allowlists. - Default values for workflow variables: all start node variable types now support defaults for faster setup - Agent node token usage: track token usage in agent nodes for better monitoring and optimization. Navigation and experience - Knowledge base sorting: sort documents by status for smoother management. - Extensible goto anything commands: a new architecture for faster navigation across projects. Plus performance, security, and infra improvements across the board. Full changelog: github.com/langgenius/dif… Happy building!

@gertjanwilde @MrAhmadAwais @firecrawl @gertjanwilde are you still having issues logging in? mind sending a DM to me?

@MrAhmadAwais Congrats! I'm running into problems, tho, trying to build a scraper agent. The @spider_rust login doesn't seem to be working, which prevents me from creating an API key. When I ask the agent to use @firecrawl instead, it still uses Spider.

For the first time, you can vibe-code any AI agent. Meet Chai.new — Computer Human AI by Langbase ☕ 🔹Prompt: "make an agent that…" 🔹Sip: chai builds any AI agent 🔹Ship: every agent gets a UI 🤯 Like your on-demand AI Engineer. What will you s(h)ip today?

@chrislaupama Dm’d ya

@spider_rust I’ve paid for credits but they haven’t been added to my account? INV 2194-0865

Use Spider with @julep_ai cookbook github.com/julep-ai/julep…

This is how I use LLM to scrape 99% of websites Many people didn't realise you can build agentic scraper to: 1. Handle Authentication, Human verification, Captcha 2. Handle pagination & complex UI interactions 3. Adaptive as website structure change 4. Scrape large set of data What used to take hours now can be automated in mins; Here I show case how do you automate a scraping job on Upwork where people are paying $50~$80 per hour; 0:00 Intro 1:54 Methods overview 4:52 Web Scraper agent using @firecrawl @spider_rust @JinaAI_ 8:37 Handle website auth & captcha using @AgentQL 20:27 AI buy tickets @MultiOn_AI If you have any further question or want to get deep dive into the code example in the video, you can join my community where I post tips weekly: link.agent.rocks/1BBC0Tp

Another great video from @jasonzhou1993 "This is how I scrape 99% websites via LLM" youtube.com/watch?v=7kbQnL…

congrats on the raise!

Super excited about @spider_rust! It’s going to significantly boost web crawling speed, making content gathering for #kai @GleapSDK faster and more accurate 🚀 #llm #aibot #aiagent #rag

@sergedoub @dify_ai Thanks for reaching out. I'll DM you to get more info

@spider_rust when trying things out in @dify_ai I made a crawl and a scrape step but they both resulted in /crawl in dashboard. Output in dify looks like scrape, just payload from the URL.

@1000_tools are you guys still taking in new tools? I just sent ya a DM

What every founder wants: → attract more leads → secure more clients → drive more sales This toolkit promises just that and more 👇 1000.tools/cold-outreach-…

@jasonzhou1993 If you're looking for a blazing fast crawler for scraping that is cost effective—then check us out spider.cloud

@jasonzhou1993 Spider is mentioned here: youtube.com/watch?v=YtBEc6…

Great tutorial on OpenAI Structured Output by @jasonzhou1993 — thanks for the Spider shoutout! youtube.com/watch?v=YtBEc6…

4/ Spider Web Scraper & Crawler component: A great tool for scraping and crawling webpages using @spider_rust .

Great update from the Dify team! ICYM: Spider @spider_rust can be used as a built-in tool in Workflow or as an LLM-callable tool in Agent

Next Generation web crawling and scraping that can handle thousands - millions of pages in seconds. The fastest and most affordable service built fully in Rust Lang. link.medium.com/lypEQ5KPRLb

Under a second to crawl over 100 pages? What just happened? spider.cloud