Scrapling 🕷️

🚨 Scrapling v0.4.3 is here This update introduces a ton of changes, including 3 new MCP tools to persist browser sessions across tools, a new option to capture background requests during page fetch to easily scrape SPAs, a new sanitizer to protect against common prompt-injection attacks, and more. Here's the full release notes: github.com/D4Vinci/Scrapl… Let me know what you think! We are so back 🔥 Expect the next update soon 🚀

A big clap to our new Platinum Partner @PetroSky_Cloud 👏

So when they do it , it’s good. But when we do it, it’s bad? You can still do it for free: scrapling.readthedocs.io/en/latest/fetc…

A big clap to our new Platinum Partner @Nsocksproxy 👏

A big clap to our new Platinum Partner @TikHubio 👏

🚨 Scrapling v0.4.2 is here A new maintenance update with important changes 👉Bug fixes - The function get_all_text() now captures tail text nodes. This will make the MCP server and commands see text that was missed before. - Referer now returns a bare Google url instead of a Google search URL. The previous logic was incorrect and may have produced a fingerprinting signal. - Fixed an issue with extra flags concatenation in all browsers. - Fixed a type hints issue with Python versions below 3.12 that caused it to crash. 👉Other changes - Updated the Agent Skill on the repo and on Clawhub. - Updated all browsers and Playwright versions to the latest. - Added a French translation to the main README file. Full release notes here: github.com/D4Vinci/Scrapl… Let me know what you think

Congratulations, everyone, we have published an Agent skill for Scrapling 🎉 This skill should be readable by @openclaw, Claude Code, and other agentic tools. It encapsulates almost all of the documentation website's content in Markdown, so the agent doesn't have to guess anything. It can be used to answer almost 90% of any questions you would have about scrapling. We tested it on OpenClaw and Claude Code. If you encounter any issues, please open a ticket or use our Discord server. It's uploaded to the GitHub repo and to ClawHub right now. Check the links below 👇 ▶️Also, thanks to everyone who tried to make skills for scrapling and uploaded them to ClawHub, but almost all of them have some inaccuracies or issues in some parts. Now the official skill is here, please use it instead. Let me know your thoughts 🔥

Thanks, @AnthropicAI, for this Gift, and @lydiahallie for helping with the bug I faced. Scrapling made +16.5k GitHub stars in the last two weeks, and this will help me go above and beyond! Great things are coming ahead!

A big clap to our new Platinum Partner, Hyper Solutions.👏 Make sure to check them: hypersolutions.co/?utm_source=gi…

OpenClaw users can now calm down a bit🧘‍♂️ I'm testing the OpenClaw skill for Scrapling and will officially publish it soon. Also, expect it to be added to Clawhub 🦀 After that, expect the Claude Code Plugin/Skill soon.

@maciejewskii0 @RoundtableSpace Through post requests or through automation. The automation part isn’t available if you are using it from the MCP since it requires coding.

@RoundtableSpace ok but how does it handle login walls

OpenClaw can now scrape any website without getting blocked - zero bot detection, bypasses Cloudflare natively, 774x faster than BeautifulSoup. No selector maintenance. No workarounds. Just data. THIS IS AN UNFAIR ADVANTAGE AND IT'S FULLY OPEN SOURCE.

A huge clap to our biggest partner, DataImpulse. 👏 Make sure to check them: dataimpulse.com/?utm_source=sc…

@DataChaz Thanks for writing about us 🌹

@Scrapling_dev Your library is great! I just posted a short thread about it :) x.com/DataChaz/statu…

A big clap to our new Platinum Partner @BirdProxies_ 👏

🚨 Scrapling v0.4.1 is here I have never imagined this version of Scrapling to do so well like that, and all this feedback from the community 🙏 Yesterday, Scrapling was #1 on GitHub's trending list across all programming languages, and this update is my way of saying thanks! Here's what to expect with this update: - Cloudflare solving is now much more efficient and nearly twice as fast. - The stealth mode of the browser is now better and faster than before. - Improved the MCP schema so it's now accepted by strict tools like Open Code and VS Code Copilot without issues. - Improved the MCP server tokens consumption by a large margin. - Scrapling's MCP server is now registered on the MCP registry. - Added a new code snippet to show how to install the browsers deps through code instead of using the commandline to allow easier automation. and more. Check out the full details here: github.com/D4Vinci/Scrapl… So, what do you think about this update?

Scrapling is an open-source (BSD-3) Python library for adaptive web scraping, with features like stealthy fetching, Cloudflare bypass (e.g., Turnstile via solve_cloudflare), and fast parsing. Benchmarks show it's up to 698x faster than BeautifulSoup with Lxml (post claims 774x—minor exaggeration). Original post's "unfair advantage over every other AI agent" is hyperbole, as it's publicly available, not exclusive to OpenClaw. Comment accurately notes it's a general tool with optional hype. Sources: GitHub repo, PyPI.

🚨 OpenClaw just got an unfair advantage over every other AI agent on the internet. It's called Scrapling and it scrapes undetectable, adaptive websites without breaking when they update their structure. No bot detection. No selector maintenance. No Cloudflare nightmares. OpenClaw tells Scrapling what to extract. Scrapling handles the stealth. Clean data lands in your agent in seconds. → 774x faster than BeautifulSoup with Lxml → Bypasses ALL types of Cloudflare Turnstile automatically → pip install "scrapling[ai]" and your AI agent is scraping in 60 seconds Works everywhere: → HTTP + browser automation → CSS, XPath, text, regex selectors → Async sessions for parallel scraping → CLI with zero code required If you're building AI agents that need real web data, this is the scraping backbone OpenClaw has been missing. 100% Opensource. BSD-3 license. Link in first comment 👇

@GithubProjects @D4Vinci1 Thanks for writing about us 🌹

Scrapling v0.4 is here — the biggest update yet 🕷️ New: Async Spider Framework A full crawling framework with a Scrapy-like API — define a Spider, set your URLs, and go. - Concurrent crawling with per-domain throttling - Mix HTTP, headless, and stealth browser sessions in one spider - Pause with Ctrl+C, resume later from checkpoint - Stream items in real-time with async for - Blocked request detection and automatic retries - Built-in JSON/JSONL export - Detailed crawl stats and lifecycle hooks - uvloop support for faster execution New: Proxy Rotation: Thread-safe ProxyRotator with custom rotation strategies. Works with all fetchers and spider sessions. Override per-request anytime. Browser Fetcher Improvements: - Block requests to specific domains with blocked_domains - Automatic retries with proxy-aware error detection - Response metadata tracking across requests - Response.follow() for easy link-following Bug Fixes: - Parser optimized for repeated operations - Fixed browser not closing on errored pages - Fixed Playwright loop leak on CDP connection failure - Full mypy/pyright compliance Upgrade: pip install scrapling --upgrade Full release notes & docs: github.com/D4Vinci/Scrapl… Try it out and let me know what you think!

A week ago we published version 0.3.13 which is very big and important to upgrade to. There are a lot of changes and breaking changes, check the full release notes from here: github.com/D4Vinci/Scrapl… then check v0.3.14 too :)