The Web Scraping Club

Google has deindexed my entire Substack substack.thewebscraping.club. 4 years of work. Hundreds of posts. Gone from the search overnight. The nightmare every content creator fears just happened to me.

If one of my posts has ever helped you, the best thing you can do right now is share it with ONE person who works in scraping, automation, or data engineering. @Substack @Google, any help appreciated.

But none of that matters when Google decides you no longer exist. In the meantime, I'm back to relying on the oldest growth channel: word of mouth. It's been my main driver until now. From here on, it's everything.

Google has deindexed my entire Substack substack.thewebscraping.club. 4 years of work. Hundreds of posts. Gone from the search overnight. The nightmare every content creator fears just happened to me.

Most e-commerce scrapers waste bandwidth fetching unchanged product pages repeatedly. We tested HTTP conditional requests against Shopify stores and found 95% bandwidth savings using ETags and 304 responses. One simple header can cut proxy costs dramatically. substack.thewebscraping.club/p/http-caching…

SerpApi's motion to dismiss argues the DMCA protects copyrighted works, not websites. Their position: "Google indexed the web without permission. Now it's trying to pull up the ladder behind it." Hearing set for May 19, 2026. substack.thewebscraping.club/p/google-vs-se…

This isn't just about SerpApi. If Google's theory prevails, every CAPTCHA and bot detection system on websites with copyrighted content could invoke federal law against scrapers. The entire industry's legal framework is at stake.

Google filed a 12-page lawsuit against SerpApi using the DMCA anti-circumvention provision - the same law designed to stop DVD piracy. No cease-and-desist. No communication. Straight to federal court.

66% more proxy usage. 86% report increased anti-bot protections. Only 46% using AI. Apify + @webscrapingclub surveyed scraping pros on what's working in 2026. The full report covers proxies, infrastructure, bot detection, and AI trends. Link in thread 👇

My #hottake on #webscraping of the day: Website owners: "We blocked GPTBot in robots.txt" Reality: Respecting robots.txt is voluntary. Courts have confirmed this. If you want legal protection, put it in the Terms of Service. That's the document that actually matters. Read more on substack.thewebscraping.club/p/understandin…

#Webscraping is split into two categories: 1. Traditional: anti-bot, fingerprinting, #CAPTCHA solving 2. #LLM-native: structured output, JSON schemas, #MCP servers #AnyCrawl is category 2. Open source. 2.5k stars. You describe what you want. AI extracts it. No selectors needed. Different tools for different problems. Link to article: substack.thewebscraping.club/p/anycrawl-tes…

We tested 3 browser tools against Cloudflare in 2026. Same tools. Two sites. Opposite results. Indeed.com: 31% success (Camoufox), 0% (Chrome tools) Harrods.com: 94% success (Camoufox), 89% (Chrome tools) The difference is not the tool. It's the target configuration. No universal bypass exists. Link: substack.thewebscraping.club/p/bypassing-cl…

Selenium, Playwright, Puppeteer. You use them daily. But do you know how they actually control browsers? Three protocols: - WebDriver: HTTP, cross-browser, polling - CDP: WebSocket, Chromium-only, real-time - BiDi: WebSocket, cross-browser, real-time BiDi is the future. It won't replace CDP. Link: substack.thewebscraping.club/p/webdriver-vs…

We scraped 1000 #Nike pages with 5 tools. HTTP client: 8 minutes Browser automation: 3.5 hours Same data. Same success rate. 27x speed difference. Nike checks TLS fingerprints, not JavaScript execution, at least in the product catalog. The data is in the first response. Stop paying the browser tax when you don't need to. Link: substack.thewebscraping.club/p/scraping-nik… #webscraping

Have you ever wondered how #MachineLearning can be used to detect #bot traffic? In the latest post of The Web Scraping Club, Federico Trotta showed us some details about it. From unsupervised training to supervised one, traffic data can be clustered by user behavior and system settings, identifying some patterns that distinguish bots and human. Link to article: substack.thewebscraping.club/p/machine-lear…

Ever wondered how to scrape data from the past? Look at this article, where we share how to scrape data from The WayBack machine. substack.thewebscraping.club/p/scraping-way…

How to scrape Vinted data? Have a look at the latest article on The Web Scraping Club, where we're using a cookie factory to interact with its internal APIs Read more here: substack.thewebscraping.club/p/how-to-scrap… #webscraping #vinted