Screaming Frog

Just released @screamingfrog SEO Spider v.24: screamingfrog.co.uk/blog/seo-spide… Includes - 🤖Screaming Frog MCP 🧑‍🤝‍🧑Auto Compare Crawls 📊View Crawl Changes in Email Notifications 📧Send Crawl Export Attachments by Email 🚫Find Uncrawlable Links & lots more! Enjoy.

Congrats 🇪🇸. Best team in the comp, much deserved. ⚽️

@HahnRaphael Please can you send it through to us via support (support@screamingfrog.co.uk), so we can take a look! Cheers

Hey @screamingfrog, can you confirm a bug I've been experiencing? By default, the SF accept-encoding is set to "gzip, deflate, br, zstd". The webpage returns zstd compression, as advertised. But SF stores the body undecoded: verified by magic bytes, stored size = transferred size, and source view is obviously gibberish. Thus no crawl analysis at all. This only happens with zstd, not brotli or gzip. I don't see a public report of this behaviour.

Really late to the party, but just found out you can open more than one Screaming Frog instance on a Mac by opening terminal and typing: open -n /Applications/Screaming\ Frog\ SEO\ Spider.app Wish I knew this 5 years ago 😂. Still... Better late than never.

@RemiBrandini Send through the details to support, including any specific errors displayed - screamingfrog.co.uk/seo-spider/sup…

Hey @screamingfrog - we use SF via Claude Cowork to trigger crawls automatically. The remote bridge between Claude desktop and cloud sessions seems broken on our end.

Screaming Frog and SISTRIX AI data: Case Study, MCP Intergration - sistrix.com/blog/sistrix-s… (from @sistrix and @tomjeffery8 )

@BrendanOConnel 🙌👍 thanks mate

@screamingfrog MCP is a game changer - kudos to the team.

Just released @screamingfrog SEO Spider v.24: screamingfrog.co.uk/blog/seo-spide… Includes - 🤖Screaming Frog MCP 🧑‍🤝‍🧑Auto Compare Crawls 📊View Crawl Changes in Email Notifications 📧Send Crawl Export Attachments by Email 🚫Find Uncrawlable Links & lots more! Enjoy.

@facan Thanks for everything, and all the best in retirement, Fabrice!

My dear followers, My dear fellow Microsoftees, engineers, attorneys, marketers, webmasters, publishers, SEO champions, product leaders, journalists, and all the brilliant minds across search, AI, and yes, even my friends at Google. Today marks nearly 30 years with Microsoft. Thirty years… far too short a time to spend among such excellent and admirable people. As Bilbo once said, “I don’t know half of you half as well as I should like; and I like less than half of you half as well as you deserve”. I stand before you for three purposes. First, to tell you that I am immensely fond of you all. These three decades have been a wonderful adventure: from solving real business problems with IndexNow to helping webmasters and publishers thrive in the ever-changing world of SEO and AI. It has truly been an incredible journey. Second, to celebrate this milestone and to thank every one of you: the colleagues I worked alongside daily, the partners and publishers I collaborated with, and especially the talented people I’ve trained and mentored over the past two years. You are ready, like young hobbits setting out with Sting and courage, to carry the team and our mission forward with excellence. Third and finally, after many evenings by the fireside with my family and friends (who wisely reminded me that even the longest journey must eventually turn toward home), I have decided to take advantage of Microsoft’s Voluntary Retirement Program. So, though thirty years is far too short a time to spend among such excellent and admirable people… this is the end. I am going. I am retiring from Microsoft, effective today July 1st. GOODBYE! As Bilbo said, the road goes ever on… but for me it now turns west, toward a place of rest, starlight, song, and new tales after many adventures. I leave with a heart full of gratitude, memories brighter than mithril, and complete confidence in the bright future of the team. Thank you, Microsoft. Thank you to every colleague, partner, and webmaster who made this journey so special. Stay curious. Keep innovating. And may your content always be found; swift as eagles and sure as the stars. Fabrice Canel Retiring after 30 wonderful years at Microsoft (Bing)

@AlexEntwistle Hi Alex, pop through the details to support - screamingfrog.co.uk/seo-spider/sup… and we can help! Cheers

@screamingfrog I'm getting the following error when trying to compare two crawls of the same website in Compare mode (text rendering vs JS rendering). Any idea how to fix it?

Sad news. RIP Bruce Clay - seroundtable.com/bruce-clay-rip…

Today we released the June 2026 spam update. We'll update our ranking release history page when the rollout is complete: status.search.google.com/incidents/YUX1…

How is Reddit eating local publishers' AI visibility in non-English markets? I've been studying AI citation behavior for months. This finding still surprised me. We analyzed 64.77 million Reddit citations across 20 countries and 4 LLMs at @peec_ai In Sweden and Norway, over 70% of Reddit citations on Google AI Overview and AI Mode point to machine-translated versions of English threads. It's eating local publishers visibility. You can't compete. A THREAD 👇🧵 (1/n)

Just released a small update to 24.2 of the @screamingfrog SEO Spider - #24.2" target="_blank" rel="nofollow noopener">screamingfrog.co.uk/blog/seo-spide… Includes various small improvements and bug fixes.

New @screamingfrog tutorial 'Simulating User Interactions with Custom JavaScript' - screamingfrog.co.uk/seo-spider/tut…

@CodyStallard Hi Cody, you can use 'Bulk Export > Structured Data > Contains SD' to export it all!

@screamingfrog hey guys, I'm trying to export all custom JSON schema/structured from every page on a website for a migration, but I'm struggling. Do you have a guide to an exact doc to help with this?

@glenngabe Shocked to hear the news. So awful and sad. Thoughts with all friends, and of course family.

I unfortunately have terrible news to share. In April I shared about Jenn Bauguess, a fellow SEO who was battling an aggressive form of cancer. I was shocked to learn that Jenn passed away on Thursday in Tucson. I received the call on Saturday explaining what happened, and again, I was shocked to hear the news. We literally just had an email conversation last week talking about SEO, and just a few days later, Jenn is gone. Jenn’s son is just nine and it’s heartbreaking to know he just lost his mom, and so suddenly. I’ll share the details of the service once those arrangements have been made. But thank you to the SEO community for your help and support for Jenn. Her GoFundMe went from $9K to $20K after you all got involved. And that was a huge help for Jenn. Cancer sucks. This all happened so quickly. I’ll share more when I have the details. Amber who set up the GoFundMe posted a message yesterday about Jenn passing away. The link is below. gofundme.com/f/support-jenn…

@DanielTexasLira You can just input the XML Sitemap Index file, and it will crawl all the XML Sitemaps in it (and all the URLs in them). We wouldn't crawl nested sitemaps though (ie sitemap index to other sitemap index to sitemaps).

@screamingfrog If I have an xml sitemap that contains individual xml files within it. Do I need to list them all in screaming frog or will just including the XML sitemap be sufficient? I'm currently trying to figure out if there is an issue with the site or the crawl config.

Full article peec.ai/blog/using-mcp…

@TomekRudzki Welcome back, mate! 🔥

I'm coming back to X after quite a long break 🔥. You can expect from me tips on SEO and improving AI search visibility. I will start with sharing my setup for analyzing GSC, @screamingfrog data and AI search visibility data via MCP. Now I connect each tool over MCP and ask my questions in plain language. The model picks the right query, runs it, and reads the result back to me. Here's my setup: 1️⃣ Search Console data → I pipe it into and connect to the official Google BigQuery MCP. Then I ask things like "which pages rank well but barely get clicks?" or "was this week's drop a clicks problem or an impressions problem?" 2️⃣ AI visibility → Peec AI MCP. Which engines am I winning, who's beating me in the prompts I track, and which sources do the models actually cite. 3️⃣ Technical SEO → Screaming Frog MCP (native since v24.0). I ask for a crawl in plain English and follow up with "what changed since last crawl?" instead of diffing two exports. And when a tool has no MCP yet? I point Claude Code at the API docs, or just hand it the CSV. Either way I'm back to asking the right questions instead of building pivots. You don't need to be an engineer. You need to know what to ask. 🤓 I put the full workflows, the exact prompts, and experts' takes in a guide - link in the comments. 👇

@DanFoland Sorry Dan, slow to reply! No, there isn't a way to do a post-analysis currently. We have it on the 'todo', but at the moment search/extraction is all at runtime of the crawl.

@screamingfrog Is there a way to search through body text after a crawl? I know about custom search, wondering if there's a post-crawl way using the stored html feature.

No, that wouldn’t make sense alone. Fundamentally under the covers the only difference is list mode has crawl depth set to ‘0’. You can have diff configs between list and spider mode don’t forget though (i.e user-agent, speed etc) - so perhaps there is something else in play in your settings. If you’ve added your IP to an allowlist and you’re getting 429s though, that would suggest it’s not working as it should obv!

Hey @screamingfrog have you ever seen an issue with a crawl being rate limited (i.e., a 429 too many requests) that ONLY occurs when you're in list mode? Regular crawls of a site don't get rate limited. I've even whitelisted my IP. But a crawl of about 10k URLs in List mode gets rate limited after about 150 URLs. Even tried to use another ISP, and get the same thing. There's something about List Mode in SF that is the issue is seems.