Screaming Frog

Just released @screamingfrog SEO Spider v.23: screamingfrog.co.uk/blog/seo-spide… Includes - 💡Lighthouse & PSI Updated to Insight Audits ⚙️Crawl Retention Config 🧑‍🤝‍🧑Semantic Similarity Embedding Rules 📊Display All Links in Visualisations 🍇Links in Content Cluster Diagram & more!

The Dark Side of AI No One Talks About - moz.com/blog/dark-side… on @Moz

How Much Do Keywords Matter in 2026? - moz.com/blog/do-keywor…

@Polo_Seo Not sure I understand what you're asking. But welcome to send anything through to support@screamingfrog.co.uk? A live example is always helpful.

@screamingfrog qui ne voit pas la balise <figure> dans le code, mais qui le voit sur son gestionnaire d'xpath on en parle ?

@Caner80 Not directly in the UI, you'll need to use the 'Reports > Redirects > All Redirects' report (or we have specific ones for 'Redirects to Error' etc). If you want it to follow external URLs, then you'll need to enable 'Always Follow Redirects' via 'Config > Spider > Adv' as well.

@screamingfrog when I crawl, isn’t it possible to let the frog follow redirect chains and show the chain in separate columns like redirect 1 + statuscode, redirect 2 + statuscode, and so on, to see directly if the last part of the chain is 200 or 404?

@IrishWonder Cheers for the suggestion! We flag issues like indexable search results pages, so could possibly help. Obv concern would be anyone taking it too literally! 👍👌

Feature suggestion @screamingfrog : let me build a robots.txt file for a site based on the crawl results/analysis

Your Accept-Language Redirects Could Be Blocking Search Engines and AI Crawlers - merj.com/blog/your-acce… from @giacomozecchini

Technical SEO Tip: Screaming Frog has the power to build custom extraction commands. Simply click on a page element...and it writes extractions:

@effi_10 Sorry, bit late seeing this, but - #why-do-i-get-a-429-too-many-requests-error-response-crawling-a-shopify-site" target="_blank" rel="nofollow noopener">screamingfrog.co.uk/seo-spider/faq…

Sur #Shopify pour un crawl Screaming Frog il y a une astuce pour ne pas avoir d'erreurs 429 ? Ou c'est lié au site / thème ? Pas possible que ça rame comme ça -_-

@s_matharu Yeah, it only applies to Internal URLs - so use 'Config > CDNs' to make them Internal, and it should work nicely!

@screamingfrog I'm using the image alt generator, but my images are held on a CDN through a different host URL. How do I generate alt descriptions for these, as it seems to be only generating for what images it finds via the Internal tab?

@screamingfrog I might have read it before but is it possible run an OpenAI prompt against an existing crawl? Example, missing alt text generation

🌍 How to use embeddings to map hreflang tags at scale - A clever use of @screamingfrog's Open AI API integration to implement hreflang by @pelogia - Check it out: guspelogia.com/embeddings-hre…

@seo_sitch I've added a vote in for you for that one! Cheers for the feedback.

@screamingfrog I would +1 that feature request, I'm building workflows around these now.

@screamingfrog Is there a way to save all n-grams at once, rather than URL by URL?

Everyone talks about Google Discover like it's a black box. Post great content. Use big images. Hope for the best. That's the advice. That's been the advice for years. I decided to take a different approach. Instead of guessing, I went into Google's own SDK and read what it actually exposes during normal operation. Event constants. Telemetry counters. Configuration values. Things that are sitting right there if you know where to look. What came out was a picture of a 9-stage content pipeline, and it changed how I think about Discover entirely. Knowledge Graph entities? I need to look carefully! Here is the part that surprised me most: the collection-level filter runs before interest matching. Before the pCTR model. Before any ranking happens at all. If a publisher is blocked at that stage, Google never even evaluates whether their content matches what the user wants. It is binary. One article generates enough negative feedback, and the entire domain gets suppressed. There is no equivalent blanket boost. The penalty surface is wider than the reward surface. Some other things the telemetry confirms: og:title is not just a display label, it feeds directly into the predicted click-through rate model. The first 7 days carry the highest freshness weight. After 30 days, continuous staleness decay begins, tracked in hours. And dismissed content never comes back, tombstones are permanent records stored on-device. There are also 276 event constants, 150 concurrent A/B experiments running on a single session, and a system called NAIADES that most publishers have never heard of. The full breakdown, with every finding traced to a specific string or event, is on my blog. I also created a dashboard with all specifications with a FREE pCTR tool. Link is in the first thread!

@seo_sitch Awesome. Yeah, you do for each n-gram still I’m afraid. We might introduce an ‘all’ filter, it’s on the list!

@screamingfrog Thanks, that was surprisingly simple and I was overthinking it. It does look like I still need to go one at a time for 2-gram, 3-gram, etc.?

Technical SEO Tip: By enabling Screaming Frog's JavaScript crawling, you can take fully-rendered screenshots of every single page on your site:

@stonetatara Cheers for the suggestion, we do have this on the 'todo' list, so will add a vote in for you. Thank you!

Hi @screamingfrog , I miss a shorcut to create Segment from url (Right click on a url + Create segment) Hope you'll validate this AMAZING feature to the roadmap 🙏

Analysing and Reconstructing Site Architecture Using Breadcrumbs: A Practical SEO Guide - screamingfrog.co.uk/blog/reconstru…

@BKureishi We stagger it a little. You're welcome to update via the site though!

@screamingfrog How come the update has not rolled out on my end yet?

Just released a small update to 23.3 of the @screamingfrog SEO Spider - #23.3" target="_blank" rel="nofollow noopener">screamingfrog.co.uk/blog/seo-spide…. Includes an update to issue 'HTML Document Over 2MB' with Google's update on file size limits, alongside small improvements and bug fixes.